Backup does not work anymore after update to mailcow 2025-12

abyss02

Hi there,
i am running mailcow-dockerized an a debian 13 server:

Yesterday i updated to mailcow 2025-12, but now i start the backup running:

MAILCOW_BACKUP_LOCATION=/opt/mailcow-backup/ /opt/mailcow-dockerized/helper-scripts/backup_and_restore.sh backup all

But some of the backup files are size 0:

[root@mail mailcow-backup]# ll mailcow-2025-12-10-11-27-46/
total 20052
-rw-r–r– 1 root root 0 Dec 10 11:27 backup_crypt.tar.zst
-rw-r–r– 1 root root 20519849 Dec 10 11:27 backup_mariadb.tar.zst
-rw-r–r– 1 root root 0 Dec 10 11:27 backup_postfix.tar.zst
-rw-r–r– 1 root root 0 Dec 10 11:27 backup_redis.tar.zst
-rw-r–r– 1 root root 0 Dec 10 11:27 backup_rspamd.tar.zst
-rw-r–r– 1 root root 0 Dec 10 11:27 backup_vmail.tar.zst
-rw——- 1 root root 10607 Dec 10 11:27 mailcow.conf

While running the backup job i see these error messages:

Using 1 Thread(s) for this run.
Notice: You can set the Thread count with the THREADS Variable before you run this script.
Using /opt/mailcow-backup as backup/restore location.

Found project name mailcowdockerized
/vmail/
/vmail/mailcow.local/
/vmail/shared-mailboxes.db
/vmail/sieve/
/vmail/sieve/global_sieve_before.svbin
/vmail/sieve/global_sieve_after.svbin
/vmail/sieve/webmaster@mydomain.sieve
/vmail/sieve/global_sieve_after.sieve
/bin/sh: 1: zstd: not found
/vmail/sieve/global_sieve_before.sieve
/vmail/sieve/webmaster@mydomain.svbin
/vmail/mydomain/
/vmail/mydomain/mail/
/vmail/mydomain/mail/.dovecot.lda-dupes
/bin/tar: /backup/backup_vmail.tar.zst: Cannot write: Broken pipe
/bin/tar: Child returned status 127
/bin/tar: Error is not recoverable: exiting now
/crypt/
/crypt/ecpubkey.pem
/crypt/ecprivkey.pem
/bin/sh: 1: zstd: not found
/bin/tar: Child returned status 127
/bin/tar: Error is not recoverable: exiting now
OK
/redis/
/redis/dump.rdb
/bin/sh: 1: zstd: not found
/bin/tar: /backup/backup_redis.tar.zst: Cannot write: Broken pipe
/bin/tar: Child returned status 127
/bin/tar: Error is not recoverable: exiting now
/rspamd/
/rspamd/b669b0275c5d0df26a3926bdfa380373aacb86835537648fe1cdd7f20f0fa49e.hs
/rspamd/b9999708a94656035accbe83e4152a009dd7c344f0e042f1f164b5139ab931a4.hsmp.unser
/rspamd/b66d28e19182acc9bf01cdb8ab8b590d4b9bb209d2b0ecd21a3f2f34f9551dca.hsmp
/bin/sh: 1: zstd: not found
/bin/tar: /backup/backup_rspamd.tar.zst: Wrote only 4096 of 10240 bytes
/bin/tar: Child returned status 127
/bin/tar: Error is not recoverable: exiting now
/postfix/
/postfix/corrupt/
/postfix/lib/
/postfix/public/
/postfix/public/pickup
/postfix/public/qmgr
/postfix/public/watchdog_qmgr
/postfix/active/
/postfix/trace/
/postfix/pid/
/postfix/pid/inet.127.0.0.1:10027
/postfix/pid/unix.watchdog_discard
/postfix/pid/unix.relay
/postfix/pid/unix.trace
/postfix/pid/unix.smtp_enforced_tls
/postfix/pid/unix.watchdog_cleanup
/postfix/pid/inet.588
/postfix/pid/unix.dnsblog
/postfix/pid/unix.showq
/postfix/pid/unix.lmtp
/postfix/pid/unix.defer
/postfix/pid/unix.smtp_via_transport_maps
/postfix/pid/inet.590
/postfix/pid/unix.bounce
/postfix/pid/inet.submission
/postfix/pid/inet.589
/postfix/pid/unix.cleanup
/postfix/pid/inet.smtps
/postfix/pid/unix.local
/postfix/pid/unix.smtp
/postfix/pid/unix.retry
/postfix/pid/unix.flush
/postfix/pid/pass.smtpd
/postfix/pid/unix.smtp_sender_cleanup
/postfix/pid/master.pid
/postfix/pid/inet.127.0.0.1:10028
/postfix/deferred/
/postfix/deferred/9/
/postfix/deferred/5/
/postfix/deferred/1/
/postfix/deferred/0/
/postfix/deferred/2/
/postfix/deferred/6/
/postfix/deferred/B/
/postfix/deferred/C/
/postfix/deferred/8/
/bin/sh: 1: zstd: not found
/postfix/deferred/3/
/postfix/deferred/E/
/postfix/deferred/7/
/postfix/deferred/D/
/postfix/deferred/F/
/postfix/deferred/A/
/postfix/deferred/4/
/postfix/etc/
/postfix/private/
/postfix/bounce/
/postfix/flush/
/postfix/hold/
/postfix/saved/
/bin/tar: /backup/backup_postfix.tar.zst: Cannot write: Broken pipe
/bin/tar: Child returned status 127
/bin/tar: Error is not recoverable: exiting now
Using SQL image mariadb:10.11, starting…

Seems, that zstd is missing, but zstd is installed on my host:

[root@mail mailcow-backup]# dpkg -l | grep zstd
ii libzstd1:amd64 1.5.7+dfsg-1 amd64 fast lossless compression algorithm
ii zstd 1.5.7+dfsg-1 amd64 fast lossless compression algorithm – CLI tool

[root@mail mailcow-backup]# which zstd
/usr/bin/zstd

When i ran the backup previosly to 2025-12, i get the complete backup:

[root@mail mailcow-backup]# ll mailcow-2025-12-09-01-00-02/
total 3433508
-rw-r–r– 1 root root 555 Dec 9 01:01 backup_crypt.tar.gz
-rw-r–r– 1 root root 30728787 Dec 9 01:02 backup_mariadb.tar.gz
-rw-r–r– 1 root root 1769 Dec 9 01:01 backup_postfix.tar.gz
-rw-r–r– 1 root root 3308089 Dec 9 01:01 backup_redis.tar.gz
-rw-r–r– 1 root root 26427211 Dec 9 01:01 backup_rspamd.tar.gz
-rw-r–r– 1 root root 3455415330 Dec 9 01:01 backup_vmail.tar.gz
-rw——- 1 root root 10607 Dec 9 01:00 mailcow.conf

Any hints???

Andreas

There is no backup container running, ist that okay? Will it be startd only for the backup procedure?

[unknown] There is no backup container running, ist that okay? Will it be startd only for the backup procedure?

Okay, i found out that this one ist just temporarily running ..

abyss02

—
Okay, the backup container seems to miss zstd:

[root@mail mailcow-backup]# docker images
i Info → U In Use
IMAGE ID DISK USAGE CONTENT SIZE EXTRA
ghcr.io/mailcow/acme:1.94 7e213b95e653 109MB 0B U
ghcr.io/mailcow/backup:latest 83315cfb39ae 94.7MB 0B
ghcr.io/mailcow/clamd:1.71 e7280323ab3c 99.9MB 0B U
ghcr.io/mailcow/dockerapi:2.11 a578764a1914 112MB 0B U
ghcr.io/mailcow/dovecot:2.3.21.1 5de1344cc0a1 359MB 0B U
ghcr.io/mailcow/netfilter:1.63 429d0a2cacd9 105MB 0B U
ghcr.io/mailcow/nginx:1.05 2db944987735 113MB 0B U
ghcr.io/mailcow/olefy:1.15 df7707098546 115MB 0B U
ghcr.io/mailcow/phpfpm:8.2.29 86ba5c39b5b3 267MB 0B U
ghcr.io/mailcow/postfix-tlspol:1.8.22 87b920e5c58c 203MB 0B U
ghcr.io/mailcow/postfix:3.7.11 51f4ffc041d6 336MB 0B U
ghcr.io/mailcow/rspamd:2.4 a7eb10e0d36b 194MB 0B U
ghcr.io/mailcow/sogo:5.12.4 b112c8b4b209 633MB 0B U
ghcr.io/mailcow/unbound:1.24 c79028c964c2 89.8MB 0B U
ghcr.io/mailcow/watchdog:2.09 85f236f9c807 119MB 0B U
mailcow/backup:latest 31d852daa323 98.9MB 0B
mariadb:10.11 3a3c046f59cd 331MB 0B U
mariadb:10.5 88680d56d3a9 395MB 0B
mcuadros/ofelia:latest 9df7812f2acd 24.1MB 0B U
memcached:alpine f6411cdd430a 13.1MB 0B U
nginx:mainline-alpine a5967740120f 52.5MB 0B
redis:7-alpine ee33180a8437 41.4MB 0B
redis:7.4.2-alpine 8f5c54441eb9 41.4MB 0B
redis:7.4.6-alpine 5f703438f575 41.4MB 0B U
robbertkl/ipv6nat:latest b789d3355d66 18MB 0B
[root@mail mailcow-backup]# docker run -it 83315cfb39ae /bin/sh
which zstd
bash
root@2684c0be97f2:/# which zstd
root@2684c0be97f2:/# dpkg -l | grep zstd
ii libzstd1:amd64 1.5.4+dfsg2-5 amd64 fast lossless compression algorithm
root@2684c0be97f2:/# apt install zstd
Reading package lists… Done
Building dependency tree… Done
Reading state information… Done
The following NEW packages will be installed:
zstd
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 701 kB of archives.
After this operation, 2152 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian bookworm/main amd64 zstd amd64 1.5.4+dfsg2-5 [701 kB]
Fetched 701 kB in 0s (12.2 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package zstd.
(Reading database … 6098 files and directories currently installed.)
Preparing to unpack …/zstd_1.5.4+dfsg2-5_amd64.deb …
Unpacking zstd (1.5.4+dfsg2-5) …
Setting up zstd (1.5.4+dfsg2-5) …
root@2684c0be97f2:/#

Okay, had to stop mailcow and delete the backup docker image. after a redeploy it works now …

Sorry!

Okay, had to stop mailcow and delete the backup docker image. after a redeploy it works now …

Sorry!

timber

I can confirm I ran into the same issue and fixed it by deleting the backup image. The backup_and_restore.sh fetches the new one automatically.

mlcwuser

timber Yep, that worked. Thanks.

fjansen04

timber Can you tell me how to delete this specific image?

down16

sudo docker system prune -a –volumes
this command help to me.
and command sudo bash backup_and_restore.sh backup all finished without problem.

timber

fjansen04 Yes, you run „docker images“ and look for the ID of the „ ghcr.io/mailcow/backup:latest“ image. Then you enter „docker image rm [ID]“ where you replace [ID] with the actual image ID from above.

fjansen04

Thank you, timber! That did the trick.

KaiserN

oh nice, thanks @timber

fixed it

@DerLinkman fyi: I think that’s still flying under your radar (no GH issue so far).

DerLinkman

Nope. Fixed already, new versions of the script will fetch the latest image whenever there is a newer one available.

But, until the next update this won’t be included. Depending on the amount of questions regarding this, we might release 2025-12a tomorrow only including this “fix”.

The reason is, that the locally cached docker backup image does not contain zstd and therefore it may needs to redownload (as wroted above).

KaiserN

oh somehow is missed 5d95c48. Thanks for the hint xD

DerLinkman

KaiserN Your welcome 🙂 No worries.

KaiserN

And yes, I think it makes sense to release it independently; backups should be running.