Btw, this is the commit which changed the compression from lz4 zo zstd in August

mailcow/mailcow-dockerizedc134f0b

I guess this is why you “successfully tested the migration of old e-mails in May”

OK, I was really restless because of this issue, so I installed a new ARM server in my Hetzner cloud. I setup a new nightly mailcow installation and created one of my live domains with a mailbox.
Then I imported the crypt and vmail backup from my current mailcow, logged into Sogo and voila, dovecot complained about not being able to decrypt the mails

I tried to set the compression of dovecot to LZ4, but unfortunately it wasn’t compiled wit LZ4 support…

Error: zlib_save: Support not compiled in for handler: lz4

So I went along and used the doveadm commands from docs.mailcow.email Icon Mail crypt - mailcow: dockerized documentation

docs.mailcow.email Icon docs.mailcow.email
Mail crypt - mailcow: dockerized documentation
None
docs.mailcow.email
to decrypt the mails in my mailbox, and then encrypted them again. (I had to remove the regex parameters as the busybox find doesn’t regognize them)

Result:

I guess it would be much easier to use a dovecot image compiled with LZ4, otherwise all mailcow admins will have to do the above decrypting and recrypting if zstd is used. And that could take a very long time if you have GBs of mails…

I actually ran into this problem and I’m still trying to figuring it out.

I unfortunatelly switched from the arm branch to the nightly branch a month ago and did non see that I was actually not able to open mails prior to the switch. Noticed it some days ago prior the blog post.

I have created a new arm machine on hetzner, installed the nightly build and restored a backup from my main maschine. After that I tried the decrypt but I’m getting:
Fatal: fs_init() failed: compress: Compression method ‘lz4’ not supported.

Any hint?

Ah sorry, I forgot to mention that I had to install the ‘dovecot-common’ package in the host system and perform the decryption directly in the docker vmail volume under /var/docker/volumes/ because lz4 wasn’t compiled in the docker dovecot image. You have to adjust the path of the find command accordingly!

But if you have mixed compression mails in your maildir I think you have to be careful and ONLY do this for the LZ4 compressed mails!

But if you have mails with mixed compression in your maildir I think you have to be careful and ONLY do this for the LZ4 compressed mails!

    /var/lib/docker/volumes/mailcowdockerized_vmail-vol-1/_data is the directory

    DocFraggle

    Thanks for the hint as indeed this is now running. And yes it is a problem that I have mixed compressions in my maildir, and the script is noticing it ;-)
    Error: read(/xy/Maildir/.Archive/cur/1695363028.M982272P210299.2a3711cb79d9,S=8715,W=8852:2,S) failed: Decryption error: no private key available

    In regards to your warning: As I’m doing it at the moment on a test instance that is fine. For my live machine I’m going at the moment a different route
    1) I used imapsync to sync the mailboxes to an x86 instance and imapsync skipped the once that can not be fetched due not being able to decrypt
    2) I will decrypt all the mails on the test instance and use also imapsync to push them to the x86 instance
    This should give me at the end a full mail inbox

    Well such a mess, but hey they have warned not to use it productively, so it kinda my problem i am trying to solve!

    Thanks for your investigation and sharing it!

        asoprano you could wrap the doveadm command in another if, so that it only continues when the decryption worked with LZ4 compression. Something like

        find /var/vmail/ -type f -regextype egrep -regex '.*S=.*W=.*' | while read -r file; do
if [[ $(head -c7 "$file") == "CRYPTED" ]]; then
  if doveadm fs get compress lz4:1:crypt:private_key_path=/mail_crypt/ecprivkey.pem:public_key_path=/mail_crypt/ecpubkey.pem:posix:prefix=/ \
    "$file" > "/tmp/$(basename "$file")"; then
    if [[ -s "/tmp/$(basename "$file")" ]]; then
      chmod 600 "/tmp/$(basename "$file")"
      chown 5000:5000 "/tmp/$(basename "$file")"
      mv "/tmp/$(basename "$file")" "$file"
    else
      rm "/tmp/$(basename "$file")"
    fi
  fi
fi
done

        This should decrypt all LZ4 compressed mails. If you then switch ‘lz4’ with ‘zstd’ in the third line it should decrypt the zstd compressed mails only. Now that all your mails are decrypted you can go ahead and run the ‘doveadm fs put’ loop

        Ah, you don’t even need the extra if, as in line 4 the loop checks if doveadm created an output file

        So just run the original loop with lz4 first and afterwards use zstd, thats all

            DocFraggle you go above and beyond, thanks for the great support!
            It is indeed working :-)

              DocFraggle Joooo, that can be it actually. I recently added the lz4 support to the Alpine Linux to exclude this behaviour but i was not able to test it.

              I’ll do that today.

              If that is the resolution than i’m so happy!

              Thanks for the debugging, really really appreciate it!

                MichaelThwaite

                Good point, thx! I really thought I was safe with my normal mailcow backups until now. But fiddlesticks, the mails in the backups are all encrypted! I’ll have a look at your document and backup my mailboxes locally unencrypted as well. I’ll wait until arm64 is stable, only then will I dare to update from my current version. Thank you all for your efforts!

                  @DocFraggle yeah… i’m afraid it don’t work.

                  It still cannot decrypt the mails. Even with lz4 enabled. So there is something different, as your Debian 12 Setup seems to work. I’ll try that doveadm command now to see if there is something different.

                    DerLinkman could you please update the nightly branch with the new dovecot image? Then I can have a look

                        DerLinkman OK, I really don’t get it… the installed dovecot-core version on my Ubuntu 22.04 machine is 1:2.3.16+dfsg1-3ubuntu2.2 and is perfectly able to decrypt the files with LZ4.
                        The dovecot version inside the container is 2.3.21 (47349e2482) and can’t decrypt it…

                        I guess this is your build configuration for dovecot?

                        > cat /usr/lib/dovecot/dovecot-config 
DOVECOT_INSTALLED=yes
DOVECOT_CFLAGS="-std=gnu99 -Os -fstack-clash-protection -Wformat -Werror=format-security -fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2  "
DOVECOT_LIBS=""
DOVECOT_SSL_LIBS="-lssl -lcrypto"
DOVECOT_SQL_LIBS=""
DOVECOT_COMPRESS_LIBS=" -lz -lbz2 -llzma -llz4 -lzstd"
DOVECOT_LUA_LIBS="-L/usr/lib/lua5.3 -llua -lm"
DOVECOT_LUA_CFLAGS="-I/usr/include/lua5.3"
DOVECOT_BINARY_CFLAGS="-fPIE -DPIE"
DOVECOT_BINARY_LDFLAGS="-pie -Wl,-z -Wl,relro -Wl,-z -Wl,now"

LIBDOVECOT='-L/usr/lib/dovecot -ldovecot'
LIBDOVECOT_LOGIN='-ldovecot-login -lssl -lcrypto'
LIBDOVECOT_SQL=-ldovecot-sql
LIBDOVECOT_COMPRESS=-ldovecot-compression
LIBDOVECOT_LDA=-ldovecot-lda
LIBDOVECOT_STORAGE='-ldovecot-storage '
LIBDOVECOT_DSYNC=-ldovecot-dsync
LIBDOVECOT_LIBFTS=-ldovecot-fts
LIBDOVECOT_LUA=-ldovecot-lua


LIBDOVECOT_INCLUDE=-I/usr/include/dovecot

dovecot_pkgincludedir=/usr/include/dovecot
dovecot_pkglibdir=/usr/lib/dovecot
dovecot_pkglibexecdir=/usr/libexec/dovecot
dovecot_docdir=/usr/share/doc/dovecot
dovecot_moduledir=/usr/lib/dovecot
dovecot_statedir=/var/lib/dovecot

                        Do you know eventually how to compare the build options with the Ubuntu package? Maybe there is something still missing?

                          We use the Alpine Builds they build so that is not my build exactly 😃

                          However i try dovecot on Debian 12 which had the same issue in the past (lz4 aside 😃).

                          Debug output inside the container:

                          doveadm -D fs get compress lz4:1:crypt:private_key_path=/mail_crypt/ecprivkey.pem:public_key_path=/mail_crypt/ecpubkey.pem:posix:prefix=/ /var/vmail//var/vmail/asdökfjlaskdjfla/Maildir/.INBOX.!  asdfasdfasdf/cur/1649346412.M610300P59299.fb225c99c952\,S\=1948\,W\=2008\:2\,S 
Debug: Loading modules from directory: /usr/lib/dovecot/doveadm
Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: Error relocating /usr/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: acl_rights_update_import: symbol not found (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: Error relocating /usr/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so: quota_get_resource: symbol not found (this is usually intentional, so just ignore this message)
Debug: Module loaded: /usr/lib/dovecot/doveadm/lib10_doveadm_sieve_plugin.so
Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: Error relocating /usr/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: fts_backend_rescan: symbol not found (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: Error relocating /usr/lib/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: mail_crypt_box_get_public_key: symbol not found (this is usually intentional, so just ignore this message)
2023-10-16 11:18:33 Debug: Loading modules from directory: /usr/lib/dovecot
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/lib01_acl_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/lib05_mail_crypt_acl_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/lib10_mail_crypt_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/lib10_quota_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/lib15_notify_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/lib20_fts_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/lib20_listescape_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/lib20_mail_log_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/lib20_replication_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/lib20_zlib_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/lib21_fts_solr_plugin.so
2023-10-16 11:18:33 Debug: Loading modules from directory: /usr/lib/dovecot/doveadm
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so
2023-10-16 11:18:33 Error: read(/var/vmail/asdökfjlaskdjfla/Maildir/.INBOX.!  asdfasdfasdf/cur/1649346412.M610300P59299.fb225c99c952,S=1948,W=2008:2,S) failed: Decryption error: no private key available

                          Performed on the Ubuntu host system:

                          root@ubuntu-16gb-fsn1-1:/var/lib/docker/volumes/mailcowdockerized_vmail-vol-1/_data# doveadm -D fs get compress lz4:1:crypt:private_key_path=/var/lib/docker/volumes/mailcowdockerized_crypt-vol-1/_data/ecprivkey.pem:public_key_path=/var/lib/docker/volumes/mailcowdockerized_crypt-vol-1/_data/ecpubkey.pem:posix:prefix=./ asdfasdfasdf/asdfasdfasdf/Maildir/.INBOX.\!\ \ asdfasdfasdf/cur/1649346412.M610300P59299.fb225c99c952\,S\=1948\,W\=2008\:2\,S
Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm
Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_user_get_language_list (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message)
Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm
Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_user_get_language_list (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message)
Oct 16 09:19:41 Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm
Oct 16 09:19:41 Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message)
Oct 16 09:19:41 Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message)
Oct 16 09:19:41 Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message)
Oct 16 09:19:41 Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_user_get_language_list (this is usually intentional, so just ignore this message)
Oct 16 09:19:41 Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message)
From: "asdfasdf asdfasdfasdf" <asdfasdf@web.de>
To: "adsfasdf asdfasdf"
Subject: Muffins
...

                          [unknown] My home server is running on Debian 12 as well, with dovecot-core 1:2.3.19.1+dfsg1-2.1 it’s working as well for me

                          Okay… very interesting. I’ve just switched the Dovecot Image to Debian Bookworm as a base and now it can decrypt the mails even on ARM64 without issues. If you want to test @DocFraggle the image is called mailcow/dovecot:nightly-devel to confirm my changes.

                          Yes, I can confirm that doveadm can decompress and decrypt the mails now. Sogo isn’t able to login to dovecot anymore, but maybe something’s missing regarding with the bookworm image.

                            DocFraggle I’ve opened up a ticket at Alpines so let’s see what they say.

                            However i think we might compile dovecot ourself then on Debian 11/12 to ensure it’s still working as expected.

                              BTW, while debugging I read in the dovecot documentation that it is no problem to have Maildir files with different compression types, it will work out of the box. So no problem to switch the compression from LZ4 to zstd in the future, the problem was just the odd alpine dovecot image…

                              When this plugin is loaded Dovecot can read both compressed and uncompressed files from Maildir. The files within a Maildir can use any supported compression algorithm (e.g., some can be compressed using gzip, while others are compressed using zstd). The algorithm is detected by reading the first few bytes from the file and figuring out if it’s a valid gzip or bzip2 header. The file name doesn’t matter.

                              Zlib plugin — Dovecot documentation

                              doc.dovecot.org
                              Zlib plugin — Dovecot documentation
                              LogoCreative Commons License

                              But, of course, this should be tested thoroughly 😃