ARM64 Version

Ganzjahresgriller · 2023-10-10T10:04:39+00:00

Is there anything new there? The last blog post is from April

asoprano

DocFraggle you go above and beyond, thanks for the great support!
It is indeed working :-)

DerLinkman

DocFraggle Joooo, that can be it actually. I recently added the lz4 support to the Alpine Linux to exclude this behaviour but i was not able to test it.

I’ll do that today.

If that is the resolution than i’m so happy!

Thanks for the debugging, really really appreciate it!

epistula

MichaelThwaite

Good point, thx! I really thought I was safe with my normal mailcow backups until now. But fiddlesticks, the mails in the backups are all encrypted! I’ll have a look at your document and backup my mailboxes locally unencrypted as well. I’ll wait until arm64 is stable, only then will I dare to update from my current version. Thank you all for your efforts!

DerLinkman

@DocFraggle yeah… i’m afraid it don’t work.

It still cannot decrypt the mails. Even with lz4 enabled. So there is something different, as your Debian 12 Setup seems to work. I’ll try that doveadm command now to see if there is something different.

DocFraggle

DerLinkman could you please update the nightly branch with the new dovecot image? Then I can have a look

DerLinkman

DocFraggle Is updated!

DocFraggle

DerLinkman OK, I really don’t get it… the installed dovecot-core version on my Ubuntu 22.04 machine is 1:2.3.16+dfsg1-3ubuntu2.2 and is perfectly able to decrypt the files with LZ4.
The dovecot version inside the container is 2.3.21 (47349e2482) and can’t decrypt it…

I guess this is your build configuration for dovecot?

> cat /usr/lib/dovecot/dovecot-config 
DOVECOT_INSTALLED=yes
DOVECOT_CFLAGS="-std=gnu99 -Os -fstack-clash-protection -Wformat -Werror=format-security -fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2  "
DOVECOT_LIBS=""
DOVECOT_SSL_LIBS="-lssl -lcrypto"
DOVECOT_SQL_LIBS=""
DOVECOT_COMPRESS_LIBS=" -lz -lbz2 -llzma -llz4 -lzstd"
DOVECOT_LUA_LIBS="-L/usr/lib/lua5.3 -llua -lm"
DOVECOT_LUA_CFLAGS="-I/usr/include/lua5.3"
DOVECOT_BINARY_CFLAGS="-fPIE -DPIE"
DOVECOT_BINARY_LDFLAGS="-pie -Wl,-z -Wl,relro -Wl,-z -Wl,now"

LIBDOVECOT='-L/usr/lib/dovecot -ldovecot'
LIBDOVECOT_LOGIN='-ldovecot-login -lssl -lcrypto'
LIBDOVECOT_SQL=-ldovecot-sql
LIBDOVECOT_COMPRESS=-ldovecot-compression
LIBDOVECOT_LDA=-ldovecot-lda
LIBDOVECOT_STORAGE='-ldovecot-storage '
LIBDOVECOT_DSYNC=-ldovecot-dsync
LIBDOVECOT_LIBFTS=-ldovecot-fts
LIBDOVECOT_LUA=-ldovecot-lua


LIBDOVECOT_INCLUDE=-I/usr/include/dovecot

dovecot_pkgincludedir=/usr/include/dovecot
dovecot_pkglibdir=/usr/lib/dovecot
dovecot_pkglibexecdir=/usr/libexec/dovecot
dovecot_docdir=/usr/share/doc/dovecot
dovecot_moduledir=/usr/lib/dovecot
dovecot_statedir=/var/lib/dovecot

Do you know eventually how to compare the build options with the Ubuntu package? Maybe there is something still missing?

DerLinkman

We use the Alpine Builds they build so that is not my build exactly 😃

However i try dovecot on Debian 12 which had the same issue in the past (lz4 aside 😃).

DocFraggle

Debug output inside the container:

doveadm -D fs get compress lz4:1:crypt:private_key_path=/mail_crypt/ecprivkey.pem:public_key_path=/mail_crypt/ecpubkey.pem:posix:prefix=/ /var/vmail//var/vmail/asdökfjlaskdjfla/Maildir/.INBOX.!  asdfasdfasdf/cur/1649346412.M610300P59299.fb225c99c952\,S\=1948\,W\=2008\:2\,S 
Debug: Loading modules from directory: /usr/lib/dovecot/doveadm
Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: Error relocating /usr/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: acl_rights_update_import: symbol not found (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: Error relocating /usr/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so: quota_get_resource: symbol not found (this is usually intentional, so just ignore this message)
Debug: Module loaded: /usr/lib/dovecot/doveadm/lib10_doveadm_sieve_plugin.so
Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: Error relocating /usr/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: fts_backend_rescan: symbol not found (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: Error relocating /usr/lib/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: mail_crypt_box_get_public_key: symbol not found (this is usually intentional, so just ignore this message)
2023-10-16 11:18:33 Debug: Loading modules from directory: /usr/lib/dovecot
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/lib01_acl_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/lib05_mail_crypt_acl_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/lib10_mail_crypt_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/lib10_quota_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/lib15_notify_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/lib20_fts_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/lib20_listescape_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/lib20_mail_log_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/lib20_replication_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/lib20_zlib_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/lib21_fts_solr_plugin.so
2023-10-16 11:18:33 Debug: Loading modules from directory: /usr/lib/dovecot/doveadm
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so
2023-10-16 11:18:33 Debug: Module loaded: /usr/lib/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so
2023-10-16 11:18:33 Error: read(/var/vmail/asdökfjlaskdjfla/Maildir/.INBOX.!  asdfasdfasdf/cur/1649346412.M610300P59299.fb225c99c952,S=1948,W=2008:2,S) failed: Decryption error: no private key available

Performed on the Ubuntu host system:

root@ubuntu-16gb-fsn1-1:/var/lib/docker/volumes/mailcowdockerized_vmail-vol-1/_data# doveadm -D fs get compress lz4:1:crypt:private_key_path=/var/lib/docker/volumes/mailcowdockerized_crypt-vol-1/_data/ecprivkey.pem:public_key_path=/var/lib/docker/volumes/mailcowdockerized_crypt-vol-1/_data/ecpubkey.pem:posix:prefix=./ asdfasdfasdf/asdfasdfasdf/Maildir/.INBOX.\!\ \ asdfasdfasdf/cur/1649346412.M610300P59299.fb225c99c952\,S\=1948\,W\=2008\:2\,S
Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm
Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_user_get_language_list (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message)
Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm
Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_user_get_language_list (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message)
Oct 16 09:19:41 Debug: Loading modules from directory: /usr/lib/dovecot/modules/doveadm
Oct 16 09:19:41 Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_acl_plugin.so: undefined symbol: acl_user_module (this is usually intentional, so just ignore this message)
Oct 16 09:19:41 Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib10_doveadm_quota_plugin.so: undefined symbol: quota_user_module (this is usually intentional, so just ignore this message)
Oct 16 09:19:41 Debug: Skipping module doveadm_fts_lucene_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_lucene_plugin.so: undefined symbol: lucene_index_iter_deinit (this is usually intentional, so just ignore this message)
Oct 16 09:19:41 Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/lib20_doveadm_fts_plugin.so: undefined symbol: fts_user_get_language_list (this is usually intentional, so just ignore this message)
Oct 16 09:19:41 Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: /usr/lib/dovecot/modules/doveadm/libdoveadm_mail_crypt_plugin.so: undefined symbol: mail_crypt_box_get_pvt_digests (this is usually intentional, so just ignore this message)
From: "asdfasdf asdfasdfasdf" <asdfasdf@web.de>
To: "adsfasdf asdfasdf"
Subject: Muffins
...

[unknown] My home server is running on Debian 12 as well, with dovecot-core 1:2.3.19.1+dfsg1-2.1 it’s working as well for me

DerLinkman

Okay… very interesting. I’ve just switched the Dovecot Image to Debian Bookworm as a base and now it can decrypt the mails even on ARM64 without issues. If you want to test @DocFraggle the image is called mailcow/dovecot:nightly-devel to confirm my changes.

DocFraggle

Yes, I can confirm that doveadm can decompress and decrypt the mails now. Sogo isn’t able to login to dovecot anymore, but maybe something’s missing regarding with the bookworm image.

DerLinkman

DocFraggle I’ve opened up a ticket at Alpines so let’s see what they say.

However i think we might compile dovecot ourself then on Debian 11/12 to ensure it’s still working as expected.

DocFraggle

BTW, while debugging I read in the dovecot documentation that it is no problem to have Maildir files with different compression types, it will work out of the box. So no problem to switch the compression from LZ4 to zstd in the future, the problem was just the odd alpine dovecot image…

When this plugin is loaded Dovecot can read both compressed and uncompressed files from Maildir. The files within a Maildir can use any supported compression algorithm (e.g., some can be compressed using gzip, while others are compressed using zstd). The algorithm is detected by reading the first few bytes from the file and figuring out if it’s a valid gzip or bzip2 header. The file name doesn’t matter.

https://doc.dovecot.org/configuration_manual/zlib_plugin/

But, of course, this should be tested thoroughly 😃

DerLinkman

DocFraggle Yeah exactly i remembered this so i switched it back then as Alpine had no lz4 support during the inital tests. Only zstd got added after my request first.

However it is related something which has changed from Alpine 3.16 to 3.18.

DerLinkman

I just found something very, very, very, very interesting:

https://salsa.debian.org/debian/dovecot/-/blob/master/debian/patches/Support-openssl-3.0.patch

This is a patch to support openssl 3.0 from last year. At this time Debian 12 was in development which ships OpenSSL3.0 and a running Repo Installation of Dovecot.

DocFraggle

DerLinkman but it seems that it was fixed in the 2.3.19 sources according to

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=996273

And the version in Alpine is 2.3.21

Ah it was fixed in the DEBIAN source package, not the dovecot sources. Yes, worth a try!

DerLinkman

DocFraggle Yes but only for debian! Debian creates their own patches.

You can actually see that the Dovecot devs are/were working at this too here:

https://github.com/dovecot/core/commits/main/src/lib-dcrypt/dcrypt-openssl3.c

In the current stable versions of Dovecot there is only one dcrypt-openssl.c file not two like in the main (dev) branch.

(See https://github.com/dovecot/core/tree/release-2.3.21/src/lib-dcrypt and https://github.com/dovecot/core/tree/main/src/lib-dcrypt)

Ubuntu also make their own Patches so that is also the reason why it is working with the Ubuntu Repo packages.

DocFraggle

Well, I thought I could quickly try to build an alpine dovecot image with the dovecot main branch, but I would need an Alpine Linux OS to do that 😄 that’s too much for the time I have to spare currently 🙂

Anyways, is there a specific reason why you want to switch dovecot from the Debian image over to the Alpine image?

DerLinkman

DocFraggle Yes indeed. Alpine is supporting ARM64 better then Debian at least for Dovecot. Yes their repo version is also ARM64 compatible but it makes the implementation for newer features harder as they don’t the latest versions.

With Dovecot 2.4 (or the current master state in git) there will change a lot which is causing mailcow to not even boot up. So yeah that’s a bit shitty. It has to be done in the future…

DerLinkman

Guys! Good news! My approach to Alpine has been submitted, they merged the patch!

https://gitlab.alpinelinux.org/alpine/aports/-/commit/815bb154f5337786fa025d420a0399696dfc0149

Now it only have to work 🤞

It works indeed! mailcow Team 🤜 🤛 Alpine Team

DocFraggle

DerLinkman Great news 🥳 I just deleted my ARM VM yesterday 🙃 but you already tested it I assume from your last line above 😀

« Previous Page Next Page »