ARM64 Version

Ganzjahresgriller · 2023-10-10T10:04:39+00:00

Is there anything new there? The last blog post is from April

DocFraggle

Maybe the problem is the different compression used in armcow? The current setup uses lz4, armcow uses zstd. So when the armcow dovecot tries to use zstd to decompress a lz4-compressed file, that can’t work.

DerLinkman

DocFraggle Joooo, that can be it actually. I recently added the lz4 support to the Alpine Linux to exclude this behaviour but i was not able to test it.

I’ll do that today.

If that is the resolution than i’m so happy!

Thanks for the debugging, really really appreciate it!

MichaelThwaite

Just as a datapoint, I installed on Ubuntu 22.04 Minimal AARM64 from the nightly and I can’t fault it, it’s working superbly! Thanks.

MPT

DocFraggle

But you didn’t migrate from an existing installation, did you?
@DerLinkman wrote that you aren’t able to access old emails (I assume he meant emails created with the current mailcow version), and I think that’s because of the LZ4 compression used in the current mailcow vs. the zstd compression used in armcow

MichaelThwaite

That’s right, completely clean install. I felt it worth adding the data point as, whilst there’s a call for errors to be reported, I couldn’t find a repository for reports of successful installs that can be cross referenced when looking for patterns in issues.

epistula

A question of understanding about all the branches, release and co. In this regard I am honestly not quite fit.

I am currently still using Mailcow 2023-05a aarch64 without any problems on my mail server. With the upgrade to a new version I had waited until arm64 becomes stable.

Since this will take a while, what is the recommendation to keep the server running on the arm64 version? How do I switch during operation, for example, to the nightly version. Or let 05a run until a stable version exists?

Thanks for your help!

DocFraggle

Anybody around who ran into the problem mentioned in the blog post being unable to read pre-armcow generated emails?
I bet that changing the compression from zstd to lz4 in your dovecot config file will result in being able to read the old mails again, but you can’t read the new mails anymore 🙂

https://github.com/mailcow/mailcow-dockerized/blob/feat/arm64-cow/data/conf/dovecot/dovecot.conf#L245

MichaelThwaite

@epistula I’ll be regularly exporting all mailboxes as unencrypted Mbox files locally - I’ve only six mailboxes to worry about - just in case!

Here’s the project for IMAP Tools:
https://github.com/croyleje/imap-tools/blob/master/IMAP_Tools_User_Guide.pdf

epistula

MichaelThwaite

Good point, thx! I really thought I was safe with my normal mailcow backups until now. But fiddlesticks, the mails in the backups are all encrypted! I’ll have a look at your document and backup my mailboxes locally unencrypted as well. I’ll wait until arm64 is stable, only then will I dare to update from my current version. Thank you all for your efforts!

DocFraggle

Btw, this is the commit which changed the compression from lz4 zo zstd in August

mailcow/mailcow-dockerizedc134f0b

I guess this is why you “successfully tested the migration of old e-mails in May”

DocFraggle

OK, I was really restless because of this issue, so I installed a new ARM server in my Hetzner cloud. I setup a new nightly mailcow installation and created one of my live domains with a mailbox.
Then I imported the crypt and vmail backup from my current mailcow, logged into Sogo and voila, dovecot complained about not being able to decrypt the mails

I tried to set the compression of dovecot to LZ4, but unfortunately it wasn’t compiled wit LZ4 support…

Error: zlib_save: Support not compiled in for handler: lz4

So I went along and used the doveadm commands from https://docs.mailcow.email/manual-guides/Dovecot/u_e-dovecot-mail-crypt/ to decrypt the mails in my mailbox, and then encrypted them again. (I had to remove the regex parameters as the busybox find doesn’t regognize them)

Result:

I guess it would be much easier to use a dovecot image compiled with LZ4, otherwise all mailcow admins will have to do the above decrypting and recrypting if zstd is used. And that could take a very long time if you have GBs of mails…

asoprano

I actually ran into this problem and I’m still trying to figuring it out.

I unfortunatelly switched from the arm branch to the nightly branch a month ago and did non see that I was actually not able to open mails prior to the switch. Noticed it some days ago prior the blog post.

I have created a new arm machine on hetzner, installed the nightly build and restored a backup from my main maschine. After that I tried the decrypt but I’m getting:
Fatal: fs_init() failed: compress: Compression method ‘lz4’ not supported.

Any hint?

DocFraggle

Ah sorry, I forgot to mention that I had to install the ‘dovecot-common’ package in the host system and perform the decryption directly in the docker vmail volume under /var/docker/volumes/ because lz4 wasn’t compiled in the docker dovecot image. You have to adjust the path of the find command accordingly!

But if you have mixed compression mails in your maildir I think you have to be careful and ONLY do this for the LZ4 compressed mails!

But if you have mails with mixed compression in your maildir I think you have to be careful and ONLY do this for the LZ4 compressed mails!

asoprano

DocFraggle

Thanks for the hint as indeed this is now running. And yes it is a problem that I have mixed compressions in my maildir, and the script is noticing it ;-)
Error: read(/xy/Maildir/.Archive/cur/1695363028.M982272P210299.2a3711cb79d9,S=8715,W=8852:2,S) failed: Decryption error: no private key available

In regards to your warning: As I’m doing it at the moment on a test instance that is fine. For my live machine I’m going at the moment a different route
1) I used imapsync to sync the mailboxes to an x86 instance and imapsync skipped the once that can not be fetched due not being able to decrypt
2) I will decrypt all the mails on the test instance and use also imapsync to push them to the x86 instance
This should give me at the end a full mail inbox

Well such a mess, but hey they have warned not to use it productively, so it kinda my problem i am trying to solve!

Thanks for your investigation and sharing it!

DocFraggle

/var/lib/docker/volumes/mailcowdockerized_vmail-vol-1/_data is the directory

DocFraggle

asoprano you could wrap the doveadm command in another if, so that it only continues when the decryption worked with LZ4 compression. Something like

find /var/vmail/ -type f -regextype egrep -regex '.*S=.*W=.*' | while read -r file; do
if [[ $(head -c7 "$file") == "CRYPTED" ]]; then
  if doveadm fs get compress lz4:1:crypt:private_key_path=/mail_crypt/ecprivkey.pem:public_key_path=/mail_crypt/ecpubkey.pem:posix:prefix=/ \
    "$file" > "/tmp/$(basename "$file")"; then
    if [[ -s "/tmp/$(basename "$file")" ]]; then
      chmod 600 "/tmp/$(basename "$file")"
      chown 5000:5000 "/tmp/$(basename "$file")"
      mv "/tmp/$(basename "$file")" "$file"
    else
      rm "/tmp/$(basename "$file")"
    fi
  fi
fi
done

This should decrypt all LZ4 compressed mails. If you then switch ‘lz4’ with ‘zstd’ in the third line it should decrypt the zstd compressed mails only. Now that all your mails are decrypted you can go ahead and run the ‘doveadm fs put’ loop

Ah, you don’t even need the extra if, as in line 4 the loop checks if doveadm created an output file

So just run the original loop with lz4 first and afterwards use zstd, thats all

asoprano

DocFraggle you go above and beyond, thanks for the great support!
It is indeed working :-)

DerLinkman

@DocFraggle yeah… i’m afraid it don’t work.

It still cannot decrypt the mails. Even with lz4 enabled. So there is something different, as your Debian 12 Setup seems to work. I’ll try that doveadm command now to see if there is something different.

DocFraggle

DerLinkman could you please update the nightly branch with the new dovecot image? Then I can have a look

DerLinkman

DocFraggle Is updated!

DocFraggle

DerLinkman OK, I really don’t get it… the installed dovecot-core version on my Ubuntu 22.04 machine is 1:2.3.16+dfsg1-3ubuntu2.2 and is perfectly able to decrypt the files with LZ4.
The dovecot version inside the container is 2.3.21 (47349e2482) and can’t decrypt it…

I guess this is your build configuration for dovecot?

> cat /usr/lib/dovecot/dovecot-config 
DOVECOT_INSTALLED=yes
DOVECOT_CFLAGS="-std=gnu99 -Os -fstack-clash-protection -Wformat -Werror=format-security -fstack-protector-strong -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -Wall -W -Wmissing-prototypes -Wmissing-declarations -Wpointer-arith -Wchar-subscripts -Wformat=2 -Wbad-function-cast -fno-builtin-strftime -Wstrict-aliasing=2  "
DOVECOT_LIBS=""
DOVECOT_SSL_LIBS="-lssl -lcrypto"
DOVECOT_SQL_LIBS=""
DOVECOT_COMPRESS_LIBS=" -lz -lbz2 -llzma -llz4 -lzstd"
DOVECOT_LUA_LIBS="-L/usr/lib/lua5.3 -llua -lm"
DOVECOT_LUA_CFLAGS="-I/usr/include/lua5.3"
DOVECOT_BINARY_CFLAGS="-fPIE -DPIE"
DOVECOT_BINARY_LDFLAGS="-pie -Wl,-z -Wl,relro -Wl,-z -Wl,now"

LIBDOVECOT='-L/usr/lib/dovecot -ldovecot'
LIBDOVECOT_LOGIN='-ldovecot-login -lssl -lcrypto'
LIBDOVECOT_SQL=-ldovecot-sql
LIBDOVECOT_COMPRESS=-ldovecot-compression
LIBDOVECOT_LDA=-ldovecot-lda
LIBDOVECOT_STORAGE='-ldovecot-storage '
LIBDOVECOT_DSYNC=-ldovecot-dsync
LIBDOVECOT_LIBFTS=-ldovecot-fts
LIBDOVECOT_LUA=-ldovecot-lua


LIBDOVECOT_INCLUDE=-I/usr/include/dovecot

dovecot_pkgincludedir=/usr/include/dovecot
dovecot_pkglibdir=/usr/lib/dovecot
dovecot_pkglibexecdir=/usr/libexec/dovecot
dovecot_docdir=/usr/share/doc/dovecot
dovecot_moduledir=/usr/lib/dovecot
dovecot_statedir=/var/lib/dovecot

Do you know eventually how to compare the build options with the Ubuntu package? Maybe there is something still missing?