Cannot send and receive mails after VM move

Hunv

Hi, I am running the dockerized version of mailcow on Ubuntu 20 in a VM.
Yesterday I reinstalled my hypervisor and backuped/recovered the whole VM. During this process the name of the NIC inside of the VM changed. I corrected the NIC name to the correct name in the file /etc/netplan/00-installer-config.yaml and restarted the VM. After that I was able to access the VM via SSH. Before it was not working because the new primary adapter was in down state.
After the recovery, configure network and restarting the VM most things are working:

I can access my mailbox via mailclient
I can access my mailbox via SoGo
I can login to the admin website

But what I cannot do is to receive and send mails. In SoGo I get the message when sending a mail “cannot send message: (smtp) authentication failure”.
Outlook wants to have credentials.
As I said: I also don’t get any mails or can send mails to myself. What logs can I check? Anything I can test?

Hunv

After a short night I thing I found 1-2 hints in the postfix log.
When trying to send a Mail with SoGo, there are this log entries I get when executing

sudo docker-compose logs -f –tail=200 postfix-mailcow

mailcowdockerized-postfix-mailcow-1 | Jun 26 08:00:32 e12f9bbc03ae postfix/sogo/smtpd[3303]: connect from mailcowdockerized-sogo-mailcow-1.mailcowdockerized_mailcow-network[172.22.1.248] mailcowdockerized-postfix-mailcow-1 | Jun 26 08:00:32 e12f9bbc03ae postfix/sogo/smtpd[3303]: warning: host or service rspamd:9900 not found: Name or service not known mailcowdockerized-postfix-mailcow-1 | Jun 26 08:00:32 e12f9bbc03ae postfix/sogo/smtpd[3303]: warning: connect to Milter service inet:rspamd:9900: Cannot assign requested address mailcowdockerized-postfix-mailcow-1 | Jun 26 08:00:32 e12f9bbc03ae postfix/sogo/smtpd[3303]: NOQUEUE: milter-reject: CONNECT from mailcowdockerized-sogo-mailcow-1.mailcowdockerized_mailcow-network[172.22.1.248]: 451 4.7.1 Service unavailable - try again later; proto=SMTP mailcowdockerized-postfix-mailcow-1 | Jun 26 08:00:32 e12f9bbc03ae postfix/sogo/smtpd[3303]: NOQUEUE: milter-reject: EHLO from mailcowdockerized-sogo-mailcow-1.mailcowdockerized_mailcow-network[172.22.1.248]: 451 4.7.1 Service unavailable - try again later; proto=SMTP helo=<baec04defb79> mailcowdockerized-postfix-mailcow-1 | Jun 26 08:00:32 e12f9bbc03ae postfix/sogo/smtpd[3303]: disconnect from mailcowdockerized-sogo-mailcow-1.mailcowdockerized_mailcow-network[172.22.1.248] ehlo=1 auth=0/1 quit=1 commands=2/3

The first hint indicates that rspamd is not available.

warning: host or service rspamd:9900 not found: Name or service not known

The second hint seems to confirm this as if I google for that the error code 451 indicates that a spam mail is rejected to accept.

NOQUEUE: milter-reject: EHLO from mailcowdockerized-sogo-mailcow-1.mailcowdockerized_mailcow-network[172.22.1.248]: 451 4.7.1 Service unavailable - try again later; proto=SMTP

As a result, it looks like rspamd has a problem.

So I tried to run

sudo docker-compose logs -f –tail=200 rspam-mailcow

but without any output. Should this command work similar to the postfix command?
I also checked the UI logs page but this is also empty.

Looks for me like rspam is not running correct. The container itself is running and also restarting the container via GUI didn’t help.
How can I dive deeper into the rspam container for troubleshooting? How can I check if the container is reachable via network? Maybe it as a problem with the uplink mapping or the DNS name after the “new” network adapter for the VM?

PS:
Accessing the rspam UI doesn’t work: https://my.domain/rspamd/ after some seconds it shows
The steps in this image seems to be more general not related to rspam. Anyway I see no errors in the stated logs.

One addition I have to correct “I see no errors” in the log-command of error message of rspam UI.
I see this HTTP 502:

mailcowdockerized-nginx-mailcow-1 | 5.1.87.87 - - [26/Jun/2023:08:24:02 +0200] “GET /rspamd/ HTTP/1.0” 502 2038 “-” “GuzzleHttp/7”

Hunv

The story continues … 🙂
I found this article: https://docs.mailcow.email/de/troubleshooting/debug-common_problems/#rspamd-meldet-cannot-open-hyperscan-cache-file-varlibrspamdhs-compiled-for-a-different-platform
The pysical Hardware didn’t changed (was AMD and is stimm the same AMD CPU) but maybe this solves it anyway as the virtual hardware changed? The Problem is, that I cannot connect to the docker container as it tells mit it is restarting.

:/opt/mailcow-dockerized$ sudo docker-compose exec rspamd-mailcow bash
Error response from daemon: Container fc65855fa8076d2b09c1f663dbe6d73119b2635e738598ddcdc94f4a8cd39ea5 is restarting, wait until the container is running

Hunv

I found the next logs using

docker logs fc65855fa807 (Id of the rspam container)

It seems to be the issue with hyperscan mentioned at common problems. The only issue is that I cannot access the container to fix it the way as it is documented as the container won’t start…

2023-06-26 08:58:40 #1(main) <37072e>; re_cache; rspamd_re_cache_init: loaded hyperscan engine with cpu tune 'generic' and features ''
2023-06-26 08:58:40 #1(main) <37072e>; re_cache; rspamd_re_cache_load_hyperscan: fatal error: cannot allocate scratch for /var/lib/rspamd//791648bcd295379bd35146fd029ee28c88cfa0bd2429269cf09abc9541528fc8.hs: -6

Hunv

I figured out, that after a docker compose up -d the container is running for a couple of seconds. I executed the commands but without luck. It keeps crashing.

But, as a workaround, what did work was to downgrade the rspam-container to version 1.90. Version 1.91 and 1.92 have the problems I have, version 1.90 is running stable.
So there seems to be a change from 1.90 to 1.91 that causes the problem.
Where can I see what changed at this version? I only see the rspam-version in the changlog but not the container version.

Hunv

Any hints what can I do?

esackbauer

Maybe the same problem as here. Rspamds cache is strongly CPU dependent.
https://community.mailcow.email/d/2608-rspamd-container-stuck-in-restart-loop-after-updatesh

Hunv

After a quite long time I finally started to fix that issue… yes really late but it was running with the workaround.
What I did is:
When the rspam container was at version 1.90, I entered the rspam container and verified that the rspam service is stopped and cleared all the .hs files from /var/lib/rspamd:

sudo docker-compose exec rspamd-mailcow bash
service rspamd stop
service --status-all
rm /var/lib/rspamd/*.hs*

After that I ran the standard update script of Mailcow. Voilà it works.