Issues after updating

mendozal

Hi.
I updated from nightly to stable on an ARM system and after the update I’m getting a “What is happening?” screen.

Please do not stop the stack while we are initializing the database or do other preparations.

What is happening? - Nginx cannot connect to an upstream server or other services are not ready yet.
This is fine if mailcow was just installed or updated and can take a few minutes to complete.
Please check the logs or contact support if the error persists.

I tried connecting to the Mysql server using the suggested command using the credentials in mailcow.conf and that’s working fine.

These are a few of the latest logs:

mailcowdockerized-nginx-mailcow-1    | 2024/01/17 11:20:46 [error] 24#24: *561 no live upstreams while connecting to upstream, client: 127.0.0.1, server: mail.example.com, request: "GET /sogo-auth HTTP/1.0", upstream: "fastcgi://phpfpm", host: "mail.example.com"
mailcowdockerized-nginx-mailcow-1    | 2024/01/17 11:20:46 [error] 24#24: *559 auth request unexpected status: 502 while sending to client, client: 181.94.224.91, server: mail.example.com, request: "POST /Microsoft-Server-ActiveSync?Cmd=Ping&User=admin%40example2.com&DeviceId=androidc285787666&DeviceType=Android HTTP/1.1", host: "mail.example.com"
mailcowdockerized-nginx-mailcow-1    | 127.0.0.1 - admin@example2.com [17/Jan/2024:11:20:46 -0300] "GET /sogo-auth HTTP/1.0" 502 2015 "-" "Android-Mail/2023.11.26.586591930.Release"
mailcowdockerized-nginx-mailcow-1    | 181.94.224.91 - admin@example2.com [17/Jan/2024:11:20:46 -0300] "POST /Microsoft-Server-ActiveSync?Cmd=Ping&User=admin%40example2.com&DeviceId=androidc285787666&DeviceType=Android HTTP/1.1" 500 170 "-" "Android-Mail/2023.11.26.586591930.Release"
mailcowdockerized-nginx-mailcow-1    | 172.22.1.13 - - [17/Jan/2024:11:20:56 -0300] "GET / HTTP/1.1" 200 15 "-" "check_http/v (nagios-plugins 2.4.5)"
mailcowdockerized-nginx-mailcow-1    | 2024/01/17 11:21:08 [error] 23#23: *563 connect() failed (111: Connection refused) while connecting to upstream, client: 181.94.224.91, server: mail.example.com, request: "GET / HTTP/2.0", upstream: "fastcgi://[fd4d:6169:6c63:6f77::9]:9002", host: "mail.example.com"
mailcowdockerized-nginx-mailcow-1    | 2024/01/17 11:21:08 [error] 23#23: *563 connect() failed (111: Connection refused) while connecting to upstream, client: 181.94.224.91, server: mail.example.com, request: "GET / HTTP/2.0", upstream: "fastcgi://172.22.1.6:9002", host: "mail.example.com"
mailcowdockerized-nginx-mailcow-1    | 181.94.224.91 - - [17/Jan/2024:11:21:08 -0300] "GET / HTTP/2.0" 502 2015 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"

I tried docker compose down / up -d. And even reapplying the update.

Any ideas on what could be happening?

mendozal

Since it was not working I tried uninstalling everything, volumes, even docker and started from scratch.

After installing again from the master branch, the installation succeeded but the acme-mailcow container got stuck in “Waiting for Nginx”, no matter what I did. I tried this several times.

Finally, I decided to update that new installation to the nightly branch again, and it started to work. The mailcow-acme container started generating certificates.

After it was all ok, I restored the backup I made previous to the update and I recover everything

So, is there a path or some extra steps I need to follow in order to go from the nightly to the stable version?

Any insight is appreciated.

Thanks.

PS: I’m running this on an ARM cpu, 8Gb ram

mendozal

Someone on the telegram community told me they had removed the “search” option from the resolve.conf file inside the php-fpm contaner.

I can confirm that it works, after removing that the server seems to start working.

Also, after removing the same line from the acme container, it started to work.

The only issue is that after restarting the containers this setting adds itself again and everything stops working.

CrAazZyMaN21

mendozal
+1 Fixed it for me. Try to remove the search option from the docker host

onclefranck

Got the exact same issue and went to the nightly branch as well. The server started working back normally again. Thanks for sharing this information.

Question 1: what is the nightly branch and how is it maintained?
Question 2: you mentioned the search option can be disable from resolve,conf file in php-fpm… can you be more specific about the location of this file?

bitbrother

Using the older Version of the php-fpm Container Image also works.

Open docker-compose.yml
Search for “mailcow/phpfpm:1.86”
Replace with “mailcow/phpfpm:1.85”
Restart Stack: docker compose down && docker compose up -d

Dunno if mailcow replaces it automatically on next update, so make sure to revert changes before performing your next update.

jacotec

bitbrother You definitely saved my day! Just tried this after my server broke with the 2024-01a update, although it’s AMD64 and not ARM … that brought it back.

Recovering from heart-attack-mode now.

@[deleted] Maybe the broken 1.86 php-fpm image update should be pulled before this goes really big?

chev6900

bitbrother Thank you! this literally worked for me! 😄

DerLinkman

Hello, replacing the image is a option. However we would recommend doing the following:

Check if you have a entry called search within the /etc/resolv.conf file.
If so: remove it.
Restart mailcow
Should work now!

It is a nasty bug we cannot reproduce on every machine but which has been fixed already atleast inside the root cause (see: c-ares/c-aresissues/683) and needs to be released to Alpine as well.

However we cannot simply “downgrade” the containers as they need the newer Alpine Versions were this bug exists in.

stefan21

DerLinkman Check if you have a entry called search within the /etc/resolv.conf file.

Probably not everybody is happy (or can use) with this workaround:

Googled (but known):

"From man resolv.conf:

The search list is normally determined from the local domain name; by default, it contains only the local domain name. This may be changed by listing the desired domain search path following the search keyword with spaces or tabs separating the names. Resolver queries having fewer than ndots dots (default is 1) in them will be attempted using each component of the search path in turn until a match is found. For environments with multiple subdomains please read options ndots:n below to avoid man-in-the-middle attacks and unnecessary traffic for the root-dns-servers. Note that this process may be slow and will generate a lot of network traffic if the servers for the listed domains are not local, and that queries will time out if no server is available for one of the domains.

The search list is currently limited to six domains with a total of 256 characters."

This is helpful if you are part of a LAN environment that has its own DNS server so that people can type simple names to reach local resources. If you don’t have a home or corporate LAN with your own DNS server it’s nothing you really have to worry about.

I assume, in companies there are a number of mailcows in use with own DNS servers. In future IMVHO this should be fixed in a proper/better way.

esackbauer

stefan21
Does that apply actually?
Which internal services of mailcow need to make use of additional configured search suffixes? I am not aware of any.

stefan21

cat /etc/debian_version
12.4

uname -r
6.1.0-17-amd64

cat /etc/apt/sources.list
##deb cdrom:[Debian GNU/Linux 12.2.0 Bookworm - Official amd64 NETINST with firmware 20231007-10:28]/ bookworm main non-free-firmware

deb http://ftp-stud.hs-esslingen.de/debian/ bookworm main non-free-firmware
deb-src http://ftp-stud.hs-esslingen.de/debian/ bookworm main non-free-firmware

deb http://security.debian.org/debian-security bookworm-security main non-free-firmware
deb-src http://security.debian.org/debian-security bookworm-security main non-free-firmware

##bookworm-updates, to get updates before a point release is made;
##see https://www.debian.org/doc/manuals/debian-reference/ch02.en.html#_updates_and_backports
deb http://ftp-stud.hs-esslingen.de/debian/ bookworm-updates main non-free-firmware
deb-src http://ftp-stud.hs-esslingen.de/debian/ bookworm-updates main non-free-firmware

cat /etc/apt/sources.list.d/docker.list
deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/debian bookworm stable

OS system is up-to-date.

esackbauer Does that apply actually?

Updating from described above mailcow 2023-12a to 2024-01a will end in the bug the OP described.

esackbauer Which internal services of mailcow need to make use of additional configured search suffixes? I am not aware of any.

Hmm, the resolv.conf is AFAIK system relevant for the OS. This file is not from mailcow. I’d like to stay as much as possible to the stock of debian. Just my thoughts.

esackbauer

stefan21 the resolv.conf is AFAIK system relevant for the OS.

According to what the user mendozal has written, its not about the resolv.conf in the OS, but the one inside the php-fpm container. So it should have no side effects to the host OS or other containers.

jacotec

esackbauer As far as I understand @DerLinkman we talk about the /etc/resolv.conf of the host, not a container.

esackbauer

jacotec
No, it is about a bug in Alpine Linux and that is inside the containers.

DerLinkman

esackbauer Yes and no.

Yes it is inside Alpine Images using curl (partially) but it is influenced by the hosts resolv.conf.

stefan21

I misunderstood also. I’ll have a closer look into.
Anyway - after the latest update (as I described above) the gui is not reachable.

mrhase0815

Thank you smart people for giving all the information needed to fix this until an update is there.
What i did:
docker exec -it mailcowdockerized-php-fpm-mailcow-1 /bin/bash

vi /etc/resolv.conf
-> Remove “search” entry

docker exec -it mailcowdockerized-php-fpm-mailcow-1 kill -USR2 1

stefan21

DerLinkman Yes it is inside Alpine Images using curl (partially) but it is influenced by the hosts resolv.conf

Uncomment (or delete) the search line in the underlying OS in /etc/resolv.conf
docker exec -it mailcowdockerized-php-fpm-mailcow-1 /bin/bash
delete in /etc/resolv.conf the search line (and save) and exit
docker exec -it mailcowdockerized-php-fpm-mailcow-1 kill -USR2 1
docker compose restart php-fpm-mailcow
check if the workaround survived

Otherwise the workaround will not survive if mailcow is restarted with docker.

Personally I really don’t like to change files in the underlying OS while running docker. At a time this will end in errors.

mendozal

Updated to 2024-01b and all is working fine.
So this issue seems to be resolved.