After upgrade, Waiting for database to come up...

SpecCZ

Hi,

Please, could someone help?
After upgrade in postfix log i have repeating message:
Waiting for database to come up…

This is working and I can connect to DB, DB is up:
source mailcow.conf ; docker compose exec mysql-mailcow mysql -u${DBUSER} -p${DBPASS} ${DBNAME}

Any ideas?

Thank you.

SpecCZ

Also when i tried new fresh installation of Mailcow, same problem. All other applications are waiting for DB from beginning and are not even able to create initial tables in DB.

Any ides, please?

esackbauer

Double check that you run Mailcow on a supported platform, and that you meet all requirements.
https://docs.mailcow.email/getstarted/prerequisite-system/

Also please check the logs (not only of mariadb).

poandlsl

@SpecCZ I experienced a similar issue after installing package updates on a Rocky 9 host. I ended up reverting the following packages to the listed versions:

containerd.io              1.7.26-3.1.el9.x86_64
docker-buildx-plugin       0.22.0-1.el9.x86_64
docker-ce                  3:28.0.4-1.el9.x86_64
docker-ce-cli              1:28.0.4-1.el9.x86_64
docker-ce-rootless-extras  28.0.4-1.el9.x86_64
docker-compose-plugin      2.34.0-1.el9.x86_64

After reverting, the containers came back up and were able to communicate with the DB container. I didn’t spend too much time trying to figure out what the issue was, but I suspect that something changed with container networking and that the containers were unable to communicate with each other.

SpecCZ

Thank you @poandlsl for your post.

I also tested same on other servers with Rocky 9 (some with exactly same distribution) and it was worked fine. One server only was impacted. This is very strange.

On that impacted server I tried more Rocky 9 distributions and no one worked with latest MailCow. Finally Rocky 8.10 is working on that server with latest MailCow.

Just for info, this is really strange behaviour impacting just some combination of HW + Linux + MailCow.

SpecCZ

Update: Same problem on my another server (same Rocky 9.5 distribution).

Update of Linux packages with previous Mailcow - no problem.
When I updated Mailcow after that - problem was there.

Root cause is somewhere between latest docker Linux packages and latest Mailcow.

Fixed problem by rollback below Linux packaged to listed versions.

docker-buildx-plugin-0.22.0-1.el9.x86_64
docker-compose-plugin-2.34.0-1.el9.x86_64
docker-ce-cli-28.0.4-1.el9.x86_64
docker-ce-rootless-extras-28.0.4-1.el9.x86_64
docker-ce-28.0.4-1.el9.x86_64

RaVoR

I had the same problem on a rocky linux box. Unfortunately I was not able to solve it by reverting to a previous version of docker, so I had to dig a bit.

As my borgmatic backup failed as well and borgmatic is a lot more verbose, I was able to pinpoint the problem to access of socket files in container-volumes if selinux is enabled. It looks like somehow selinux got into the mix here. For now, mailcow is using the mysqld socket in “mysql-socket-vol” volume. For some reason, I do not fully understand yet, it is not longer possible for other containers to access the “mysql-socket-vol” although it gets mounted correctly. But as soon as the container apps try to access it, selinux will deny access and a wild “permission denied” appears. After switching selinux into permissive mode, everything worked again. This confirmed my theory about SELinux. In the quest of finding some valuable information about the issue, i found a nice blog article from RedHat (which is the base for rocky): https://www.redhat.com/en/blog/why-you-should-be-using-multi-category-security-your-linux-containers

This explains quite well, why the problem happens. So sum it up:
Each container gets its own security labels (MCS/MLS labels) and by default SELinux prevents any access to socket_files created by other containers. From a file type perspective SELinux allows the access to the socket because container_t and container_file_t have a corresponding access vector rule. From a security label perspective, SELinux (or the MCS/MLS part) will deny access to it.

So thats for the cause, but how do I allow access to the socket without disabling SELinux as a security boundary?

Edit for the sake of being a completionist:
Here’s the actual output of the SELinux audit:

[root@mail mailcow-dockerized]# ausearch -m avc -ts recent | audit2why
type=AVC msg=audit(1746602018.224:4302790): avc:  denied  { write } for  pid=319379 comm="check_mysql" name="mysqld.sock" dev="dm-8" ino=541152109 scontext=system_u:system_r:container_t:s0:c97,c180 tcontext=system_u:object_r:container_file_t:s0:c400,c869 tclass=sock_file permissive=1

        Was caused by:

#Constraint rule:

#       mlsconstrain sock_file { ioctl read getattr } ((h1 dom h2 -Fail-)  or (t1 != mcs_constrained_type -Fail-) ); Constraint DENIED
mlsconstrain sock_file { write setattr } ((h1 dom h2 -Fail-)  or (t1 != mcs_constrained_type -Fail-) ); Constraint DENIED
mlsconstrain sock_file { relabelfrom } ((h1 dom h2 -Fail-)  or (t1 != mcs_constrained_type -Fail-) ); Constraint DENIED
mlsconstrain sock_file { create relabelto } ((h1 dom h2 -Fail-)  or (t1 != mcs_constrained_type -Fail-) ); Constraint DENIED

#       Possible cause is the source level (s0:c97,c180) and target level (s0:c400,c869) are different.

SpecCZ

RaVoR
Thank you for nice post!

So, probably better to set SELinux as “disabled” and remove its support form docker.

DocFraggle

That’s the reason why I despise of selinux, it always fcks things up and leaves you in debugging hell…

RaVoR

DocFraggle I really like the additional security gained by SELinux, therefore I want to use it. Especially in a service, that is publicly available like mail servers. Anybody any ideas?

esackbauer

RaVoR I really like the additional security gained by SELinux

What exactly are you gaining?

RaVoR

esackbauer I don’t want to be offensive, but SELinux is there for a reason. Correctly configured, SELinux prevents different kind of attack vectors, from writing to wrong directories to preventing allocation of resources. There’s a reason a whole set of rules for SELinux and containers exist and is actively maintained. I cannot rely on software to be bug-free, this includes docker as well. We’re talking about a public facing internet service, so each part of additional security is another brick in the wall against malicious programs/persons. But that’s just me being a little paranoid. 🙂

SpecCZ I would set it to “permissive”, so you can enable it (without reboot), after I or somebody else fixed that issue. Permissive is basically disabling BUT with the logs being still there. If someone comes up with a solution, it is directly visible if it works for you or not.

I’ve found an open bug for this issue in github: mailcow/mailcow-dockerized6510
So the digging was already done by someone else. 🙁

esackbauer

RaVoR
I do it the other way around, disable SELinux, but use a hardened Web Application Firewall (Sophos Firewall with Intrusion Prevention) with additional blocklists from abuseipdb.com. For me that is less complicated, but even higher degree of security. But everyone has a different approach to security (and a different risk exposure), but at least I have no unsupported mailcow config 😉

RaVoR

esackbauer Officially SELinux is supported, so not directly running a unsupported config. If the past hast proven one thing, then that a WAF and/or AV solutions only create more attack-surface instead of minimizing it. But this is more philosophical discussion. Meanwhile I found the solution and will provide it to others. 🙂

mlcwuser

esackbauer I do it the other way around, disable SELinux

I use Debian, so no need to disable SELinux 😉

However, I think @RaVoR is right that WAFs are generally oversold, which can lead to dangerous conclusions, especially with SOHO users who then expose everything to the Internet because they see all the marketing buzz, and then think “it’s behind a reverse proxy with SSL and WAF, so it’s automatically secure”. I’d also say he’s right that SELinux can greatly improve the overall security of a system – if you know how to use it.

Anyway, I don’t think a WAF vs. SELinux discussion makes much sense, as the two things mostly do completely different things, so companies with low risk tolerance and high security requirements will probably want to use both, while home users like me use neither. 😉

esackbauer

RaVoR Officially SELinux is supported,

You are right:
https://docs.mailcow.email/getstarted/install/#check-selinux-specifics

RaVoR If the past hast proven one thing, then that a WAF and/or AV solutions only create more attack-surface instead of minimizing it.

SCNR:
Where is that proof? If there is proof or studies then there is no need for philosophy.
Still both NIST and BSI recommend the use of WAFs.

RaVoR

esackbauer
Was waiting for your response 🙂
Here are some interesting CVEs / Articles pointing to problems in AV software:

Old but gold: https://lock.cmpxchg8b.com/sophail.pdf
Bring-your-own-vulnerable-driver: https://dig.watch/updates/cyberattack-exploits-a-flaw-in-zonealarms-vsdatant-sys-driver
Privilege Escalation in AV products: CVE-2024-5102, NLOKSA1516, CVE-2022-4294, CVE-2022-4173, CVE-2024-23764, CVE-2023-47172, CVE-2022-28877
Denial-of-service: CVE-2022-4429, SYMSA1058, various flaws in F-Secure (just to mention one)
Code Execution: SYMSA1023, SYMSA1037

That’s just a short list, I came up with searching for about 20 Minutes. I did not specifically look for WAFs for now, but again a WAF only works well, if it knows how the application behind it, works exactly. If only using the default ruleset, you’re basically relying on a very thin security layer. This default ruleset normally only protects you from malformed input, maybe XSS attacks. If there’s something in the application using a known path and/or real vulnerability in the app using correctly formatted data, a WAF will not protect you as it cannot really detect if there’s something fishy going on. That’s where SELinux and other forms of protection come into play. Even if your application gets compromised, SELinux may prevent bad things from happening. I’m the better safe than sorry kind of guy, so having the chance of protection against no protection at all, has to be favored. If you’re using sophisticated rules specially crafted for mailcow, then everything is great, but be honest, most companies and administrators just use the out-of-the-box rulesets. I’m a network and network security consultant, so I’ve seen a lot of software, which outright lies to its’ customers and promises protection it cannot provide without supervision, definition/analysis of attack vectors and mitigation rules.

Edit: And for fun and profit ->Stop Disabling SELinux

esackbauer

I thought right that you come up with such a list (without WAF…) showing your confirmation bias 😉
That is just some wild guessing, and says nothing about the positive effects of running a patched and well maintained WAF. But you need to consider those effects, obviously, any consultant should be able to.
There is a reason why NIST and BSI see the benefits outweigh the risks…

RaVoR

If you read the wall of text, I’ve written, I’m not against using WAFs or AV in general, but, thats the point you want to miss. 😉 (confirmation bias works in both directions) I also have to acknowledge that it’s a lot easier to find cases, where something went wrong instead of cases, where everything was working as it should (that’s not a good headline to begin with 😃) So I agree, that I’m biased, but not without reason. Just to make it clear, I would never suggest to rely on peoples common sense to prevent attacks, so I recommend usings WAFs and AVs, but - and that’s the important point here - these systems should not be taken for granted as the holy grail of protection. It’s just another layer of security…

Generally speaking a security concept needs to factor in a lot of things, including but not limited to attack vectors / attack surface. A WAFs / AVs benefit may outweigh the risks, but these risks have to be assessed as well. The main point in the discussion was not about, whether AVs / WAFs enhance security or lower security, but that SELinux is there for a reason, the same way AVs and WAFs are there. So I think, we both made our point. Everybody can do whatever he/she wants, but I will stay with SELinux as an additional layer. And as I found the solution to the aforementioned problem, there’s no need to disable it anymore.

esackbauer

100% agree.

RaVoR

mlcwuser Great summary. 🙂

I normally use debian as well, but for something heavily exposed like a mail server, I wanted the extra security, so I’ve opted especially for a distribution with SELinux and container ruleset (debian unfortunately lacks the latter). I had to install it from scratch as my mailcow is not running at home, but on a vServer and my provider does not provide RockyLinux by default…

At the end, it always boils down to “what are you using it for”. In my case, the mail server powers my own company as well as some other small companies, which heavily rely on their mails. So the system is regarded critical. 🙂