Hi, I have been running mailcow for about two years now. And so far this has not happened to me before: The Postfix container keeps terminating: `2023-07-05 08:05:12,250 WARN received SIGTERM indicating exit request 2023-07-05 08:05:12,250 INFO waiting for processes, postfix, syslog-ng to die Jul 5 08:05:12 5faa64d0cb63 syslog-ng[9]: syslog-ng shutting down; version=3.28.1 2023-07-05 08:05:12,353 INFO stopped: syslog-ng (exit status 0) 2023-07-05 08:05:13,356 INFO stopped: postfix (terminated by SIGTERM) 2023-07-05 08:05:13,357 INFO reaped unknown pid 385 (terminated by SIGTERM) 2023-07-05 08:05:13,357 INFO reaped unknown pid 386 (terminated by SIGTERM) 2023-07-05 08:05:13,357 INFO reaped unknown pid 387 (terminated by SIGTERM) 2023-07-05 08:05:13,357 INFO reaped unknown pid 388 (terminated by SIGTERM) 2023-07-05 08:05:13,357 INFO reaped unknown pid 392 (terminated by SIGTERM) 2023-07-05 08:05:13,357 INFO reaped unknown pid 393 (terminated by SIGTERM) 2023-07-05 08:05:13,357 INFO reaped unknown pid 397 (exit status 15) 2023-07-05 08:05:13,357 INFO reaped unknown pid 668 (terminated by SIGTERM) 2023-07-05 08:05:13,357 INFO reaped unknown pid 745 (terminated by SIGTERM) 2023-07-05 08:05:13,357 INFO reaped unknown pid 806 (terminated by SIGTERM) 2023-07-05 08:05:13,358 INFO reaped unknown pid 840 (terminated by SIGTERM) 2023-07-05 08:05:13,358 INFO reaped unknown pid 841 (terminated by SIGTERM) 2023-07-05 08:05:13,358 INFO stopped: processes (terminated by SIGTERM) 2023-07-05 08:05:13,358 INFO reaped unknown pid 855 (exit status 0)' I can manually spin the container up and it will run for a while but after an hour or it will terminate with the same log entries. The cow is up to date with the (second) May update. Any ideas what is happening?

The first line says it. Something is trying to stop postfix. Please check other logs e.g. Watchdog etc. https://docs.mailcow.email/troubleshooting/debug-attach_service/

Postfix (container) keeps terminating

boohoomoo

esackbauer

Thanks. Yes, so apparently Watchdog requests a restart after Postfix has hit a predefined error limit:

`Wed Jul 5 12:01:17 CEST 2023 Postfix hit error limit

…

Wed Jul 5 12:02:56 CEST 2023 Sending restart command to 37d337978fcc068af250e30462e0e422ecb2fa4c45a60e4ba906177c2d5fcec5…

{
“type”: “danger”,
“msg”: “UnixHTTPConnectionPool(host=‘localhost’, port=None): Read timed out. (read timeout=70)”
}Wed Jul 5 12:04:06 CEST 2023 Wait for restarted container to settle and continue watching…`

But the problem is that Postfix shuts down und never comes back…

esackbauer

Before the watchdog request SIGTERM there is already a problem: “Read timed out. (read timeout=70)”, maybe you are looking at the wrong location in postfix log.

boohoomoo

Hmm.

Between the last restart and the new shutdown, I only find these two in the logs that indicate that something is off:

chown: cannot access '/usr/share/man/man1/mailq.1.gz': No such file or directory

This one probably has nothing to do with the issue.

The other one seems to point in the same direction as the Watchdog log:

Jul 5 14:59:58 37d337978fcc postfix/master[385]: warning: unix_trigger_event: read timeout for service private/tlsmgr

Everything else reads like business as usual.

So for some reason (too many timeouts), Watchdog requests Postfix to restart but that doesn’t work. To me, the most important question right now is: why?

Obviously, those timeouts need to be addressed as well, but I do have a suspicion: My server is having performance issues at the moment due to insufficient storage speed. So this slows everything down. I am on it, but it will take a bit.

In the meantime, if I could make sure that Postfix comes back up after shutting down, this would help a lot.

esackbauer

My server is having performance issues at the moment due to insufficient storage speed.
warning: unix_trigger_event: read timeout for service private/tlsmgr

That is most likely your problem. And most likely there is nothing else you can do about it.

boohoomoo

esackbauer

Agreed, at least this is one of my problems, which I am about to address. But I can’t do that now. It will take some time.

So I need a solution that I can implement until then. And that would be to make sure that Postfix actually restarts (or that Watchdogs give it more slack before calling for a restart). Any ideas how to do this?

esackbauer

Stopping the watchdog-mailcow container? But I doubt that will help, in fact watchdog is only noticing that postfix already is failing due to I/O errors.

boohoomoo

esackbauer

I considered stopping the watchdog container. But I was afraid that stopping it might cause other issues down the road.

Instead, I made my own watchdog script outside docker. It checks every 30 seconds whether postfix is still running and if it isn’t, it starts it again.

The second measure was to reduce the load on my server. With less load, postfix does not crash that often anymore.

And finally, I am replacing the slow SSDs with enterprise grade SSDs.

That should then finally fix it.

boohoomoo

I will try that as a stopgap measure and see how it goes. Hopefully, Postfix is not actually failing but just responding too slowly for Watchdog’s liking.

Coming back to Postfix’s failing to restart - any ideas why it might be failing to restart?

esackbauer

any ideas why it might be failing to restart?

No idea, what do the logs say? Most likely you cannot restart it because its still running. Your first log shows that it fails to stop properly.

boohoomoo

Just what I posted above. After that: nothing.

But from that state I can simply start it again manually. So it doesn’t seem to be in too bad a shape…

mwlistscom

For the googlers out there - I had the same issue with postfix restarts, and identical error -

“INFO stopped: postfix (terminated by SIGTERM)”

In my setup I must have modified extra.cf to include my DMZ at some point in the past :

data/conf/postfix/extra.cf:

I followed this KB to fix it -

https://docs.mailcow.email/manual-guides/Postfix/u_e-postfix-unauthenticated-relaying/

I added my local networks, and 172.22.1.0/24 - which is the default mailcow network. It wasn’t there, so I assume I removed it at some point in the past not knowing what it was at the time.

Postfixt stop crashing - Enjoy !

LxanderUA

mwlistscom I added my local networks, and 172.22.1.0/24
Thanks for your help.
I had the same issue, and your comments solved it.
I think after adding the local network parameter to extra.cf, we overrode the internal value that contained the internal network.