Postfix container exited but not crashed

qiju

I’m currently running Mailcow in test mode. This is the second time the Postfix container has stopped while all other containers are still running. It doesn’t appear that anything has crashed.

Docker ps for postfix

NAME                                  IMAGE                               COMMAND                  SERVICE           CREATED      STATUS                  PORTS
mailcowdockerized-postfix-mailcow-1   ghcr.io/mailcow/postfix:3.10.12-1   "/docker-entrypoint.…"   postfix-mailcow   4 days ago   Exited (0) 2 days ago

Docker logs documenting the termination

2026-07-25 02:45:34,156 WARN received SIGTERM indicating exit request
2026-07-25 02:45:38,020 INFO waiting for processes, postfix, syslog-ng to die
Jul 25 02:45:38 390e8766e8f9 syslog-ng[10]: syslog-ng shutting down; version='4.8.1'
2026-07-25 02:45:38,124 INFO stopped: syslog-ng (exit status 0)
2026-07-25 02:45:41,129 INFO waiting for processes, postfix to die
2026-07-25 02:45:42,668 WARN received SIGQUIT indicating exit request
2026-07-25 02:45:44,670 INFO waiting for processes, postfix to die
2026-07-25 02:45:47,674 INFO waiting for processes, postfix to die
2026-07-25 02:45:49,339 WARN killing 'postfix' (9) with SIGKILL
2026-07-25 02:45:50,341 WARN stopped: postfix (terminated by SIGKILL)
2026-07-25 02:45:50,341 WARN stopped: processes (terminated by SIGTERM)

Excerpt from Docker inspect of the postfix container

{
	"Name": "/mailcowdockerized-postfix-mailcow-1",
	"State": {
		"Status": "exited",
		"Running": false,
		"Paused": false,
		"Restarting": false,
		"OOMKilled": false,
		"Dead": false,
		"Pid": 0,
		"ExitCode": 0,
		"Error": "",
		"StartedAt": "2026-07-24T00:56:14.434327961Z",
		"FinishedAt": "2026-07-25T00:46:41.347974081Z"
	},
	"RestartPolicy": {
		"Name": "always",
		"MaximumRetryCount": 0
	}
}

What could be the reason that Postfix is terminated, seemingly without anything crashing?

qiju

In the watchdog logs I found the following:

Fri Jul 24 02:50:15 CEST 2026 Postfix health level: 0% (0/8), health trend: -2
Fri Jul 24 02:50:16 CEST 2026 Postfix hit error limit
SMTP OK - 0.044 sec. response time|time=0.044068s;;;0.000000
SMTP OK - 0.014 sec. response time|time=0.013894s;;;0.000000
SMTP OK - 0.080 sec. response time|time=0.079676s;;;0.000000
SMTP OK - 0.015 sec. response time|time=0.014737s;;;0.000000
...
SMTP OK - 0.016 sec. response time|time=0.015648s;;;0.000000
SMTP OK - 0.094 sec. response time|time=0.094012s;;;0.000000
SMTP OK - 0.016 sec. response time|time=0.015754s;;;0.000000
SMTP OK - 0.356 sec. response time|time=0.355858s;;;0.000000
SMTP OK - 0.017 sec. response time|time=0.016934s;;;0.000000
CRITICAL - Socket timeout
SMTP OK - 0.039 sec. response time|time=0.038857s;;;0.000000
CRITICAL - Socket timeout
SMTP OK - 0.014 sec. response time|time=0.013541s;;;0.000000
CRITICAL - Socket timeout
SMTP OK - 0.019 sec. response time|time=0.018709s;;;0.000000
CRITICAL - Socket timeout
SMTP OK - 0.019 sec. response time|time=0.019354s;;;0.000000
Fri Jul 24 02:51:04 CEST 2026 Sending restart command to 390e8766e8f92e8a16b0802f53b5ae0739a7067fe06dc62e946af9b1e7711f3b...
{
    "type": "danger",
    "msg": "UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=70)"
}Fri Jul 24 02:52:14 CEST 2026 Wait for restarted container to settle and continue watching...
...
Sat Jul 25 00:46:00 CEST 2026 Postfix health level: 100% (8/8), health trend: 0
Sat Jul 25 00:47:05 CEST 2026 Postfix health level: 100% (8/8), health trend: 0
Sat Jul 25 00:48:14 CEST 2026 Postfix health level: 100% (8/8), health trend: 0
...
Sat Jul 25 02:44:48 CEST 2026 Postfix health level: 0% (0/8), health trend: -2
Sat Jul 25 02:44:49 CEST 2026 Postfix hit error limit
CRITICAL - Socket timeout
SMTP OK - 0.013 sec. response time|time=0.013148s;;;0.000000
SMTP OK - 0.040 sec. response time|time=0.039940s;;;0.000000
SMTP OK - 0.015 sec. response time|time=0.014624s;;;0.000000
SMTP OK - 0.052 sec. response time|time=0.051947s;;;0.000000
SMTP OK - 0.014 sec. response time|time=0.013819s;;;0.000000
...
SMTP OK - 0.093 sec. response time|time=0.093476s;;;0.000000
SMTP OK - 0.015 sec. response time|time=0.014934s;;;0.000000
SMTP OK - 0.124 sec. response time|time=0.124482s;;;0.000000
SMTP OK - 0.016 sec. response time|time=0.015844s;;;0.000000
CRITICAL - Socket timeout
SMTP OK - 0.013 sec. response time|time=0.013298s;;;0.000000
CRITICAL - Socket timeout
SMTP OK - 0.013 sec. response time|time=0.012853s;;;0.000000
CRITICAL - Socket timeout
SMTP OK - 0.014 sec. response time|time=0.013573s;;;0.000000
CRITICAL - Socket timeout
SMTP OK - 0.013 sec. response time|time=0.013016s;;;0.000000
Sat Jul 25 02:45:03 CEST 2026 Sending restart command to 390e8766e8f92e8a16b0802f53b5ae0739a7067fe06dc62e946af9b1e7711f3b...
{
    "type": "danger",
    "msg": "UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=70)"
}Sat Jul 25 02:46:13 CEST 2026 Wait for restarted container to settle and continue watching...
...
Sat Jul 25 02:46:48 CEST 2026 Postfix health level: 50% (4/8), health trend: -4
...
Sat Jul 25 02:56:43 CEST 2026 Postfix health level: 0% (0/8), health trend: -4
Sat Jul 25 02:56:44 CEST 2026 Postfix hit error limit

CRITICAL - Socket timeout
CRITICAL - Socket timeout
CRITICAL - Socket timeout
CRITICAL - Socket timeout
...
Sat Jul 25 02:58:48 CEST 2026 Postfix health level: 0% (0/8), health trend: -4
Sat Jul 25 02:58:49 CEST 2026 Postfix hit error limit

CRITICAL - Socket timeout
CRITICAL - Socket timeout
CRITICAL - Socket timeout
CRITICAL - Socket timeout
CRITICAL - Socket timeout
CRITICAL - Socket timeout
CRITICAL - Socket timeout
CRITICAL - Socket timeout
...
Sat Jul 25 02:59:19 CEST 2026 Postfix health level: 50% (4/8), health trend: -4

I’m not sure about what’s going on.

qiju

Restarting the Container from Dashboard results in an error:

The red “Error:” text appears directly, without delay.

ETNyx

Hello,

my guess is that watchdog it-self is “secondary” cause. Due to your Docker inspect (no omm, not dead, just plain exit).

Can you check if those incidents are occurring in same time. This timing could show us that for example some kind of backup is occurring and when conditions are right, let’s say sql backup is running in lock mode and postfix needs to verify identity against that database, than it seems to be unresponsive and watchdog shut it down,… Or due to high I/O it seems to freeze,… Or something like this.

qiju

It hasn’t happened again so far; the first time, I didn’t really look into it and just restarted everything.

However, I found the following in the Docker logs in journald:

Jul 25 02:45:09 dockerd[907]: time="2026-07-25T02:45:03.709817275+02:00" level=info msg="stopping restart-manager" container=390e8766e8f92e8a16b0802f53b5ae0739a7067fe06dc62e946af9b1e7711f3b
Jul 25 02:45:43 dockerd[907]: time="2026-07-25T02:45:43.657079500+02:00" level=info msg="Container failed to exit within 10s of signal 15 - using the force" container=390e8766e8f92e8a16b0802f53b5ae0739a7067fe06dc62e946af9b1e7711f3b
Jul 25 02:45:43 dockerd[907]: time="2026-07-25T02:45:43.657141087+02:00" level=info msg="stopping restart-manager" container=390e8766e8f92e8a16b0802f53b5ae0739a7067fe06dc62e946af9b1e7711f3b
Jul 25 02:46:25 dockerd[907]: time="2026-07-25T02:46:25.354717496+02:00" level=warning msg="Container failed to exit within 10s of kill - trying direct SIGKILL" container=390e8766e8f92e8a16b0802f53b5ae0739a7067fe06dc62e946af9b1e7711f3b error="context deadline exceeded"
Jul 25 02:46:29 dockerd[907]: time="2026-07-25T02:46:29.356738223+02:00" level=error msg="error killing container: context deadline exceeded" container=390e8766e8f92e8a16b0802f53b5ae0739a7067fe06dc62e946af9b1e7711f3b error="tried to kill container, but did not receive an exit event"
Jul 25 02:46:41 dockerd[907]: time="2026-07-25T02:46:41.363308102+02:00" level=info msg="received task-delete event from containerd" container=390e8766e8f92e8a16b0802f53b5ae0739a7067fe06dc62e946af9b1e7711f3b module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Jul 25 02:46:48 dockerd[907]: time="2026-07-25T02:46:48.320994705+02:00" level=info msg="request cancelled by client" error="Cannot restart container 390e8766e8f92e8a16b0802f53b5ae0739a7067fe06dc62e946af9b1e7711f3b: tried to kill container, but did not receive an exit event" method=POST module=api request-url="/v1.54/containers/390e8766e8f92e8a16b0802f53b5ae0739a7067fe06dc62e946af9b1e7711f3b/restart?t=10" status=499 vars="map[name:390e8766e8f92e8a16b0802f53b5ae0739a7067fe06dc62e946af9b1e7711f3b version:1.54]"
Jul 25 02:47:52 dockerd[907]: time="2026-07-25T02:47:52.270187774+02:00" level=error msg="failed to remove mount temp dir" dir=/var/lib/docker/rootfs/overlayfs/390e8766e8f92e8a16b0802f53b5ae0739a7067fe06dc62e946af9b1e7711f3b error="remove /var/lib/docker/rootfs/overlayfs/390e8766e8f92e8a16b0802f53b5ae0739a7067fe06dc62e946af9b1e7711f3b: no such file or directory"

It looks to me as though the watchdog tried to restart the container, but an error (timeout?) occurred in Docker during the process, resulting in the container being terminated without being restarted.

I wasn’t able to quickly figure out from the watchdog.sh script how it reacts when a container is stopped. Is a restart command sufficient to restart a stopped container? Or does a different command need to be issued to Docker in such a case?

Shouldn’t the watchdog generally restart a stopped container, as in this case?

To me, it doesn’t look like the watchdog even tried to do this. The container with that ID still existed; it was just stopped. I restarted it manually with the command: docker compose up -d postfix-mailcow

ETNyx

Well yes, I took a quick look at the watchdog script. It seems that when a restart fails and gets escalated to SIGKILL (as in your case), the watchdog never tries to restart that container again, because it only lists live containers. So the dead container itself is never restarted.

As this seems to be an edge case, you could try opening an issue on GitHub to make the watchdog script more resilient.