CRITICAL - Socket Timeout

marshalleq

Hi all, I’m having this recurring issue of late where I have to restart mailcow to get mail to deliver again. Clients can’t connect before this and also mail is not delivered to / from the server.

The bit of context is, there are some disk problems in the machine caused by what I thought was a resolved overheating LSI controller. That in itself isn’t an issue anymore, but during that it seems that performing a resilver (I use ZFS file system), the mail server does this. I’m not 100% sure these two things are linked, but just throwing it out there in case the logs point to hardware error. By the way, the mail server is on different disks on a different controller and is still functioning in that I can ssh into the box and just restart the services. The host is TrueNAS and has 128G RAM of which 16G is allocated to Mailcow for its 3 users and about 10 mailboxes. The CPU is a threadripper 2950x and there is plenty of disk space left, it’s on a 6 disk ssd mirror.

If you scroll down tot he middle of the log below you will see both the no child available to handle incoming request and further down the Critical Socket Timeout error - I’m not sure which comes first, I assume the latter.

Also, due to the new requirement to have a certain version of docker, I actually created a brand new VM, along with everything and did a mailcow migration to this one. It was running quite well for a few days until this morning where it was offline again and it just so happened the disk issue recurred last night and has been resilvering overnight.. Anyway, just to point out the system is all clean from scratch barring that it’s the same data / config etc.

Any ideas what I should do to fix this?

Thanks.

sogo-mailcow-1       | Oct 27 07:51:53 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
postfix-mailcow-1    | Oct 27 07:51:53 61057710b505 postfix/smtps/smtpd[3793]: lost connection after AUTH from unknown[36.138.132.109]
postfix-mailcow-1    | Oct 27 07:51:53 61057710b505 postfix/smtps/smtpd[3793]: disconnect from unknown[36.138.132.109] ehlo=1 auth=0/1 commands=1/2
postfix-mailcow-1    | Oct 27 07:51:54 61057710b505 postfix/smtps/smtpd[3795]: warning: unknown[212.50.48.86]: SASL PLAIN authentication failed: (reason unavailable), sasl_username=arts
netfilter-mailcow-1  | 212.50.48.86 matched rule id 3 (warning: unknown[212.50.48.86]: SASL PLAIN authentication failed: (reason unavailable), sasl_username=arts)
netfilter-mailcow-1  | 9 more attempts in the next 600 seconds until 212.50.48.86/32 is banned
sogo-mailcow-1       | Oct 27 07:51:54 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
postfix-mailcow-1    | Oct 27 07:51:55 61057710b505 postfix/smtps/smtpd[3795]: lost connection after AUTH from unknown[212.50.48.86]
postfix-mailcow-1    | Oct 27 07:51:55 61057710b505 postfix/smtps/smtpd[3795]: disconnect from unknown[212.50.48.86] ehlo=1 auth=0/1 commands=1/2
sogo-mailcow-1       | Oct 27 07:51:56 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:51:57 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:51:59 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:52:00 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
postfix-mailcow-1    | Oct 27 07:52:01 61057710b505 postfix/smtps/smtpd[3797]: connect from unknown[183.236.75.189]
sogo-mailcow-1       | Oct 27 07:52:02 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
dockerapi-mailcow-1  | INFO:     172.22.1.5:55200 - "GET /containers/json HTTP/1.1" 200 OK
dockerapi-mailcow-1  | INFO:     172.22.1.5:55206 - "GET /containers/d3cade719bc7ca4b62949447c5487c73e4dc8dbdfd25f4a8408aac28bb7350ac/json HTTP/1.1" 200 OK
watchdog-mailcow-1   | Sun Oct 27 07:52:02 NZDT 2024 Container is running for less than 360 seconds, skipping action...
watchdog-mailcow-1   | Sun Oct 27 07:52:02 NZDT 2024 Redis health level: 100% (5/5), health trend: 0
watchdog-mailcow-1   | Sun Oct 27 07:52:02 NZDT 2024 Clamd health level: 100% (15/15), health trend: 0
watchdog-mailcow-1   | Sun Oct 27 07:52:02 NZDT 2024 PHP-FPM health level: 100% (5/5), health trend: 0
postfix-mailcow-1    | Oct 27 07:52:02 61057710b505 postfix/smtps/smtpd[3797]: Anonymous TLS connection established from unknown[183.236.75.189]: TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
postfix-mailcow-1    | Oct 27 07:52:03 61057710b505 postfix/postscreen[3798]: CONNECT from [45.125.66.223]:60296 to [172.22.1.253]:25
php-fpm-mailcow-1    | fd4d:6169:6c63:6f77::3 -  27/Oct/2024:07:52:03 +1300 "GET /forwardinghosts.php" 200
nginx-mailcow-1      | fd4d:6169:6c63:6f77::11 - - [27/Oct/2024:07:52:03 +1300] "GET /forwardinghosts.php?host=45.125.66.223 HTTP/1.1" 200 19 "-" "curl/7.88.1"
postfix-mailcow-1    | Oct 27 07:52:03 61057710b505 whitelist_forwardinghosts: Look up 45.125.66.223 on whitelist, result 200 DUNNO
postfix-mailcow-1    | Oct 27 07:52:03 61057710b505 postfix/postscreen[3798]: PREGREET 11 after 0.35 from [45.125.66.223]:60296: EHLO User\r\n
postfix-mailcow-1    | Oct 27 07:52:03 61057710b505 postfix/dnsblog[3816]: addr 45.125.66.223 listed by domain bl.mailspike.net as 127.0.0.2
sogo-mailcow-1       | Oct 27 07:52:03 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
postfix-mailcow-1    | Oct 27 07:52:03 61057710b505 postfix/dnsblog[3813]: addr 45.125.66.223 listed by domain hostkarma.junkemailfilter.com as 127.0.0.5
sogo-mailcow-1       | Oct 27 07:52:05 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
php-fpm-mailcow-1    | 172.22.1.3 -  27/Oct/2024:07:52:05 +1300 "HEAD /forwardinghosts.php" 200
nginx-mailcow-1      | 172.22.1.4 - - [27/Oct/2024:07:52:05 +1300] "HEAD /forwardinghosts.php HTTP/1.1" 200 0 "-" "rspamd-3.9.1"
php-fpm-mailcow-1    | 172.22.1.3 -  27/Oct/2024:07:52:05 +1300 "GET /forwardinghosts.php" 200
nginx-mailcow-1      | fd4d:6169:6c63:6f77::4 - - [27/Oct/2024:07:52:05 +1300] "GET /forwardinghosts.php HTTP/1.1" 200 43 "-" "rspamd-3.9.1"
sogo-mailcow-1       | Oct 27 07:52:06 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
dovecot-mailcow-1    | Oct 27 07:52:07 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for arts: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:52:07 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for arts: Worker process died unexpectedly
postfix-mailcow-1    | Oct 27 07:52:07 61057710b505 postfix/postscreen[3798]: DISCONNECT [45.125.66.223]:60296
sogo-mailcow-1       | Oct 27 07:52:08 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
postfix-mailcow-1    | Oct 27 07:52:09 61057710b505 postfix/smtps/smtpd[3797]: warning: unknown[183.236.75.189]: SASL PLAIN authentication failed: (reason unavailable), sasl_username=arts
netfilter-mailcow-1  | 183.236.75.189 matched rule id 3 (warning: unknown[183.236.75.189]: SASL PLAIN authentication failed: (reason unavailable), sasl_username=arts)
netfilter-mailcow-1  | 9 more attempts in the next 600 seconds until 183.236.75.189/32 is banned
php-fpm-mailcow-1    | fd4d:6169:6c63:6f77::3 -  27/Oct/2024:07:52:09 +1300 "HEAD /settings.php" 304
nginx-mailcow-1      | 172.22.1.4 - - [27/Oct/2024:07:52:09 +1300] "HEAD /settings.php HTTP/1.1" 304 0 "-" "rspamd-3.9.1"
sogo-mailcow-1       | Oct 27 07:52:09 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
postfix-mailcow-1    | Oct 27 07:52:09 61057710b505 postfix/smtps/smtpd[3797]: lost connection after AUTH from unknown[183.236.75.189]
postfix-mailcow-1    | Oct 27 07:52:09 61057710b505 postfix/smtps/smtpd[3797]: disconnect from unknown[183.236.75.189] ehlo=1 auth=0/1 commands=1/2
watchdog-mailcow-1   | Sun Oct 27 07:52:10 NZDT 2024 Unbound health level: 100% (5/5), health trend: 0
sogo-mailcow-1       | Oct 27 07:52:11 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
watchdog-mailcow-1   | Sun Oct 27 07:52:12 NZDT 2024 SOGo health level: 33% (1/3), health trend: -2
sogo-mailcow-1       | Oct 27 07:52:12 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
watchdog-mailcow-1   | Sun Oct 27 07:52:13 NZDT 2024 ACME health level: 100% (1/1), health trend: 0
sogo-mailcow-1       | Oct 27 07:52:14 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
watchdog-mailcow-1   | Sun Oct 27 07:52:15 NZDT 2024 Olefy health level: 100% (5/5), health trend: 0
sogo-mailcow-1       | Oct 27 07:52:16 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:52:17 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
dovecot-mailcow-1    | Oct 27 07:52:18 f446bcac59bf dovecot: lmtp(5375): Connect from 172.22.1.5
dovecot-mailcow-1    | Oct 27 07:52:18 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted USER request for watchdog@invalid: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:52:18 f446bcac59bf dovecot: lmtp(watchdog@invalid)<5375><2LR6OWI6HWf/FAAANQEoeQ>: Error: auth-master: userdb lookup(watchdog@invalid): Auth USER lookup failed
dovecot-mailcow-1    | Oct 27 07:52:18 f446bcac59bf dovecot: lmtp(5375): Error: lmtp-server: conn 172.22.1.5:36920 [36]: rcpt watchdog@invalid: Failed to lookup user watchdog@invalid: Internal error occurred. Refer to server log for more information.
dovecot-mailcow-1    | Oct 27 07:52:18 f446bcac59bf dovecot: lmtp(5375): Disconnect from 172.22.1.5: Logged out (state=MAIL FROM)
dovecot-mailcow-1    | Oct 27 07:52:19 f446bcac59bf dovecot: imap-login: Disconnected: Aborted login by logging out (no auth attempts in 1 secs): user=<>, rip=172.22.1.5, lip=172.22.1.250, TLS, TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
dovecot-mailcow-1    | Oct 27 07:52:19 f446bcac59bf dovecot: imap-login: Disconnected: Aborted login by logging out (no auth attempts in 0 secs): user=<>, rip=172.22.1.5, lip=172.22.1.250
dovecot-mailcow-1    | Oct 27 07:52:19 f446bcac59bf dovecot: managesieve-login: Disconnected: Connection closed (no auth attempts in 0 secs): user=<>, rip=172.22.1.5, lip=172.22.1.250
watchdog-mailcow-1   | Sun Oct 27 07:52:19 NZDT 2024 Dovecot health level: 33% (4/12), health trend: -1
sogo-mailcow-1       | Oct 27 07:52:19 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
watchdog-mailcow-1   | Sun Oct 27 07:52:19 NZDT 2024 Mail queue health level: 100% (20/20), health trend: 0
sogo-mailcow-1       | Oct 27 07:52:20 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:52:22 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
watchdog-mailcow-1   | Sun Oct 27 07:52:23 NZDT 2024 Fail2ban health level: 100% (1/1), health trend: 0
sogo-mailcow-1       | Oct 27 07:52:23 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:52:25 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:52:26 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
watchdog-mailcow-1   | Sun Oct 27 07:52:27 NZDT 2024 Ratelimit health level: 100% (1/1), health trend: 0
sogo-mailcow-1       | Oct 27 07:52:28 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
php-fpm-mailcow-1    | fd4d:6169:6c63:6f77::3 - uernameredactedj@domainredacted.net 27/Oct/2024:07:52:28 +1300 "GET /sogo-auth.php" 200
nginx-mailcow-1      | 127.0.0.1 - uernameredactedj@domainredacted.net [27/Oct/2024:07:52:28 +1300] "GET /sogo-auth HTTP/1.0" 200 0 "-" "Apple-iPhone14C6/2201.3370"
sogo-mailcow-1       | Oct 27 07:52:29 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:52:31 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
watchdog-mailcow-1   | Sun Oct 27 07:52:31 NZDT 2024 Postfix health level: 100% (8/8), health trend: 0
nginx-mailcow-1      | 2024/10/27 07:52:31 [error] 24#24: *10279 upstream timed out (110: Operation timed out) while connecting to upstream, client: 192.168.43.165, server: spock.domainredacted.net, request: "POST /Microsoft-Server-ActiveSync?User=atoms@domainredacted.net&DeviceId=7CKPG4A03D7E74R7NRLB8BH528&DeviceType=iPhone&Cmd=Sync HTTP/1.1", upstream: "http://172.22.1.248:20000/SOGo/Microsoft-Server-ActiveSync?User=atoms@domainredacted.net&DeviceId=7CKPG4A03D7E74R7NRLB8BH528&DeviceType=iPhone&Cmd=Sync", host: "webmail.domainredacted.net"
nginx-mailcow-1      | 192.168.43.165 - atoms@domainredacted.net [27/Oct/2024:07:52:31 +1300] "POST /Microsoft-Server-ActiveSync?User=atoms@domainredacted.net&DeviceId=7CKPG4A03D7E74R7NRLB8BH528&DeviceType=iPhone&Cmd=Sync HTTP/1.1" 504 160 "-" "Apple-iPhone14C6/2201.3370"
php-fpm-mailcow-1    | 172.22.1.3 - uernameredactedj@domainredacted.co.nz 27/Oct/2024:07:52:32 +1300 "GET /sogo-auth.php" 200
nginx-mailcow-1      | 127.0.0.1 - uernameredactedj@domainredacted.co.nz [27/Oct/2024:07:52:32 +1300] "GET /sogo-auth HTTP/1.0" 200 0 "-" "Apple-iPhone14C6/2201.3370"
sogo-mailcow-1       | Oct 27 07:52:32 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
nginx-mailcow-1      | 2024/10/27 07:52:33 [error] 24#24: *10284 upstream timed out (110: Operation timed out) while connecting to upstream, client: 192.168.43.165, server: spock.domainredacted.net, request: "POST /Microsoft-Server-ActiveSync?User=uernameredactedj@domainredacted.co.nz&DeviceId=7CKPG4A03D7E74R7NRLB8BH528&DeviceType=iPhone&Cmd=Ping HTTP/1.1", upstream: "http://172.22.1.248:20000/SOGo/Microsoft-Server-ActiveSync?User=uernameredactedj@domainredacted.co.nz&DeviceId=7CKPG4A03D7E74R7NRLB8BH528&DeviceType=iPhone&Cmd=Ping", host: "webmail.domainredacted.co.nz"
nginx-mailcow-1      | 192.168.43.165 - uernameredactedj@domainredacted.co.nz [27/Oct/2024:07:52:33 +1300] "POST /Microsoft-Server-ActiveSync?User=uernameredactedj@domainredacted.co.nz&DeviceId=7CKPG4A03D7E74R7NRLB8BH528&DeviceType=iPhone&Cmd=Ping HTTP/1.1" 504 160 "-" "Apple-iPhone14C6/2201.3370"
php-fpm-mailcow-1    | fd4d:6169:6c63:6f77::3 - atoms@domainredacted.net 27/Oct/2024:07:52:34 +1300 "GET /sogo-auth.php" 200
nginx-mailcow-1      | 127.0.0.1 - atoms@domainredacted.net [27/Oct/2024:07:52:34 +1300] "GET /sogo-auth HTTP/1.0" 200 0 "-" "Apple-iPhone14C6/2201.3370"
sogo-mailcow-1       | Oct 27 07:52:34 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:52:35 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:52:37 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:52:38 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
nginx-mailcow-1      | 172.22.1.5 - - [27/Oct/2024:07:52:40 +1300] "GET / HTTP/1.1" 200 15 "-" "check_http/v (nagios-plugins 2.4.5)"
watchdog-mailcow-1   | Sun Oct 27 07:52:40 NZDT 2024 Nginx health level: 100% (5/5), health trend: 0
sogo-mailcow-1       | Oct 27 07:52:40 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
watchdog-mailcow-1   | Sun Oct 27 07:52:40 NZDT 2024 Olefy health level: 100% (5/5), health trend: 0
watchdog-mailcow-1   | Sun Oct 27 07:52:41 NZDT 2024 Mail queue health level: 100% (20/20), health trend: 0
sogo-mailcow-1       | Oct 27 07:52:41 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
php-fpm-mailcow-1    | 172.22.1.3 -  27/Oct/2024:07:52:43 +1300 "HEAD /settings.php" 304
nginx-mailcow-1      | 172.22.1.4 - - [27/Oct/2024:07:52:43 +1300] "HEAD /settings.php HTTP/1.1" 304 0 "-" "rspamd-3.9.1"
sogo-mailcow-1       | Oct 27 07:52:43 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:52:44 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
php-fpm-mailcow-1    | fd4d:6169:6c63:6f77::3 -  27/Oct/2024:07:52:46 +1300 "HEAD /forwardinghosts.php" 200
nginx-mailcow-1      | 172.22.1.4 - - [27/Oct/2024:07:52:46 +1300] "HEAD /forwardinghosts.php HTTP/1.1" 200 0 "-" "rspamd-3.9.1"
php-fpm-mailcow-1    | 172.22.1.3 -  27/Oct/2024:07:52:46 +1300 "GET /forwardinghosts.php" 200
nginx-mailcow-1      | 172.22.1.4 - - [27/Oct/2024:07:52:46 +1300] "GET /forwardinghosts.php HTTP/1.1" 200 43 "-" "rspamd-3.9.1"
sogo-mailcow-1       | Oct 27 07:52:46 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
watchdog-mailcow-1   | Sun Oct 27 07:52:47 NZDT 2024 Dovecot replication health level: 100% (20/20), health trend: 0
rspamd-mailcow-1     | 2024-10-27 07:52:47 #44(controller) <5022e9>; csession; rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned, 0 regexps matched, 3511 regexps total, 2578 regexps cached, 0B scanned using pcre, 102B scanned total
watchdog-mailcow-1   | Sun Oct 27 07:52:47 NZDT 2024 Rspamd health level: 100% (5/5), health trend: 0
sogo-mailcow-1       | Oct 27 07:52:47 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:52:49 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:52:51 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:52:52 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:52:54 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:52:55 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:52:57 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:52:58 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:53:00 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
watchdog-mailcow-1   | Sun Oct 27 07:53:00 NZDT 2024 PHP-FPM health level: 100% (5/5), health trend: 0
sogo-mailcow-1       | Oct 27 07:53:01 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
watchdog-mailcow-1   | Sun Oct 27 07:53:02 NZDT 2024 Olefy health level: 100% (5/5), health trend: 0
watchdog-mailcow-1   | Sun Oct 27 07:53:02 NZDT 2024 ACME health level: 100% (1/1), health trend: 0
sogo-mailcow-1       | Oct 27 07:53:03 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
watchdog-mailcow-1   | Sun Oct 27 07:53:03 NZDT 2024 MySQL/MariaDB health level: 100% (5/5), health trend: 0
sogo-mailcow-1       | Oct 27 07:53:04 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
nginx-mailcow-1      | 172.22.1.5 - - [27/Oct/2024:07:53:06 +1300] "GET / HTTP/1.1" 200 15 "-" "check_http/v (nagios-plugins 2.4.5)"
watchdog-mailcow-1   | Sun Oct 27 07:53:06 NZDT 2024 Nginx health level: 100% (5/5), health trend: 0
sogo-mailcow-1       | Oct 27 07:53:06 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:53:07 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
watchdog-mailcow-1   | Sun Oct 27 07:53:08 NZDT 2024 Fail2ban health level: 100% (1/1), health trend: 0
watchdog-mailcow-1   | Sun Oct 27 07:53:08 NZDT 2024 Clamd health level: 100% (15/15), health trend: 0
dovecot-mailcow-1    | Oct 27 07:53:09 f446bcac59bf dovecot: lmtp(5375): Connect from 172.22.1.5
dovecot-mailcow-1    | Oct 27 07:53:09 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted USER request for watchdog@invalid: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:09 f446bcac59bf dovecot: lmtp(watchdog@invalid)<5375><oJQ0BZU6HWf/FAAANQEoeQ>: Error: auth-master: userdb lookup(watchdog@invalid): Auth USER lookup failed
dovecot-mailcow-1    | Oct 27 07:53:09 f446bcac59bf dovecot: lmtp(5375): Error: lmtp-server: conn 172.22.1.5:52686 [37]: rcpt watchdog@invalid: Failed to lookup user watchdog@invalid: Internal error occurred. Refer to server log for more information.
dovecot-mailcow-1    | Oct 27 07:53:09 f446bcac59bf dovecot: lmtp(5375): Disconnect from 172.22.1.5: Logged out (state=MAIL FROM)
dovecot-mailcow-1    | Oct 27 07:53:09 f446bcac59bf dovecot: imap-login: Disconnected: Aborted login by logging out (no auth attempts in 0 secs): user=<>, rip=172.22.1.5, lip=172.22.1.250, TLS, TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
sogo-mailcow-1       | Oct 27 07:53:09 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
dovecot-mailcow-1    | Oct 27 07:53:09 f446bcac59bf dovecot: imap-login: Disconnected: Aborted login by logging out (no auth attempts in 0 secs): user=<>, rip=172.22.1.5, lip=172.22.1.250
dovecot-mailcow-1    | Oct 27 07:53:09 f446bcac59bf dovecot: managesieve-login: Disconnected: Connection closed (no auth attempts in 0 secs): user=<>, rip=172.22.1.5, lip=172.22.1.250
watchdog-mailcow-1   | Sun Oct 27 07:53:09 NZDT 2024 Dovecot health level: 25% (3/12), health trend: -1
sogo-mailcow-1       | Oct 27 07:53:10 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
dovecot-mailcow-1    | Oct 27 07:53:12 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredactedj@domainredacted.net: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:12 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredactedj@domainredacted.net: Worker process died unexpectedly
sogo-mailcow-1       | Oct 27 07:53:12 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:53:13 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:53:15 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
dovecot-mailcow-1    | Oct 27 07:53:16 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredactedj@domainredacted.co.nz: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:16 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredactedj@domainredacted.co.nz: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:16 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for atoms@domainredacted.net: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:16 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for atoms@domainredacted.net: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:16 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredacted.o@domain2redacted.co.nz: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:16 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredacted.o@domain2redacted.co.nz: Worker process died unexpectedly
sogo-mailcow-1       | Oct 27 07:53:16 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
dovecot-mailcow-1    | Oct 27 07:53:18 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredactedj@domainredacted.net: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:18 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredactedj@domainredacted.net: Worker process died unexpectedly
sogo-mailcow-1       | Oct 27 07:53:18 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
watchdog-mailcow-1   | Sun Oct 27 07:53:19 NZDT 2024 Postfix health level: 100% (8/8), health trend: 0
sogo-mailcow-1       | Oct 27 07:53:19 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
watchdog-mailcow-1   | Sun Oct 27 07:53:20 NZDT 2024 Dovecot replication health level: 100% (20/20), health trend: 0
sogo-mailcow-1       | Oct 27 07:53:21 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
watchdog-mailcow-1   | Sun Oct 27 07:53:21 NZDT 2024 Mail queue health level: 100% (20/20), health trend: 0
watchdog-mailcow-1   | Sun Oct 27 07:53:21 NZDT 2024 Redis health level: 100% (5/5), health trend: 0
dovecot-mailcow-1    | Oct 27 07:53:22 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for atoms@domainredacted.net: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:22 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredacted.o@domain2redacted.co.nz: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:22 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredactedj@domainredacted.co.nz: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:22 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for atoms@domainredacted.net: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:22 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredacted.o@domain2redacted.co.nz: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:22 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredactedj@domainredacted.co.nz: Worker process died unexpectedly
watchdog-mailcow-1   | Sun Oct 27 07:53:22 NZDT 2024 Ratelimit health level: 100% (1/1), health trend: 0
sogo-mailcow-1       | Oct 27 07:53:22 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
php-fpm-mailcow-1    | fd4d:6169:6c63:6f77::3 -  27/Oct/2024:07:53:23 +1300 "HEAD /settings.php" 304
nginx-mailcow-1      | 172.22.1.4 - - [27/Oct/2024:07:53:23 +1300] "HEAD /settings.php HTTP/1.1" 304 0 "-" "rspamd-3.9.1"
dovecot-mailcow-1    | Oct 27 07:53:24 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredactedj@domainredacted.net: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:24 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredactedj@domainredacted.net: Worker process died unexpectedly
sogo-mailcow-1       | Oct 27 07:53:24 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
watchdog-mailcow-1   | Sun Oct 27 07:53:25 NZDT 2024 Unbound health level: 100% (5/5), health trend: 0
rspamd-mailcow-1     | 2024-10-27 07:53:25 #44(controller) <add847>; csession; rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned, 0 regexps matched, 3511 regexps total, 2578 regexps cached, 0B scanned using pcre, 102B scanned total
watchdog-mailcow-1   | Sun Oct 27 07:53:25 NZDT 2024 Rspamd health level: 100% (5/5), health trend: 0
sogo-mailcow-1       | Oct 27 07:53:25 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:53:27 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
dovecot-mailcow-1    | Oct 27 07:53:28 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredactedj@domainredacted.co.nz: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:28 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredactedj@domainredacted.co.nz: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:28 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredacted.o@domain2redacted.co.nz: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:28 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredacted.o@domain2redacted.co.nz: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:28 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for atoms@domainredacted.net: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:28 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for atoms@domainredacted.net: Worker process died unexpectedly
sogo-mailcow-1       | Oct 27 07:53:28 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
dovecot-mailcow-1    | Oct 27 07:53:30 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredactedj@domainredacted.net: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:30 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredactedj@domainredacted.net: Worker process died unexpectedly
sogo-mailcow-1       | Oct 27 07:53:30 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
nginx-mailcow-1      | 172.22.1.5 - - [27/Oct/2024:07:53:31 +1300] "GET / HTTP/1.1" 200 15 "-" "check_http/v (nagios-plugins 2.4.5)"
watchdog-mailcow-1   | Sun Oct 27 07:53:31 NZDT 2024 Nginx health level: 100% (5/5), health trend: 0
watchdog-mailcow-1   | Sun Oct 27 07:53:31 NZDT 2024 SOGo health level: 0% (0/3), health trend: -2
sogo-mailcow-1       | Oct 27 07:53:32 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
dovecot-mailcow-1    | Oct 27 07:53:32 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredactedj@domainredacted.net: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:32 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredactedj@domainredacted.net: Worker process died unexpectedly
watchdog-mailcow-1   | Sun Oct 27 07:53:32 NZDT 2024 SOGo hit error limit
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
watchdog-mailcow-1   | CRITICAL - Socket timeout
sogo-mailcow-1       | Oct 27 07:53:33 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
dovecot-mailcow-1    | Oct 27 07:53:34 f446bcac59bf dovecot: imap-login: Disconnected: Aborted login by logging out (auth service reported temporary failure): user=<uernameredactedj@domainredacted.net>, method=PLAIN, rip=192.168.43.121, lip=172.22.1.250, TLS, TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
dovecot-mailcow-1    | Oct 27 07:53:34 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredacted.o@domain2redacted.co.nz: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:34 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredacted.o@domain2redacted.co.nz: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:34 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for atoms@domainredacted.net: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:34 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredactedj@domainredacted.co.nz: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:34 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for atoms@domainredacted.net: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:34 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredactedj@domainredacted.co.nz: Worker process died unexpectedly
php-fpm-mailcow-1    | 172.22.1.3 -  27/Oct/2024:07:53:34 +1300 "HEAD /forwardinghosts.php" 200
nginx-mailcow-1      | 172.22.1.4 - - [27/Oct/2024:07:53:34 +1300] "HEAD /forwardinghosts.php HTTP/1.1" 200 0 "-" "rspamd-3.9.1"
php-fpm-mailcow-1    | fd4d:6169:6c63:6f77::3 -  27/Oct/2024:07:53:34 +1300 "GET /forwardinghosts.php" 200
nginx-mailcow-1      | fd4d:6169:6c63:6f77::4 - - [27/Oct/2024:07:53:34 +1300] "GET /forwardinghosts.php HTTP/1.1" 200 43 "-" "rspamd-3.9.1"
sogo-mailcow-1       | Oct 27 07:53:35 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
dovecot-mailcow-1    | Oct 27 07:53:36 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredactedj@domainredacted.co.nz: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:36 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredactedj@domainredacted.co.nz: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:36 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredacted.o@domain2redacted.co.nz: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:36 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for uernameredacted.o@domain2redacted.co.nz: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:36 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for atoms@domainredacted.net: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:36 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted PASSV request for atoms@domainredacted.net: Worker process died unexpectedly
sogo-mailcow-1       | Oct 27 07:53:36 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
dovecot-mailcow-1    | Oct 27 07:53:38 f446bcac59bf dovecot: imap-login: Disconnected: Aborted login by logging out (auth service reported temporary failure): user=<uernameredactedj@domainredacted.co.nz>, method=PLAIN, rip=192.168.43.121, lip=172.22.1.250, TLS, TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
dovecot-mailcow-1    | Oct 27 07:53:38 f446bcac59bf dovecot: imap-login: Disconnected: Aborted login by logging out (auth service reported temporary failure): user=<atoms@domainredacted.net>, method=PLAIN, rip=192.168.43.121, lip=172.22.1.250, TLS, TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
dovecot-mailcow-1    | Oct 27 07:53:38 f446bcac59bf dovecot: imap-login: Disconnected: Aborted login by logging out (auth service reported temporary failure): user=<uernameredacted.o@domain2redacted.co.nz>, method=PLAIN, rip=192.168.43.121, lip=172.22.1.250, TLS, TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
sogo-mailcow-1       | Oct 27 07:53:38 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:53:39 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:53:41 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:53:42 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
dockerapi-mailcow-1  | INFO:     172.22.1.5:50920 - "GET /containers/json HTTP/1.1" 200 OK
dockerapi-mailcow-1  | INFO:     172.22.1.5:50932 - "GET /containers/d3cade719bc7ca4b62949447c5487c73e4dc8dbdfd25f4a8408aac28bb7350ac/json HTTP/1.1" 200 OK
watchdog-mailcow-1   | Sun Oct 27 07:53:42 NZDT 2024 Container is running for less than 360 seconds, skipping action...
watchdog-mailcow-1   | Sun Oct 27 07:53:42 NZDT 2024 Dovecot replication health level: 100% (20/20), health trend: 0
watchdog-mailcow-1   | Sun Oct 27 07:53:42 NZDT 2024 PHP-FPM health level: 100% (5/5), health trend: 0
nginx-mailcow-1      | 2024/10/27 07:53:43 [error] 24#24: *10304 upstream timed out (110: Operation timed out) while connecting to upstream, client: 192.168.43.165, server: spock.domainredacted.net, request: "POST /Microsoft-Server-ActiveSync?User=uernameredactedj@domainredacted.net&DeviceId=7CKPG4A03D7E74R7NRLB8BH528&DeviceType=iPhone&Cmd=Ping HTTP/1.1", upstream: "http://172.22.1.248:20000/SOGo/Microsoft-Server-ActiveSync?User=uernameredactedj@domainredacted.net&DeviceId=7CKPG4A03D7E74R7NRLB8BH528&DeviceType=iPhone&Cmd=Ping", host: "webmail.domainredacted.net"
nginx-mailcow-1      | 192.168.43.165 - uernameredactedj@domainredacted.net [27/Oct/2024:07:53:43 +1300] "POST /Microsoft-Server-ActiveSync?User=uernameredactedj@domainredacted.net&DeviceId=7CKPG4A03D7E74R7NRLB8BH528&DeviceType=iPhone&Cmd=Ping HTTP/1.1" 504 160 "-" "Apple-iPhone14C6/2201.3370"
sogo-mailcow-1       | Oct 27 07:53:44 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:53:45 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
nginx-mailcow-1      | 2024/10/27 07:53:47 [error] 24#24: *10309 upstream timed out (110: Operation timed out) while connecting to upstream, client: 192.168.43.165, server: spock.domainredacted.net, request: "POST /Microsoft-Server-ActiveSync?User=uernameredactedj@domainredacted.co.nz&DeviceId=7CKPG4A03D7E74R7NRLB8BH528&DeviceType=iPhone&Cmd=Sync HTTP/1.1", upstream: "http://172.22.1.248:20000/SOGo/Microsoft-Server-ActiveSync?User=uernameredactedj@domainredacted.co.nz&DeviceId=7CKPG4A03D7E74R7NRLB8BH528&DeviceType=iPhone&Cmd=Sync", host: "webmail.domainredacted.co.nz"
nginx-mailcow-1      | 192.168.43.165 - uernameredactedj@domainredacted.co.nz [27/Oct/2024:07:53:47 +1300] "POST /Microsoft-Server-ActiveSync?User=uernameredactedj@domainredacted.co.nz&DeviceId=7CKPG4A03D7E74R7NRLB8BH528&DeviceType=iPhone&Cmd=Sync HTTP/1.1" 504 160 "-" "Apple-iPhone14C6/2201.3370"
sogo-mailcow-1       | Oct 27 07:53:47 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
php-fpm-mailcow-1    | 172.22.1.3 - uernameredactedj@domainredacted.net 27/Oct/2024:07:53:47 +1300 "GET /sogo-auth.php" 200
nginx-mailcow-1      | 127.0.0.1 - uernameredactedj@domainredacted.net [27/Oct/2024:07:53:47 +1300] "GET /sogo-auth HTTP/1.0" 200 0 "-" "Apple-iPhone14C6/2201.3370"
sogo-mailcow-1       | Oct 27 07:53:48 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
nginx-mailcow-1      | 2024/10/27 07:53:49 [error] 22#22: *10314 upstream timed out (110: Operation timed out) while connecting to upstream, client: 192.168.43.165, server: spock.domainredacted.net, request: "POST /Microsoft-Server-ActiveSync?User=atoms@domainredacted.net&DeviceId=7CKPG4A03D7E74R7NRLB8BH528&DeviceType=iPhone&Cmd=Ping HTTP/1.1", upstream: "http://172.22.1.248:20000/SOGo/Microsoft-Server-ActiveSync?User=atoms@domainredacted.net&DeviceId=7CKPG4A03D7E74R7NRLB8BH528&DeviceType=iPhone&Cmd=Ping", host: "webmail.domainredacted.net"
nginx-mailcow-1      | 192.168.43.165 - atoms@domainredacted.net [27/Oct/2024:07:53:49 +1300] "POST /Microsoft-Server-ActiveSync?User=atoms@domainredacted.net&DeviceId=7CKPG4A03D7E74R7NRLB8BH528&DeviceType=iPhone&Cmd=Ping HTTP/1.1" 504 160 "-" "Apple-iPhone14C6/2201.3370"
php-fpm-mailcow-1    | fd4d:6169:6c63:6f77::3 - uernameredactedj@domainredacted.co.nz 27/Oct/2024:07:53:49 +1300 "GET /sogo-auth.php" 200
nginx-mailcow-1      | 127.0.0.1 - uernameredactedj@domainredacted.co.nz [27/Oct/2024:07:53:49 +1300] "GET /sogo-auth HTTP/1.0" 200 0 "-" "Apple-iPhone14C6/2201.3370"
rspamd-mailcow-1     | 2024-10-27 07:53:49 #44(controller) <d9ad69>; csession; rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned, 0 regexps matched, 3511 regexps total, 2578 regexps cached, 0B scanned using pcre, 102B scanned total
watchdog-mailcow-1   | Sun Oct 27 07:53:49 NZDT 2024 Rspamd health level: 100% (5/5), health trend: 0
sogo-mailcow-1       | Oct 27 07:53:50 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
watchdog-mailcow-1   | Sun Oct 27 07:53:50 NZDT 2024 Olefy health level: 100% (5/5), health trend: 0
watchdog-mailcow-1   | Sun Oct 27 07:53:51 NZDT 2024 Postfix health level: 100% (8/8), health trend: 0
sogo-mailcow-1       | Oct 27 07:53:51 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
dovecot-mailcow-1    | Oct 27 07:53:52 f446bcac59bf dovecot: lmtp(5375): Connect from 172.22.1.5
dovecot-mailcow-1    | Oct 27 07:53:52 f446bcac59bf dovecot: auth: Error: auth-worker: Aborted USER request for watchdog@invalid: Worker process died unexpectedly
dovecot-mailcow-1    | Oct 27 07:53:52 f446bcac59bf dovecot: lmtp(watchdog@invalid)<5375><2ChSDMA6HWf/FAAANQEoeQ>: Error: auth-master: userdb lookup(watchdog@invalid): Auth USER lookup failed
dovecot-mailcow-1    | Oct 27 07:53:52 f446bcac59bf dovecot: lmtp(5375): Error: lmtp-server: conn 172.22.1.5:59048 [38]: rcpt watchdog@invalid: Failed to lookup user watchdog@invalid: Internal error occurred. Refer to server log for more information.
dovecot-mailcow-1    | Oct 27 07:53:52 f446bcac59bf dovecot: lmtp(5375): Disconnect from 172.22.1.5: Logged out (state=MAIL FROM)
dovecot-mailcow-1    | Oct 27 07:53:52 f446bcac59bf dovecot: imap-login: Disconnected: Aborted login by logging out (no auth attempts in 0 secs): user=<>, rip=172.22.1.5, lip=172.22.1.250, TLS, TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
dovecot-mailcow-1    | Oct 27 07:53:52 f446bcac59bf dovecot: imap-login: Disconnected: Aborted login by logging out (no auth attempts in 0 secs): user=<>, rip=172.22.1.5, lip=172.22.1.250
dovecot-mailcow-1    | Oct 27 07:53:52 f446bcac59bf dovecot: managesieve-login: Disconnected: Connection closed (no auth attempts in 0 secs): user=<>, rip=172.22.1.5, lip=172.22.1.250
watchdog-mailcow-1   | Sun Oct 27 07:53:52 NZDT 2024 Dovecot health level: 17% (2/12), health trend: -1
watchdog-mailcow-1   | Sun Oct 27 07:53:52 NZDT 2024 SOGo health level: 33% (1/3), health trend: -2
sogo-mailcow-1       | Oct 27 07:53:53 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:53:54 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:53:56 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:53:57 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:53:59 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!
sogo-mailcow-1       | Oct 27 07:54:00 d3cade719bc7 sogod [9]: [ERROR] <0x0x5c88534d6bb0[WOWatchDog]> No child available to handle incoming request!

SimplyCorbett

“Some disk problems” followed with mailcow issues..

I’ll be honest I stopped reading at this point. Wipe mailcow entirely and reinstall and then restore from your last full mailcow backup (that worked).

Good luck! You should have weekly backups for these situations.

marshalleq

Seems like you didn’t read properly. Have you ever used ZFS?

SimplyCorbett

marshalleq ZFS is not a backup solution. RAID ≠ backup.

Mailcow has a fancy backup script that copies literally everything from the admin settings, the mail, everything.

If you dont have backups idk.

Actually even just having a cronjob setup to run a weekly backup and pipe the output into a directory on the same ZFS array would have likely been enough for this issue, although you really should be saving it to a different host.

marshalleq

Dude, you haven’t read properly. a) The disk issues are on different disks on a different controller. I have a whole other thread about this over on truenas if you want to fault find that for me. 😉. If you know ZFS you know it actually tells you if there are issues with data too and names any corrupted files, which the mail server doesn’t have any / never had any because it’s disks are not having issues. b) And answering your final idea, I have actually completely backed this up, restored it to a new host, which is also mentioned above. Where did you get the idea I don’t have backups?

SimplyCorbett

marshalleq But if you have backups from a working copy they should also work when restored. So.. you may want to be raising an issue on github if you feel its a problem with a mailcow update.

marshalleq

Where did I say it was a problem with a mailcow update? All I’m posting here for is to see if anyone knows what these logs are saying and if they’ve come across it / can offer help. Github is generally not a support forum, so I’d want to be confident it is actually a code bug before posting there.

SimplyCorbett

Honestly if you are restoring known good working backups and running into this issue and it is not a mailcow update causing it then the only other thing I can think of is you have a unstable system.

Worker process died unexpectedly

Does not fill me with confidence that your system is entirely stable.

marshalleq

OK thanks, in your experience, what causes Worker Process died unexpectedly?

esackbauer

I had once a weird issue on my firewall, where processes kept crashing and producing core dumps.
No indication in the log files of any hardware error or related issue.
After a lot of troubleshooting it turned out a be a defective memory module. My firewall was not using ECC memory.
I suggest your server has ECC memory installed? anything in the BIOS logs regarding memory errors?

marshalleq

I have done extensive memory testing on the box due to the disk issues. No memory errors showing at all. Some months back I switched servers and this new one doesn’t have ECC, it could be that, but it’s unlikely given the extensive memory testing I think. However, I may just look at getting some. Might be a bit cheaper given it’s DDR-4 and we’re at DDR-5 now I think.

SimplyCorbett

I dont think you ever said. How many active clients are using your server?

marshalleq

Good point, I didn’t say clients, but I did say 3 users. I’ve doubled the active Sogo workers anyway. In effect, me as one user I have maybe 5 mailboxes plus other the other 2 users have maybe 4 so I guess calling it 10 users and I have 3 devices, plus they will have a few too so perhaps that makes it like 30 users?

SimplyCorbett

Well, what happens if you restore your backup on some cheap $5 VPS from say hetzner?

Does it work as expected?

If it does not, try to restore from a -earlier- backup. And keep doing this until you find one that works.

At which point my assumption would be something in the backup corrupted during your hardware issues early on.

To clarify how are you restoring your backups and when you backed up your mailcow instance did you backup all?

You are restoring backups from the mailcow backup script and not ZFS snapshots, correct?

marshalleq

Assuming you’re replying to the last questions you asked, your reply makes no sense to me. I don’t understand the relationship between numbers of users and you giving me advice on backups. Did you mean to post this somewhere else? Are you just trying to get your moo level up or something?

SimplyCorbett

marshalleq Nah, I was asking about users because I wanted to know if you were running into resource exhaustion. With 3 users and 16GB RAM on the VM thats unlikely.