ipv6, olefy, and clamd errors after update

cancan

Hello All,

I just ran the update.sh and now I am getting these error emails:

ipv6-config:
enable_ipv6 is true in docker-compose.yml, but an IPv6 link could not be established. Please verify your IPv6 connection.

olefy-mailcow:
CRITICAL - Socket timeout
CRITICAL - Socket timeout
CRITICAL - Socket timeout
CRITICAL - Socket timeout
CRITICAL - Socket timeout
CRITICAL - Socket timeout
CRITICAL - Socket timeout
CRITICAL - Socket timeout
CRITICAL - Socket timeout

clamd-mailcow:
CRITICAL - Socket timeout
CRITICAL - Socket timeout
CRITICAL - Socket timeout
CRITICAL - Socket timeout
CRITICAL - Socket timeout
CRITICAL - Socket timeout
CRITICAL - Socket timeout
CRITICAL - Socket timeout

Anyone know off the bat why this is happening?

maxileith

Hello,

have you found out what the cause of this is? I seem to have the same problem.

cancan

No, never did. Ended up reinstalling as I didnt have much mail on the server..

diekuh

Guys, please try to post some details. More logs, system information etc. 🙂

arhi

mailcowdockerized_clamd-mailcow_1 exited with code 143
clamd-mailcow_1      |   File: /var/lib/clamav/whitelist.ign2
clamd-mailcow_1      |   Size: 190              Blocks: 8          IO Block: 4096   regular file
clamd-mailcow_1      | Device: 31h/49d  Inode: 14942450    Links: 1
clamd-mailcow_1      | Access: (0644/-rw-r--r--)  Uid: (  700/  clamav)   Gid: (  700/  clamav)
clamd-mailcow_1      | Access: 2021-04-12 08:46:25.316548929 +0200
clamd-mailcow_1      | Modify: 2021-04-12 09:01:03.906915012 +0200
clamd-mailcow_1      | Change: 2021-04-12 09:01:03.910915064 +0200
clamd-mailcow_1      |  Birth: -
clamd-mailcow_1      | dos2unix: converting file /var/lib/clamav/whitelist.ign2 to Unix format...
clamd-mailcow_1      | Running freshclam...
clamd-mailcow_1      | Mon Apr 12 09:01:03 2021 -> ClamAV update process started at Mon Apr 12 09:01:03 2021
clamd-mailcow_1      | Mon Apr 12 09:01:03 2021 -> ^Your ClamAV installation is OUTDATED!
clamd-mailcow_1      | Mon Apr 12 09:01:03 2021 -> ^Local version: 0.103.1 Recommended version: 0.103.2
clamd-mailcow_1      | Mon Apr 12 09:01:03 2021 -> DON'T PANIC! Read https://www.clamav.net/documents/upgrading-clamav
clamd-mailcow_1      | Mon Apr 12 09:01:03 2021 -> daily.cld database is up to date (version: 26137, sigs: 3969790, f-level: 63, builder: raynman)
clamd-mailcow_1      | Mon Apr 12 09:01:03 2021 -> main.cvd database is up to date (version: 59, sigs: 4564902, f-level: 60, builder: sigmgr)
clamd-mailcow_1      | Mon Apr 12 09:01:03 2021 -> bytecode.cvd database is up to date (version: 333, sigs: 92, f-level: 63, builder: awillia2)
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Limits: Global time limit set to 120000 milliseconds.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Limits: Global size limit set to 52428800 bytes.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Limits: File size limit set to 26214400 bytes.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Limits: Recursion level limit set to 5.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Limits: Files limit set to 200.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Limits: MaxEmbeddedPE limit set to 10485760 bytes.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Limits: MaxHTMLNormalize limit set to 10485760 bytes.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Limits: MaxHTMLNoTags limit set to 2097152 bytes.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Limits: MaxScriptNormalize limit set to 5242880 bytes.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Limits: MaxZipTypeRcg limit set to 1048576 bytes.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Limits: MaxPartitions limit set to 50.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Limits: MaxIconsPE limit set to 100.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Limits: MaxRecHWP3 limit set to 16.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Limits: PCREMatchLimit limit set to 100000.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Limits: PCRERecMatchLimit limit set to 2000.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Limits: PCREMaxFileSize limit set to 26214400.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Archive support enabled.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> AlertExceedsMax heuristic detection disabled.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Heuristic alerts enabled.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Portable Executable support enabled.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> ELF support enabled.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Mail files support enabled.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> OLE2 support enabled.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> PDF support enabled.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> SWF support enabled.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> HTML support enabled.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> XMLDOCS support enabled.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> HWP3 support enabled.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Heuristic: precedence enabled
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Self checking every 3600 seconds.
clamd-mailcow_1      | Mon Apr 12 09:01:14 2021 -> Set stacksize to 8454144
mailcowdockerized_olefy-mailcow_1 exited with code 137
olefy-mailcow_1      | olefy INFO <module> olefy listen address: ['0.0.0.0'] (type: <class 'list'>)
olefy-mailcow_1      | olefy INFO <module> olefy listen port: 10055
olefy-mailcow_1      | olefy INFO <module> olefy tmp dir: /tmp
olefy-mailcow_1      | olefy INFO <module> olefy python path: /usr/bin/python3
olefy-mailcow_1      | olefy INFO <module> olefy olvba path: /usr/bin/olevba3
olefy-mailcow_1      | olefy INFO <module> olefy log level: 20
olefy-mailcow_1      | olefy INFO <module> olefy min file length: 500
olefy-mailcow_1      | olefy INFO <module> olefy delete tmp file: 1
olefy-mailcow_1      | olefy INFO <module> olefy delete tmp file when failed: 1
olefy-mailcow_1      | olefy INFO <module> serving on ('0.0.0.0', 10055)
clamd-mailcow_1      | Mon Apr 12 09:10:04 2021 -> instream(172.22.1.7@57846): OK
clamd-mailcow_1      | receiving incremental file list
clamd-mailcow_1      | ./
clamd-mailcow_1      |
clamd-mailcow_1      | sent 220 bytes  received 255 bytes  316.67 bytes/sec
clamd-mailcow_1      | total size is 16,674,854  speedup is 35,104.96
clamd-mailcow_1      | RELOADING
mailcowdockerized_olefy-mailcow_1 exited with code 137
olefy-mailcow_1      | olefy INFO <module> olefy listen address: ['0.0.0.0'] (type: <class 'list'>)
olefy-mailcow_1      | olefy INFO <module> olefy listen port: 10055
olefy-mailcow_1      | olefy INFO <module> olefy tmp dir: /tmp
olefy-mailcow_1      | olefy INFO <module> olefy python path: /usr/bin/python3
olefy-mailcow_1      | olefy INFO <module> olefy olvba path: /usr/bin/olevba3
olefy-mailcow_1      | olefy INFO <module> olefy log level: 20
olefy-mailcow_1      | olefy INFO <module> olefy min file length: 500
olefy-mailcow_1      | olefy INFO <module> olefy delete tmp file: 1
olefy-mailcow_1      | olefy INFO <module> olefy delete tmp file when failed: 1
olefy-mailcow_1      | olefy INFO <module> serving on ('0.0.0.0', 10055)

maxileith

Hello,

I have now managed to fix the CLAMD and Olefy problem. I don’t know why, but after an update the old CLAMD and Olefy containers were not shut down and not deleted. CLAMD and Olefy were running in two different versions at the same time. The logs did not show any errors. After I manually terminated and deleted the containers, everything worked normally.

About the IPv6 problem: The problem arose after I fixed this problem: Post

arhi

getting watchdog email non stop about olefy and clamd “socket timeout” .. any debug info you need just lemme know how to get it out (I know nothing about docker)

arhi

docker-compose down
systemctl restart docker
docker-compose down
docker-compose up -d

didn’t help anything
then following everything from documentation to disable ipv6:
https://mailcow.github.io/mailcow-dockerized-docs/firststeps-disable_ipv6/

vi docker-compose.yml
touch docker-compose.override.yml
vi docker-compose.override.yml
docker-compose down
systemctl stop docker
docker-compose up -d
vi data/conf/unbound/unbound.conf
docker-compose restart unbound-mailcow
vi data/conf/postfix/extra.cf
docker-compose restart postfix-mailcow

no help 🙁 I still get emails from watchdog about olefy and clamd

[root@mailcow mailcow-dockerized]# docker-compose logs --tail=1 -f clamd-mailcow
Attaching to mailcowdockerized_clamd-mailcow_1
clamd-mailcow_1      | Mon Apr 12 09:23:51 2021 -> Set stacksize to 8454144
clamd-mailcow_1      | Mon Apr 12 09:30:53 2021 -> instream(172.22.1.19@38944): OK
clamd-mailcow_1      |
clamd-mailcow_1      | receiving incremental file list
clamd-mailcow_1      | ./
clamd-mailcow_1      | blurl.ndb
clamd-mailcow_1      | junk.ndb
clamd-mailcow_1      | jurlbl.ndb
clamd-mailcow_1      | phish.ndb
clamd-mailcow_1      | phishtank.ndb
clamd-mailcow_1      | rogue.hdb
clamd-mailcow_1      | sanesecurity.ftm
clamd-mailcow_1      | scam.ndb
clamd-mailcow_1      | sigwhitelist.ign2
clamd-mailcow_1      | spamimg.hdb
clamd-mailcow_1      |
clamd-mailcow_1      | sent 410 bytes  received 16,679,579 bytes  4,765,711.14 bytes/sec
clamd-mailcow_1      | total size is 16,674,854  speedup is 1.00
clamd-mailcow_1      | RELOADING
clamd-mailcow_1      | Mon Apr 12 09:33:44 2021 -> Reading databases from /var/lib/clamav
clamd-mailcow_1      | Mon Apr 12 09:33:55 2021 -> Database correctly reloaded (8657511 signatures)
clamd-mailcow_1      | Mon Apr 12 09:33:55 2021 -> Database reload completed.
clamd-mailcow_1      | Mon Apr 12 09:33:55 2021 -> Activating the newly loaded database...
clamd-mailcow_1      | Mon Apr 12 09:33:56 2021 -> Reading databases from /var/lib/clamav
clamd-mailcow_1      | Mon Apr 12 09:34:08 2021 -> Database correctly reloaded (8657511 signatures)
clamd-mailcow_1      | Mon Apr 12 09:34:08 2021 -> Database reload completed.
clamd-mailcow_1      | Mon Apr 12 09:34:08 2021 -> Activating the newly loaded database...
mailcowdockerized_clamd-mailcow_1 exited with code 143
[root@mailcow mailcow-dockerized]#

and in second shell

[root@mailcow mailcow-dockerized]# docker-compose logs --tail=1 -f olefy-mailcow
Attaching to mailcowdockerized_olefy-mailcow_1
olefy-mailcow_1      | olefy INFO <module> serving on ('0.0.0.0', 10055)
mailcowdockerized_olefy-mailcow_1 exited with code 137
[root@mailcow mailcow-dockerized]#

so clamd is dying with code 143 and olefy is dying with code 137

no clue how to debug this further …

config changes (only what’s in the ipv6 documentation):

[root@mailcow mailcow-dockerized]# git diff
diff --git a/data/conf/postfix/main.cf b/data/conf/postfix/main.cf
index 8f5886a..70e9097 100644
--- a/data/conf/postfix/main.cf
+++ b/data/conf/postfix/main.cf
@@ -200,4 +200,5 @@ parent_domain_matches_subdomains = debug_peer_list,fast_flush_domains,mynetworks
 # User overrides #

 myhostname = mailcow.mysql.rs
-
+smtp_address_preference = ipv4
+inet_protocols = ipv4
diff --git a/data/conf/unbound/unbound.conf b/data/conf/unbound/unbound.conf
index 27110c0..90c16ee 100644
--- a/data/conf/unbound/unbound.conf
+++ b/data/conf/unbound/unbound.conf
@@ -4,7 +4,7 @@ server:
   interface: ::0
   logfile: /dev/console
   do-ip4: yes
-  do-ip6: yes
+  do-ip6: no
   do-udp: yes
   do-tcp: yes
   do-daemonize: no
diff --git a/docker-compose.yml b/docker-compose.yml
index 235af68..5d41e5e 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -597,7 +597,7 @@ networks:
     driver: bridge
     driver_opts:
       com.docker.network.bridge.name: br-mailcow
-    enable_ipv6: true
+    enable_ipv6: false
     ipam:
       driver: default
       config:
[root@mailcow mailcow-dockerized]#

arhi

[root@mailcow mailcow-dockerized]# free -m
              total        used        free      shared  buff/cache   available
Mem:          15884        4563        9410          16        1911       10977
Swap:          8191           0        8191
[root@mailcow mailcow-dockerized]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        7.8G     0  7.8G   0% /dev
tmpfs           7.8G     0  7.8G   0% /dev/shm
tmpfs           7.8G  9.9M  7.8G   1% /run
tmpfs           7.8G     0  7.8G   0% /sys/fs/cgroup
/dev/sda1      1000G   12G  938G   2% /
tmpfs           1.6G     0  1.6G   0% /run/user/0
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/70554b85a67f5049b897617b056fd217f33e2e5cea257ad8a10a84a42bd6b5cf/merged
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/a81489afb27424b3c4e9653cc9e7f7537be76cdb28031fb7e0efab9ce3f21a19/merged
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/b009fa3fe12d5fd95ad415b53211e50858d5d0565e6f0345be3932acc037713c/merged
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/77155cb2d9a6e481d041af33f2b475265c3ad815ed8d862cf560380bdd188ea8/merged
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/15269440362a9d8da588c452ddb546ab9b3aff94b68729063297bff5eed15b53/merged
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/0b003d02deda43c009aad912994a3193b6ac3ffa25249e0138f3b5882e8cb507/merged
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/05192abcca80759ae4d70b99c22ed8931563ff9e2be907e66dbb4bd7951ff6c0/merged
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/279fc28f1965e5187d21deb4f3448293ddcd47e4f85d9e1104357185b7f826bc/merged
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/14d01fe8da03f1dd0c3c3e4ba491c30edba0062cd6ae788df954285858511714/merged
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/beb637fdb6eede2854f80bfbee8149a95ad984f3ec4a038fd6846d337fd17463/merged
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/8a80f97c3cd1d43fe5f4217f68f1e9456b5259c067d5bc38453364da1fcd41be/merged
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/516515973ff32eeb92475a1d699ca56bb1a3d01a6b3fcc57897976d61b59aafe/merged
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/ce7a8e93222ed29ee7a5aa3eea28fdf011b6e200fca4ad49a2d5322b4b30bc84/merged
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/e7ac9c609ad19a06eff5e8ebd18e60144945807d26f31a0f5fb74a587a5f9fee/merged
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/fde9129e9bbb7cce26d6fc48354ae6d10cd0b772e119a7ccb8b5c683bbb01667/merged
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/9499fa68e66ae6d655ed94f8f3971857886befc93e3a89753d7751f93012fe92/merged
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/344aa9f51d77ccb43f0c34995838ea17e5949c5509ebbbee93a593a765f48b4e/merged
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/bdb7cbc54306d0a23057e172757a19d23226fde577778dde5a3ca9b0c77c9ca0/merged
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/33ffa08faa51184d7083399e79e3558a79edb3ff423cce439c3181ab0283d10a/merged
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/89297248b4740e67cf1d46a72df7079e7a5f6dda2354f39fd04d721dfeae4e1a/merged
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/ab37c71e744a6306b3819c13fe16c4052b84fe2f1997bc67430a9cec04a51710/merged
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/0befe051e0937796816244b05da19ab71610403ea6ccf8805526c4cad05f0a6e/merged
overlay        1000G   12G  938G   2% /var/lib/docker/overlay2/2f956d824f4d2ffeea0e04017d6b39447a8ff33d83033b183bf22fd6849d857e/merged
[root@mailcow mailcow-dockerized]#

arhi

All issues I have started when I made these changes:

[root@mailcow mailcow-dockerized]# git diff
diff --git a/data/conf/postfix/master.cf b/data/conf/postfix/master.cf
index ffd1ac4..8933280 100644
--- a/data/conf/postfix/master.cf
+++ b/data/conf/postfix/master.cf
@@ -1,10 +1,11 @@
 # inter-mx with postscreen on 25/tcp
 smtp       inet  n       -       n       -       1       postscreen
 10025      inet  n       -       n       -       1       postscreen
+2525       inet  n       -       -       -       -       smtpd
   -o postscreen_upstream_proxy_protocol=haproxy
   -o syslog_name=haproxy
 smtpd      pass  -       -       n       -       -       smtpd
-  -o smtpd_helo_restrictions=permit_mynetworks,reject_non_fqdn_helo_hostname
+  -o smtpd_helo_restrictions=permit_mynetworks
   -o smtpd_sasl_auth_enable=no
   -o smtpd_sender_restrictions=permit_mynetworks,reject_unlisted_sender,reject_unknown_sender_domain

diff --git a/docker-compose.yml b/docker-compose.yml
index 235af68..a6480d2 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -146,6 +146,7 @@ services:
         - SUBMISSION_PORT=${SUBMISSION_PORT:-587}
         - SMTPS_PORT=${SMTPS_PORT:-465}
         - SMTP_PORT=${SMTP_PORT:-25}
+        - SMTPx_PORT=${SMTPx_PORT:-2525}
         - XMPP_C2S_PORT=${XMPP_C2S_PORT:-5222}
         - XMPP_S2S_PORT=${XMPP_S2S_PORT:-5269}
         - API_KEY=${API_KEY:-invalid}
@@ -285,6 +286,7 @@ services:
         - NET_BIND_SERVICE
       ports:
         - "${SMTP_PORT:-25}:25"
+        - "${SMTPx_PORT:-2525}:2525"
         - "${SMTPS_PORT:-465}:465"
         - "${SUBMISSION_PORT:-587}:587"
       restart: always
@@ -597,7 +599,7 @@ networks:
     driver: bridge
     driver_opts:
       com.docker.network.bridge.name: br-mailcow
-    enable_ipv6: true
+    enable_ipv6: false
     ipam:
       driver: default
       config:
[root@mailcow mailcow-dockerized]#

I did docker-compose up -d after these changes, it rebuilt some containers and started complainig about clamd and olefy … reverted all changes back but .. problems are not going away

arhi

With great help from chris on tg this seems to be solved. I don’t know the docker fu and that was the biggest fear in migrating to mailcow so I’m not sure what really happened and how but this is how it was solved

docker-compose down complained it cannot something something network

[root@mailcow mailcow-dockerized]# docker-compose down
Stopping mailcowdockerized_netfilter-mailcow_1 ... done
Stopping mailcowdockerized_rspamd-mailcow_1    ... done
Stopping mailcowdockerized_acme-mailcow_1      ... done
Stopping mailcowdockerized_nginx-mailcow_1     ... done
Stopping mailcowdockerized_dovecot-mailcow_1   ... done
Stopping mailcowdockerized_postfix-mailcow_1   ... done
Stopping mailcowdockerized_php-fpm-mailcow_1   ... done
Stopping mailcowdockerized_mysql-mailcow_1     ... done
Stopping mailcowdockerized_watchdog-mailcow_1  ... done
Stopping mailcowdockerized_redis-mailcow_1     ... done
Stopping mailcowdockerized_memcached-mailcow_1 ... done
Stopping mailcowdockerized_unbound-mailcow_1   ... done
Stopping mailcowdockerized_dockerapi-mailcow_1 ... done
Stopping mailcowdockerized_ejabberd-mailcow_1  ... done
Stopping mailcowdockerized_sogo-mailcow_1      ... done
Stopping mailcowdockerized_olefy-mailcow_1     ... done
Stopping mailcowdockerized_solr-mailcow_1      ... done
Stopping mailcowdockerized_clamd-mailcow_1     ... done
Removing mailcowdockerized_ipv6nat-mailcow_1   ... done
Removing mailcowdockerized_netfilter-mailcow_1 ... done
Removing mailcowdockerized_rspamd-mailcow_1    ... done
Removing mailcowdockerized_acme-mailcow_1      ... done
Removing mailcowdockerized_nginx-mailcow_1     ... done
Removing mailcowdockerized_dovecot-mailcow_1   ... done
Removing mailcowdockerized_postfix-mailcow_1   ... done
Removing mailcowdockerized_php-fpm-mailcow_1   ... done
Removing mailcowdockerized_mysql-mailcow_1     ... done
Removing mailcowdockerized_watchdog-mailcow_1  ... done
Removing mailcowdockerized_redis-mailcow_1     ... done
Removing mailcowdockerized_memcached-mailcow_1 ... done
Removing mailcowdockerized_unbound-mailcow_1   ... done
Removing mailcowdockerized_dockerapi-mailcow_1 ... done
Removing mailcowdockerized_ejabberd-mailcow_1  ... done
Removing mailcowdockerized_sogo-mailcow_1      ... done
Removing mailcowdockerized_olefy-mailcow_1     ... done
Removing mailcowdockerized_solr-mailcow_1      ... done
Removing mailcowdockerized_clamd-mailcow_1     ... done
Removing network mailcowdockerized_mailcow-network
ERROR: error while removing network: network mailcowdockerized_mailcow-network id eb786536748d6682573275b9d38e97c07a6657256be1986b4588bdb8ac33d615 has active endpoints

so this “ERROR: error while removing network: network mailcowdockerized_mailcow-network id eb786536748d6682573275b9d38e97c07a6657256be1986b4588bdb8ac33d615 has active endpoints” gave Chris idea to ask for docker ps so I did

[root@mailcow mailcow-dockerized]# docker ps
CONTAINER ID   IMAGE                    COMMAND                  CREATED        STATUS             PORTS       NAMES
b6f0b7756109   mailcow/watchdog:1.91    "/bin/sh -c '/watchd…"   12 hours ago   Up About an hour               mailcow-dockerized_watchdog-mailcow_2
90cd1e59b908   mailcow/dockerapi:1.38   "python3 -u /app/doc…"   12 hours ago   Up About an hour               mailcow-dockerized_dockerapi-mailcow_2
e7343936c4ab   memcached:alpine         "docker-entrypoint.s…"   12 hours ago   Up About an hour   11211/tcp   mailcow-dockerized_memcached-mailcow_2
cc59cb867dc4   mailcow/olefy:1.7        "python3 -u /app/ole…"   12 hours ago   Up About an hour               mailcow-dockerized_olefy-mailcow_2
632cdbda35c3   mailcow/clamd:1.39       "/sbin/tini -g -- /c…"   12 hours ago   Up About an hour               mailcow-dockerized_clamd-mailcow_2

and here we see some *_2 containers that should not exist. How they came to be no one knows, he mentioned “swarm” but I never did anything similar so … why/how no clue but to fix this these needed to be nuked so:

[root@mailcow mailcow-dockerized]# docker stop b6f0b7756109
b6f0b7756109
[root@mailcow mailcow-dockerized]# docker rm b6f0b7756109
b6f0b7756109
[root@mailcow mailcow-dockerized]# docker stop 90cd1e59b908
90cd1e59b908
[root@mailcow mailcow-dockerized]# docker rm 90cd1e59b908
90cd1e59b908
[root@mailcow mailcow-dockerized]# docker stop e7343936c4ab
e7343936c4ab
[root@mailcow mailcow-dockerized]# docker rm e7343936c4ab
e7343936c4ab
[root@mailcow mailcow-dockerized]# docker stop cc59cb867dc4
cc59cb867dc4
[root@mailcow mailcow-dockerized]# docker rm cc59cb867dc4
cc59cb867dc4
[root@mailcow mailcow-dockerized]# docker stop 632cdbda35c3
632cdbda35c3
[root@mailcow mailcow-dockerized]# docker rm 632cdbda35c3
632cdbda35c3
[root@mailcow mailcow-dockerized]# docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
[root@mailcow mailcow-dockerized]#

now, stop works great and start gets us where we need to be

[root@mailcow mailcow-dockerized]# docker-compose down --remove-orphans
Removing network mailcowdockerized_mailcow-network
[root@mailcow mailcow-dockerized]# docker-compose up -d
Creating network "mailcowdockerized_mailcow-network" with driver "bridge"
Creating mailcowdockerized_watchdog-mailcow_1  ... done
Creating mailcowdockerized_sogo-mailcow_1      ... done
Creating mailcowdockerized_clamd-mailcow_1     ... done
Creating mailcowdockerized_ejabberd-mailcow_1  ... done
Creating mailcowdockerized_dockerapi-mailcow_1 ... done
Creating mailcowdockerized_redis-mailcow_1     ... done
Creating mailcowdockerized_olefy-mailcow_1     ... done
Creating mailcowdockerized_unbound-mailcow_1   ... done
Creating mailcowdockerized_memcached-mailcow_1 ... done
Creating mailcowdockerized_solr-mailcow_1      ... done
Creating mailcowdockerized_mysql-mailcow_1     ... done
Creating mailcowdockerized_php-fpm-mailcow_1   ... done
Creating mailcowdockerized_postfix-mailcow_1   ... done
Creating mailcowdockerized_dovecot-mailcow_1   ... done
Creating mailcowdockerized_nginx-mailcow_1     ... done
Creating mailcowdockerized_acme-mailcow_1      ... done
Creating mailcowdockerized_rspamd-mailcow_1    ... done
Creating mailcowdockerized_netfilter-mailcow_1 ... done
Creating mailcowdockerized_ipv6nat-mailcow_1   ... done
[root@mailcow mailcow-dockerized]# docker ps
CONTAINER ID   IMAGE                    COMMAND                  CREATED          STATUS          PORTS                                                                                                                                        NAMES
c930ad5cfc56   mailcow/netfilter:1.41   "python3 -u /server.…"   13 seconds ago   Up 13 seconds                                                                                                                                                mailcowdockerized_netfilter-mailcow_1
b32c94a10353   mailcow/rspamd:1.76      "/docker-entrypoint.…"   13 seconds ago   Up 13 seconds                                                                                                                                                mailcowdockerized_rspamd-mailcow_1
59a6a072e70b   mailcow/acme:1.78        "/sbin/tini -g -- /s…"   13 seconds ago   Up 13 seconds                                                                                                                                                mailcowdockerized_acme-mailcow_1
effd20ae0e03   nginx:mainline-alpine    "/docker-entrypoint.…"   14 seconds ago   Up 11 seconds   0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                                                                                                     mailcowdockerized_nginx-mailcow_1
723f213af5a7   mailcow/postfix:1.61     "/docker-entrypoint.…"   15 seconds ago   Up 14 seconds   0.0.0.0:25->25/tcp, 0.0.0.0:465->465/tcp, 0.0.0.0:587->587/tcp, 588/tcp                                                                      mailcowdockerized_postfix-mailcow_1
268cfb1099ad   mailcow/dovecot:1.145    "/docker-entrypoint.…"   15 seconds ago   Up 13 seconds   0.0.0.0:110->110/tcp, 0.0.0.0:143->143/tcp, 0.0.0.0:993->993/tcp, 0.0.0.0:995->995/tcp, 0.0.0.0:4190->4190/tcp, 127.0.0.1:19991->12345/tcp   mailcowdockerized_dovecot-mailcow_1
bdd3b2a0c8fa   mailcow/phpfpm:1.75      "/docker-entrypoint.…"   15 seconds ago   Up 14 seconds   9000/tcp                                                                                                                                     mailcowdockerized_php-fpm-mailcow_1
a2bb9f5228a9   mariadb:10.5             "docker-entrypoint.s…"   15 seconds ago   Up 15 seconds   127.0.0.1:13306->3306/tcp                                                                                                                    mailcowdockerized_mysql-mailcow_1
6a35a3393266   mailcow/solr:1.7         "docker-entrypoint.s…"   17 seconds ago   Up 15 seconds   127.0.0.1:18983->8983/tcp                                                                                                                    mailcowdockerized_solr-mailcow_1
6919596827fc   memcached:alpine         "docker-entrypoint.s…"   17 seconds ago   Up 15 seconds   11211/tcp                                                                                                                                    mailcowdockerized_memcached-mailcow_1
40882d80925f   redis:5-alpine           "docker-entrypoint.s…"   17 seconds ago   Up 15 seconds   127.0.0.1:7654->6379/tcp                                                                                                                     mailcowdockerized_redis-mailcow_1
d0eae2aa8753   mailcow/olefy:1.7        "python3 -u /app/ole…"   17 seconds ago   Up 15 seconds                                                                                                                                                mailcowdockerized_olefy-mailcow_1
80fddfd6709d   mailcow/unbound:1.13     "/docker-entrypoint.…"   17 seconds ago   Up 15 seconds   53/tcp, 53/udp                                                                                                                               mailcowdockerized_unbound-mailcow_1
ecfa1637a618   mailcow/dockerapi:1.38   "python3 -u /app/doc…"   17 seconds ago   Up 16 seconds                                                                                                                                                mailcowdockerized_dockerapi-mailcow_1
e9ba2ce9db55   mailcow/ejabberd:1.5     "/docker-entrypoint.…"   17 seconds ago   Up 15 seconds   1883/tcp, 0.0.0.0:5222->5222/tcp, 4369-4399/tcp, 0.0.0.0:5269->5269/tcp, 5280/tcp, 0.0.0.0:5443->5443/tcp                                    mailcowdockerized_ejabberd-mailcow_1
c53bbb1ce8d2   mailcow/watchdog:1.91    "/bin/sh -c '/watchd…"   17 seconds ago   Up 16 seconds                                                                                                                                                mailcowdockerized_watchdog-mailcow_1
f21844b96345   mailcow/sogo:1.99        "/docker-entrypoint.…"   17 seconds ago   Up 15 seconds                                                                                                                                                mailcowdockerized_sogo-mailcow_1
61bbf9056629   mailcow/clamd:1.39       "/sbin/tini -g -- /c…"   17 seconds ago   Up 15 seconds                                                                                                                                                mailcowdockerized_clamd-mailcow_1
[root@mailcow mailcow-dockerized]#

issue solved, no more dying clamav nor olefy

how this happened in the first place - still no clue, the history is rather clean:

basic setup
   59  yum install python3 python3-libs python3-pip python3-setuptools
   60  curl -L "https://github.com/docker/compose/releases/download/1.29.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
   61  docker-compose
   62  chmod a+x /usr/local/bin/docker-compose
   63  docker-compose
   64  docker-compose --version
   65  systemctl enable docker.service
   66  systemctl restart docker.service
   67  yum install container-selinux
   68  docker info | grep selinux
   69  docker info
   70  vi /etc/docker/daemon.json
   71  systemctl restart docker.service
   72  docker info
   73  docker info | grep selinux
   
   
install mailcow   
   74  cd /opt
   75  umask
   76  git clone https://github.com/mailcow/mailcow-dockerized
   77  cd mailcow-dockerized/
   78  ls -la
   79  ./generate_config.sh
   80  vi mailcow.conf
   81  docker-compose pull
   82  docker-compose up -d
   83  netstat -antu | grep 25
   84  yum whatprovides */netstat
   85  yum install net-tools
   86  netstat -antu | grep 25
   87  netstat -antup | grep 25
   88  ps ax | grep 1062
   89  rpm -qf /usr/libexec/postfix/master
   90  yum remove postfix
   91  systemctl status postfix
   92  ps ax | grep master
   93  docker-compose up -d
   94  docker volume ls
   95  docker-compose down
   96  docker-compose up -d
   97  docker volume ls
   98  cd /opt
   99  ls
  100  cd mailcow-dockerized/
  101  git pull
  102  ./update.sh
  103  ./update.sh --gc
  104  docker system prune
  105  docker volume ls
  106  sync
  

And now I $#%^^%&#^ it up:
  114  cd /opt/mailcow-dockerized/
  116  docker-compose restart acme-mailcow
  117  docker-compose logs --tail=200 -f acme-mailcow
  118  vi mailcow.conf
  119  git pull
  120  git status
  121  git pull
  122  docker-compose up -d
  123  docker-compose restart acme-mailcow
  146  cd data
  147  cd conf/
  150  cd postfix/
  154  vi main.cf
  157  vi master.cf
  162  vi allow_mailcow_local.regexp
  163  vi anonymize_headers.pcre
  164  vi custom_transport.pcre
  165  vu smtp_dsn_filter
  166  vi smtp_dsn_filter
  171  cd ..
  173  cd ..
  175  docker-compose up -d
  176  docker-compose logs --tail="all" -f postfix-mailcow
  177  docker-compose logs --tail="all" -f postfix-mailcow | grep getipass.com
  178  docker-compose logs --tail="all" -f postfix-mailcow | grep getipass
  179  docker-compose logs --tail="all" -f postfix-mailcow
  180  docker-compose logs --tail -f postfix-mailcow
  181  docker-compose logs --tail="all" -f postfix-mailcow
  182  docker-compose up -d
  183  docker-compose down
  184  systemctl restart docker
  185  docker-compose down
  186  docker-compose up -d
  187  docker-compose logs --tail="all" -f postfix-mailcow
  188  cd /opt/mailcow-dockerized/
  191  vi mailcow.conf
  192  vi docker-compose.yml
  193  vi mailcow.conf
  194  vi docker-compose.yml
  204  vi data/conf/postfix/custom_postscreen_whitelist.cidr
  205  docker-compose up -d
  213  systemctl status firewalld
  228  yum install bind-utils
  231  cd /opt/mailcow-dockerized/
  233  docker-compose logs --tail="all" -f postfix-mailcow
  234  docker-compose logs --tail="all" -f postfix-mailcow | grep xyz
  235  docker-compose logs --tail="all" | grep xyz
  239  docker-compose logs --tail="all" -f rspamd
  240  docker-compose logs --tail=1 postfix-mailcow dovecot-mailcow rspamd-mailcow
  241  docker-compose logs --tail=1 postfix-mailcow dovecot-mailcow rspamd-mailcow -f
  242  docker-compose logs --tail=1 -f postfix-mailcow dovecot-mailcow rspamd-mailcow
  256  docker-compose logs --tail=1 -f postfix-mailcow dovecot-mailcow rspamd-mailcow
  271  cd data/
  273  cd conf/postfix/
  275  vi master.cf
  277  vi master.cf
  278  docker-compose up -d
  284  firewall-cmd --list-all
  285  firewall-cmd --permanent --zone=public --add-port=2525/tcp
  286  firewall-cmd --permanent --zone=public --add-port=2526/tcp
  287  cd /opt/mailcow-dockerized/data/conf/postfix/
  289  vi main.cf
  290  vi master.cf
  291  firewall-cmd --permanent --zone=public --add-port=10025/tcp
  292  cd ..
  293  docker-compose up -d
  295  cd data/conf/postfix/
  297  vi main.cf
  298  vi master.cf
  300  cd ..
  302  cd ..
  304  vi docker-compose.yml
  305  docker-compose up -d
  307  vi data/conf/postfix/master.cf
  308  docker-compose up -d
  309  telnet localhost 2525
  310  vi data/conf/postfix/master.cf
  311  docker-compose up -d
  313  reboot
  317  cd /opt/mailcow-dockerized/
  318  docker-compose logs --tail=1 -f postfix-mailcow dovecot-mailcow rspamd-mailcow
  319  sync
  320  cd /opt/mailcow-dockerized/
  321  git diff
  326  shutdown -rf now
  331  cd /opt/mailcow-dockerized/


Trying to fix it:

  335  git checkout docker-compose.yml
  336  vi docker-compose.yml
  337  git checkout data/conf/postfix/master.cf
  338  docker-compose up -d
  339  git diff
  340  cd /opt/mailcow-dockerized/
  341  sync
  342  reboot
  343  w
  344  ps ax
  345  cd /opt/mailcow-dockerized/
  346  git pull
  347  firewall-cmd --list-all
  348  git pull
  349  docker-compose logs --tail=1 -f clamd-mailcow
  350  docker-compose logs --tail=1 -f clamd-mailcow olefy-mailcow
  351  git diff
  352  vi docker-compose.yml
  353  git diff
  354  ls
  355  ./update.sh
  356  docker-compose up -d
  357  docker-compose down
  358  systemctl restart docker
  359  docker-compose down
  360  docker-compose up -d
  361  docker-compose down
  362  reboot
  363  uptime
  364  date
  365  cd /opt/mailcow-dockerized/
  366  docker-compose down
  367  systemctl restart docker
  368  docker-compose down
  369  docker-compose up -d
  370  vi docker-compose.yml
  371  touch docker-compose.override.yml
  372  vi docker-compose.override.yml
  373  docker-compose down
  374  systemctl stop docker
  375  ps ax | grep dock
  376  docker-compose up -d
  377  vi data/conf/unbound/unbound.conf
  378  docker-compose restart unbound-mailcow
  379  vi data/conf/postfix/extra.cf
  380  docker-compose restart postfix-mailcow
  381  docker-compose logs --tail=1 -f clamd-mailcow
  382  history
  383  git diff
  384  free -m
  385  df -h
  386  dmesg
  387  tail -f /var/log/messages
  388  ps ax
  389  ps ax | grep ole
  390  yum install sysstat
  391  cd /var/log
  392  grep -i oom messages
  393  dmesg | grep -i oom
  394  dmesg | grep -i "out of memory"
  395  dmesg | grep -i "memory"
  396  vi /etc/selinux/config
  397  docker --version
  398  docker-compose --version
  399  containerd --version
  400  ctr --address /var/run/docker/containerd/containerd.sock version
  401  sync
  402  ls -la
  403  pwd


And finally, the fix:

  404  cd /opt/mailcow-dockerized/
  405  ls -la
  406  docker-compose down
  407  docker ps
  408  docker-compose down --remove-orphans
  409  docker ps
  410  docker stop b6f0b7756109
  411  docker rm b6f0b7756109
  412  docker stop 90cd1e59b908
  413  docker rm 90cd1e59b908
  414  docker stop e7343936c4ab
  415  docker rm e7343936c4ab
  416  docker stop cc59cb867dc4
  417  docker rm cc59cb867dc4
  418  docker stop 632cdbda35c3
  419  docker rm 632cdbda35c3
  420  docker ps
  421  docker-compose down --remove-orphans
  422  docker-compose up -d
  423  docker ps
  424  docker ps | grep _2
  425  docker-compose logs --tail=1 -f olefy-mailcow clamd-mailcow

arhi

attaching a whole history file for reference

save-history.txt

12kB

robertrud

Thank you for this. I had the same problem. It’s solved now.

arhi

robertrud you are welcome, but the real hero is the cn=chris on telegram without who I would never figure this out myself 😃

Luckily this was fresh install, single console used, so I have “full history” of what was done, so someone versed in docker can see how those duplicate instances happen since I never used anything wrt “scale/swarm/….” and write some warning “what never to do”. I suspect me stopping/starting docker and running docker-compose somehow made those _2 instances.

I know “some” about Linux (was sysadmin ages ago), but never used containers in any serious fashion so that was my biggest fear migrating to mailcow, anyhow, support on telegram is awesome so I feel much safer after this first problem was expressly solved.

Great tool, I’m super happy I migrated to mailcow… great community… best licence ( https://www.servercow.de/mailcow?lang=en#sal )