Pakets from private network to public network

Kkingfisher · Sep 20, 2022

We recently recognized that tcp packets from the private docker network being routed to external addresses. This is irritating to us. How can this be explained and avoided?

root@mail:/opt/mailcow-dockerized # tcpdump -n net 172.22.1.0/24 -i eth0
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
20:39:50.734995 IP 172.22.1.250.imap > 37.24.xxx.xxx.20631: Flags [R], seq 128516594, win 0, length 0
20:39:50.735015 IP 172.22.1.250.imap > 37.24.xxx.xxx.20631: Flags [R], seq 128516594, win 0, length 0
21:10:24.470528 IP 172.22.1.250.imap > 37.24.xxx.xxx.9917: Flags [R], seq 3298072508, win 0, length 0
21:10:24.470596 IP 172.22.1.250.imap > 37.24.xxx.xxx.ndl-tcp-ois-gw: Flags [R], seq 3847556523, win 0, length 0
21:10:24.470619 IP 172.22.1.250.imap > 37.24.xxx.xxx.9917: Flags [R], seq 3298072508, win 0, length 0
21:10:24.470637 IP 172.22.1.250.imap > 37.24.xxx.xxx.ndl-tcp-ois-gw: Flags [R], seq 3847556523, win 0, length 0

Masquerading is active according to attached output of nft list ruleset.

nft.txt

20kB

Kkingfisher · Sep 21, 2022

Here are the versions of active images

root@mail:/opt/mailcow-dockerized # docker images --format "table {{.Repository}}\t{{.Tag}}"
REPOSITORY              TAG
mailcow/dovecot         1.19
mailcow/sogo            1.111
mariadb                 10.5
memcached               alpine
redis                   6-alpine
nginx                   mainline-alpine
mailcow/clamd           1.54
mailcow/phpfpm          1.79
mailcow/olefy           1.10
mailcow/dockerapi       1.42
mailcow/acme            1.82
mailcow/unbound         1.16
mailcow/netfilter       1.48
mailcow/postfix         1.67
phpmyadmin/phpmyadmin   latest
busybox                 latest
mailcow/rspamd          1.90
mailcow/dovecot         1.162
mailcow/watchdog        1.96
mailcow/solr            1.8.1
mailcow/olefy           1.9
mailcow/dockerapi       1.41
debian                  bullseye-slim
mailcow/clamd           1.50
mcuadros/ofelia         latest
robbertkl/ipv6nat       latest

Kkingfisher · Sep 22, 2022

According to the ruleset from iptables connection from the private net to external addresses should not be possible. Ether it is a misconfiguration of mailcow (where, what?) or a kernel bug? Feels anyone capable to have a look at this?

Try yourself with your mailcow net and external interface and wait some minutes (traffic assumed): tcpdump -n net 172.22.1.0/24 -i eth0

root@mail:/opt/mailcow-dockerized # iptables-save
# Generated by iptables-save v1.8.4 on Thu Sep 22 08:25:06 2022
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.22.1.0/24 ! -o br-mailcow -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.22.1.249/32 -d 172.22.1.249/32 -p tcp -m tcp --dport 6379 -j MASQUERADE
-A POSTROUTING -s 172.22.1.3/32 -d 172.22.1.3/32 -p tcp -m tcp --dport 8983 -j MASQUERADE
-A POSTROUTING -s 172.22.1.6/32 -d 172.22.1.6/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A POSTROUTING -s 172.22.1.9/32 -d 172.22.1.9/32 -p tcp -m tcp --dport 3306 -j MASQUERADE
-A POSTROUTING -s 172.22.1.250/32 -d 172.22.1.250/32 -p tcp -m tcp --dport 12345 -j MASQUERADE
-A POSTROUTING -s 172.22.1.250/32 -d 172.22.1.250/32 -p tcp -m tcp --dport 4190 -j MASQUERADE
-A POSTROUTING -s 172.22.1.250/32 -d 172.22.1.250/32 -p tcp -m tcp --dport 995 -j MASQUERADE
-A POSTROUTING -s 172.22.1.250/32 -d 172.22.1.250/32 -p tcp -m tcp --dport 993 -j MASQUERADE
-A POSTROUTING -s 172.22.1.250/32 -d 172.22.1.250/32 -p tcp -m tcp --dport 143 -j MASQUERADE
-A POSTROUTING -s 172.22.1.250/32 -d 172.22.1.250/32 -p tcp -m tcp --dport 110 -j MASQUERADE
-A POSTROUTING -s 172.22.1.253/32 -d 172.22.1.253/32 -p tcp -m tcp --dport 587 -j MASQUERADE
-A POSTROUTING -s 172.22.1.253/32 -d 172.22.1.253/32 -p tcp -m tcp --dport 465 -j MASQUERADE
-A POSTROUTING -s 172.22.1.253/32 -d 172.22.1.253/32 -p tcp -m tcp --dport 25 -j MASQUERADE
-A POSTROUTING -s 172.22.1.14/32 -d 172.22.1.14/32 -p tcp -m tcp --dport 443 -j MASQUERADE
-A POSTROUTING -s 172.22.1.14/32 -d 172.22.1.14/32 -p tcp -m tcp --dport 80 -j MASQUERADE
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A DOCKER -i br-mailcow -j RETURN
-A DOCKER -i docker0 -j RETURN
-A DOCKER -d 127.0.0.1/32 ! -i br-mailcow -p tcp -m tcp --dport 7654 -j DNAT --to-destination 172.22.1.249:6379
-A DOCKER -d 127.0.0.1/32 ! -i br-mailcow -p tcp -m tcp --dport 18983 -j DNAT --to-destination 172.22.1.3:8983
-A DOCKER ! -i br-mailcow -p tcp -m tcp --dport 8080 -j DNAT --to-destination 172.22.1.6:80
-A DOCKER -d 127.0.0.1/32 ! -i br-mailcow -p tcp -m tcp --dport 13306 -j DNAT --to-destination 172.22.1.9:3306
-A DOCKER -d 127.0.0.1/32 ! -i br-mailcow -p tcp -m tcp --dport 19991 -j DNAT --to-destination 172.22.1.250:12345
-A DOCKER ! -i br-mailcow -p tcp -m tcp --dport 4190 -j DNAT --to-destination 172.22.1.250:4190
-A DOCKER ! -i br-mailcow -p tcp -m tcp --dport 995 -j DNAT --to-destination 172.22.1.250:995
-A DOCKER ! -i br-mailcow -p tcp -m tcp --dport 993 -j DNAT --to-destination 172.22.1.250:993
-A DOCKER ! -i br-mailcow -p tcp -m tcp --dport 143 -j DNAT --to-destination 172.22.1.250:143
-A DOCKER ! -i br-mailcow -p tcp -m tcp --dport 110 -j DNAT --to-destination 172.22.1.250:110
-A DOCKER ! -i br-mailcow -p tcp -m tcp --dport 587 -j DNAT --to-destination 172.22.1.253:587
-A DOCKER ! -i br-mailcow -p tcp -m tcp --dport 465 -j DNAT --to-destination 172.22.1.253:465
-A DOCKER ! -i br-mailcow -p tcp -m tcp --dport 25 -j DNAT --to-destination 172.22.1.253:25
-A DOCKER ! -i br-mailcow -p tcp -m tcp --dport 443 -j DNAT --to-destination 172.22.1.14:443
-A DOCKER ! -i br-mailcow -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.22.1.14:80
COMMIT
# Completed on Thu Sep 22 08:25:06 2022
# Generated by iptables-save v1.8.4 on Thu Sep 22 08:25:06 2022
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-USER - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-mailcow -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-mailcow -j DOCKER
-A FORWARD -i br-mailcow ! -o br-mailcow -j ACCEPT
-A FORWARD -i br-mailcow -o br-mailcow -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 172.22.1.249/32 ! -i br-mailcow -o br-mailcow -p tcp -m tcp --dport 6379 -j ACCEPT
-A DOCKER -d 172.22.1.3/32 ! -i br-mailcow -o br-mailcow -p tcp -m tcp --dport 8983 -j ACCEPT
-A DOCKER -d 172.22.1.6/32 ! -i br-mailcow -o br-mailcow -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER -d 172.22.1.9/32 ! -i br-mailcow -o br-mailcow -p tcp -m tcp --dport 3306 -j ACCEPT
-A DOCKER -d 172.22.1.250/32 ! -i br-mailcow -o br-mailcow -p tcp -m tcp --dport 12345 -j ACCEPT
-A DOCKER -d 172.22.1.250/32 ! -i br-mailcow -o br-mailcow -p tcp -m tcp --dport 4190 -j ACCEPT
-A DOCKER -d 172.22.1.250/32 ! -i br-mailcow -o br-mailcow -p tcp -m tcp --dport 995 -j ACCEPT
-A DOCKER -d 172.22.1.250/32 ! -i br-mailcow -o br-mailcow -p tcp -m tcp --dport 993 -j ACCEPT
-A DOCKER -d 172.22.1.250/32 ! -i br-mailcow -o br-mailcow -p tcp -m tcp --dport 143 -j ACCEPT
-A DOCKER -d 172.22.1.250/32 ! -i br-mailcow -o br-mailcow -p tcp -m tcp --dport 110 -j ACCEPT
-A DOCKER -d 172.22.1.253/32 ! -i br-mailcow -o br-mailcow -p tcp -m tcp --dport 587 -j ACCEPT
-A DOCKER -d 172.22.1.253/32 ! -i br-mailcow -o br-mailcow -p tcp -m tcp --dport 465 -j ACCEPT
-A DOCKER -d 172.22.1.253/32 ! -i br-mailcow -o br-mailcow -p tcp -m tcp --dport 25 -j ACCEPT
-A DOCKER -d 172.22.1.14/32 ! -i br-mailcow -o br-mailcow -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d 172.22.1.14/32 ! -i br-mailcow -o br-mailcow -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-mailcow ! -o br-mailcow -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-USER -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o br-mailcow -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
COMMIT
# Completed on Thu Sep 22 08:25:06 2022
# Warning: iptables-legacy tables present, use iptables-legacy-save to see them

Kkingfisher · Sep 23, 2022

Someone?

Ggnasch · Sep 23, 2022

Happens here too, although rare:
you can correlate the ip’s in the tcpdump against your postfix and netfilter logs in the mailcow admin ui.
first occurrence was refusal of a mail delivery by dns blacklist,
second one was a timeout after auth.
Please post your results.
Thx,
gnasch

Kkingfisher · Sep 23, 2022

gnasch

I have double checked the logs. There are no direct correlations between the timestamps of the tcpdump entries and those of Netfilter, Postfix or Dovecot.

Strange that you have direct correlations.