It has happened again. Here is the output from iptables while incoming ipv4 (only) traffic to port 25 is being blocked:
root@mailcow:~# sudo iptables -L -v -n --line-numbers
Chain INPUT (policy ACCEPT 1571K packets, 112M bytes)
num pkts bytes target prot opt in out source destination
1 1571K 112M MAILCOW 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* mailcow */
2 0 0 ACCEPT 17 -- eth0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:52821 /* wireguard-input-rule */
3 1571K 112M MAILCOW 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* mailcow */
4 1571K 112M MAILCOW 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* mailcow */
5 1571K 112M MAILCOW 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* mailcow */
6 1571K 112M MAILCOW 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* mailcow */
Chain FORWARD (policy DROP 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
1 70821 34M MAILCOW 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* mailcow */
2 55145 33M DOCKER-USER 0 -- * * 0.0.0.0/0 0.0.0.0/0
3 55145 33M DOCKER-FORWARD 0 -- * * 0.0.0.0/0 0.0.0.0/0
4 0 0 ACCEPT 0 -- eth0 wg0 0.0.0.0/0 10.123.67.0/24 ctstate RELATED,ESTABLISHED /* wireguard-forward-rule */
5 0 0 ACCEPT 0 -- wg0 eth0 10.123.67.0/24 0.0.0.0/0 /* wireguard-forward-rule */
6 0 0 MAILCOW 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* mailcow */
7 0 0 MAILCOW 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* mailcow */
8 0 0 MAILCOW 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* mailcow */
9 0 0 MAILCOW 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* mailcow */
10 0 0 MAILCOW 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* mailcow */
Chain OUTPUT (policy ACCEPT 195K packets, 6190M bytes)
num pkts bytes target prot opt in out source destination
Chain DOCKER (2 references)
num pkts bytes target prot opt in out source destination
1 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.253 tcp dpt:587
2 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.253 tcp dpt:465
3 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.253 tcp dpt:25
4 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.250 tcp dpt:12345
5 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.250 tcp dpt:4190
6 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.250 tcp dpt:995
7 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.250 tcp dpt:993
8 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.250 tcp dpt:143
9 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.250 tcp dpt:110
10 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.10 tcp dpt:443
11 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.10 tcp dpt:80
12 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.249 tcp dpt:6379
13 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.7 tcp dpt:3306
14 210 12600 ACCEPT 6 -- !docker0 docker0 0.0.0.0/0 172.17.0.2 tcp dpt:9001
15 0 0 DROP 0 -- !br-mailcow br-mailcow 0.0.0.0/0 0.0.0.0/0
16 0 0 DROP 0 -- !docker0 docker0 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-BRIDGE (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 DOCKER 0 -- * br-mailcow 0.0.0.0/0 0.0.0.0/0
2 210 12600 DOCKER 0 -- * docker0 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-CT (1 references)
num pkts bytes target prot opt in out source destination
1 12757 2063K ACCEPT 0 -- * br-mailcow 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
2 12418 2392K ACCEPT 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
Chain DOCKER-FORWARD (1 references)
num pkts bytes target prot opt in out source destination
1 55145 33M DOCKER-CT 0 -- * * 0.0.0.0/0 0.0.0.0/0
2 29970 29M DOCKER-ISOLATION-STAGE-1 0 -- * * 0.0.0.0/0 0.0.0.0/0
3 29970 29M DOCKER-BRIDGE 0 -- * * 0.0.0.0/0 0.0.0.0/0
4 18366 1365K ACCEPT 0 -- br-mailcow * 0.0.0.0/0 0.0.0.0/0
5 11394 27M ACCEPT 0 -- docker0 * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
num pkts bytes target prot opt in out source destination
1 18366 1365K DOCKER-ISOLATION-STAGE-2 0 -- br-mailcow !br-mailcow 0.0.0.0/0 0.0.0.0/0
2 11394 27M DOCKER-ISOLATION-STAGE-2 0 -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (2 references)
num pkts bytes target prot opt in out source destination
1 0 0 DROP 0 -- * docker0 0.0.0.0/0 0.0.0.0/0
2 0 0 DROP 0 -- * br-mailcow 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
num pkts bytes target prot opt in out source destination
1 55145 33M RETURN 0 -- * * 0.0.0.0/0 0.0.0.0/0
Chain MAILCOW (11 references)
num pkts bytes target prot opt in out source destination
1 0 0 DROP 0 -- * * 81.30.107.0/24 0.0.0.0/0
2 0 0 DROP 6 -- !br-mailcow br-mailcow 0.0.0.0/0 0.0.0.0/0 /* mailcow isolation */
3 0 0 DROP 0 -- * * 193.46.255.40 0.0.0.0/0
4 0 0 DROP 0 -- * * 81.30.107.0/24 0.0.0.0/0
5 15683 907K DROP 6 -- !br-mailcow br-mailcow 0.0.0.0/0 0.0.0.0/0 /* mailcow isolation */
Removing the duplicate blocking rule 5 from the chain MAILCOW removes the blockage:
root@mailcow:~# iptables -D MAILCOW 5
root@mailcow:~# sudo iptables -L -v -n --line-numbers
Chain INPUT (policy ACCEPT 1571K packets, 112M bytes)
num pkts bytes target prot opt in out source destination
1 1571K 112M MAILCOW 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* mailcow */
2 0 0 ACCEPT 17 -- eth0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:52821 /* wireguard-input-rule */
3 1571K 112M MAILCOW 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* mailcow */
4 1571K 112M MAILCOW 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* mailcow */
5 1571K 112M MAILCOW 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* mailcow */
6 1571K 112M MAILCOW 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* mailcow */
Chain FORWARD (policy DROP 0 packets, 0 bytes)
num pkts bytes target prot opt in out source destination
1 70835 34M MAILCOW 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* mailcow */
2 55155 33M DOCKER-USER 0 -- * * 0.0.0.0/0 0.0.0.0/0
3 55155 33M DOCKER-FORWARD 0 -- * * 0.0.0.0/0 0.0.0.0/0
4 0 0 ACCEPT 0 -- eth0 wg0 0.0.0.0/0 10.123.67.0/24 ctstate RELATED,ESTABLISHED /* wireguard-forward-rule */
5 0 0 ACCEPT 0 -- wg0 eth0 10.123.67.0/24 0.0.0.0/0 /* wireguard-forward-rule */
6 0 0 MAILCOW 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* mailcow */
7 0 0 MAILCOW 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* mailcow */
8 0 0 MAILCOW 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* mailcow */
9 0 0 MAILCOW 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* mailcow */
10 0 0 MAILCOW 0 -- * * 0.0.0.0/0 0.0.0.0/0 /* mailcow */
Chain OUTPUT (policy ACCEPT 195K packets, 6190M bytes)
num pkts bytes target prot opt in out source destination
Chain DOCKER (2 references)
num pkts bytes target prot opt in out source destination
1 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.253 tcp dpt:587
2 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.253 tcp dpt:465
3 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.253 tcp dpt:25
4 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.250 tcp dpt:12345
5 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.250 tcp dpt:4190
6 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.250 tcp dpt:995
7 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.250 tcp dpt:993
8 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.250 tcp dpt:143
9 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.250 tcp dpt:110
10 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.10 tcp dpt:443
11 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.10 tcp dpt:80
12 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.249 tcp dpt:6379
13 0 0 ACCEPT 6 -- !br-mailcow br-mailcow 0.0.0.0/0 172.22.1.7 tcp dpt:3306
14 210 12600 ACCEPT 6 -- !docker0 docker0 0.0.0.0/0 172.17.0.2 tcp dpt:9001
15 0 0 DROP 0 -- !br-mailcow br-mailcow 0.0.0.0/0 0.0.0.0/0
16 0 0 DROP 0 -- !docker0 docker0 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-BRIDGE (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 DOCKER 0 -- * br-mailcow 0.0.0.0/0 0.0.0.0/0
2 210 12600 DOCKER 0 -- * docker0 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-CT (1 references)
num pkts bytes target prot opt in out source destination
1 12757 2063K ACCEPT 0 -- * br-mailcow 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
2 12422 2392K ACCEPT 0 -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
Chain DOCKER-FORWARD (1 references)
num pkts bytes target prot opt in out source destination
1 55155 33M DOCKER-CT 0 -- * * 0.0.0.0/0 0.0.0.0/0
2 29976 29M DOCKER-ISOLATION-STAGE-1 0 -- * * 0.0.0.0/0 0.0.0.0/0
3 29976 29M DOCKER-BRIDGE 0 -- * * 0.0.0.0/0 0.0.0.0/0
4 18368 1365K ACCEPT 0 -- br-mailcow * 0.0.0.0/0 0.0.0.0/0
5 11398 27M ACCEPT 0 -- docker0 * 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
num pkts bytes target prot opt in out source destination
1 18368 1365K DOCKER-ISOLATION-STAGE-2 0 -- br-mailcow !br-mailcow 0.0.0.0/0 0.0.0.0/0
2 11398 27M DOCKER-ISOLATION-STAGE-2 0 -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-ISOLATION-STAGE-2 (2 references)
num pkts bytes target prot opt in out source destination
1 0 0 DROP 0 -- * docker0 0.0.0.0/0 0.0.0.0/0
2 0 0 DROP 0 -- * br-mailcow 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
num pkts bytes target prot opt in out source destination
1 55155 33M RETURN 0 -- * * 0.0.0.0/0 0.0.0.0/0
Chain MAILCOW (11 references)
num pkts bytes target prot opt in out source destination
1 0 0 DROP 0 -- * * 81.30.107.0/24 0.0.0.0/0
2 0 0 DROP 6 -- !br-mailcow br-mailcow 0.0.0.0/0 0.0.0.0/0 /* mailcow isolation */
3 0 0 DROP 0 -- * * 193.46.255.40 0.0.0.0/0
4 0 0 DROP 0 -- * * 81.30.107.0/24 0.0.0.0/0
As you see the original universal block rule 2 have no matches during normal funtional operation of mailcow. Its the duplicate one that is suddenly matching any traffic and removing it restores incoming access.