IPv6 after reboot doesn't work before restarting all containers

artemislena

T.: We’ve verified the system we’re running on generally has working IPv6. However, IPv6 internet access from within the mailcow containers doesn’t work initially (neither outbound (tested w ping and traceroute (doesn’t get past first hop, DNS resolution works tho) nor inbound (tested w curl and nc from external machine, but curling over IPv6 using the public address from local machine does work, so it’s not a socket binding issue)), it only works after restarting all containers. Kinda suspecting the Docker/mailcow-configured firewall (more on that in a moment), but just restarting the netfilter container ain’t enough for making’t work, we gotta do the full docker compose down and then docker compose up.

Reason y we think it’s a firewall issue: There’s a big difference between the iptables rules before and after restarting stuff, especially looking at how at first, there’s several references to docker0 before the restart that that after the restart seem’a be changed to br-mailcow.

root@mailcow:~/ > ip6tables -L -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  707  223K MAILCOW    all  --  any    any     anywhere             anywhere             /* mailcow */
  719  224K nixos-fw   all  --  any    any     anywhere             anywhere            

Chain FORWARD (policy DROP 203 packets, 17881 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  203 17881 MAILCOW    all  --  any    any     anywhere             anywhere             /* mailcow */
  203 17881 DOCKER-USER  all  --  any    any     anywhere             anywhere            
  203 17881 DOCKER-FORWARD  all  --  any    any     anywhere             anywhere            
    0     0 REJECT     all  --  wg0    ens4    anywhere             anywhere             reject-with icmp6-port-unreachable
    0     0 REJECT     all  --  wg0    lo      anywhere             anywhere             reject-with icmp6-port-unreachable
    0     0 REJECT     all  --  ens4   any     anywhere             anywhere             reject-with icmp6-port-unreachable

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  anywhere             fd4d:6169:6c63:6f77::12  tcp dpt:https
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  anywhere             fd4d:6169:6c63:6f77::12  tcp dpt:http
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  anywhere             fd4d:6169:6c63:6f77::11  tcp dpt:urd
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  anywhere             fd4d:6169:6c63:6f77::11  tcp dpt:smtp
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  anywhere             fd4d:6169:6c63:6f77::2  tcp dpt:imaps
    0     0 DROP       all  --  !docker0 docker0  anywhere             anywhere            

Chain DOCKER-BRIDGE (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all  --  any    docker0  anywhere             anywhere            

Chain DOCKER-CT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     all  --  any    docker0  anywhere             anywhere             ctstate RELATED,ESTABLISHED

Chain DOCKER-FORWARD (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  203 17881 DOCKER-CT  all  --  any    any     anywhere             anywhere            
  203 17881 DOCKER-ISOLATION-STAGE-1  all  --  any    any     anywhere             anywhere            
  203 17881 DOCKER-BRIDGE  all  --  any    any     anywhere             anywhere            
    0     0 ACCEPT     all  --  docker0 any     anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  any    docker0  anywhere             anywhere            

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain MAILCOW (2 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain nixos-fw (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 nixos-fw-accept  all  --  lo     any     anywhere             anywhere            
  411  197K nixos-fw-accept  all  --  any    any     anywhere             anywhere             ctstate RELATED,ESTABLISHED
    0     0 nixos-fw-accept  tcp  --  any    any     anywhere             anywhere             tcp dpt:ssh
    0     0 nixos-fw-accept  tcp  --  wg0    any     anywhere             anywhere             tcp dpt:smtp
    0     0 nixos-fw-accept  tcp  --  wg0    any     anywhere             anywhere             tcp dpt:http
    0     0 nixos-fw-accept  tcp  --  wg0    any     anywhere             anywhere             tcp dpt:https
    0     0 nixos-fw-accept  tcp  --  wg0    any     anywhere             anywhere             tcp dpt:urd
    0     0 nixos-fw-accept  tcp  --  wg0    any     anywhere             anywhere             tcp dpt:imaps
    0     0 nixos-fw-accept  udp  --  any    any     anywhere             anywhere             udp dpts:60000:61000
    0     0 DROP       ipv6-icmp --  any    any     anywhere             anywhere             ipv6-icmp redirect
    0     0 DROP       ipv6-icmp --  any    any     anywhere             anywhere             ipv6-icmptype 139
  157 10056 nixos-fw-accept  ipv6-icmp --  any    any     anywhere             anywhere            
    0     0 nixos-fw-accept  udp  --  any    any     anywhere             fe80::/64            udp dpt:dhcpv6-client
  151 17058 nixos-fw-log-refuse  all  --  any    any     anywhere             anywhere            

Chain nixos-fw-accept (11 references)
 pkts bytes target     prot opt in     out     source               destination         
  568  207K ACCEPT     all  --  any    any     anywhere             anywhere            

Chain nixos-fw-log-refuse (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    1    64 LOG        tcp  --  any    any     anywhere             anywhere             tcp flags:FIN,SYN,RST,ACK/SYN LOG level info prefix "refused connection: "
  150 16994 nixos-fw-refuse  all  --  any    any     anywhere             anywhere             PKTTYPE != unicast
    1    64 nixos-fw-refuse  all  --  any    any     anywhere             anywhere            

Chain nixos-fw-refuse (2 references)
 pkts bytes target     prot opt in     out     source               destination         
  151 17058 DROP       all  --  any    any     anywhere             anywhere            
root@mailcow:~/ > systemctl restart mailcow
root@mailcow:~/ > ip6tables -L -v          
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  927  393K MAILCOW    all  --  any    any     anywhere             anywhere             /* mailcow */
 2790 2037K nixos-fw   all  --  any    any     anywhere             anywhere            

Chain FORWARD (policy DROP 219 packets, 19306 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  308 88030 MAILCOW    all  --  any    any     anywhere             anywhere             /* mailcow */
  527  107K DOCKER-USER  all  --  any    any     anywhere             anywhere            
  527  107K DOCKER-FORWARD  all  --  any    any     anywhere             anywhere            
    0     0 REJECT     all  --  wg0    ens4    anywhere             anywhere             reject-with icmp6-port-unreachable
    0     0 REJECT     all  --  wg0    lo      anywhere             anywhere             reject-with icmp6-port-unreachable
    0     0 REJECT     all  --  ens4   any     anywhere             anywhere             reject-with icmp6-port-unreachable

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  anywhere             fd4d:6169:6c63:6f77::10  tcp dpt:urd
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  anywhere             fd4d:6169:6c63:6f77::10  tcp dpt:smtp
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  anywhere             fd4d:6169:6c63:6f77::e  tcp dpt:https
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  anywhere             fd4d:6169:6c63:6f77::e  tcp dpt:http
    0     0 ACCEPT     tcp  --  !br-mailcow br-mailcow  anywhere             fd4d:6169:6c63:6f77::9  tcp dpt:imaps
    0     0 DROP       all  --  !br-mailcow br-mailcow  anywhere             anywhere            

Chain DOCKER-BRIDGE (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DOCKER     all  --  any    br-mailcow  anywhere             anywhere            

Chain DOCKER-CT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  151 73414 ACCEPT     all  --  any    br-mailcow  anywhere             anywhere             ctstate RELATED,ESTABLISHED

Chain DOCKER-FORWARD (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  527  107K DOCKER-CT  all  --  any    any     anywhere             anywhere            
  376 33922 DOCKER-ISOLATION-STAGE-1  all  --  any    any     anywhere             anywhere            
  376 33922 DOCKER-BRIDGE  all  --  any    any     anywhere             anywhere            
  157 14616 ACCEPT     all  --  br-mailcow any     anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
  157 14616 DOCKER-ISOLATION-STAGE-2  all  --  br-mailcow !br-mailcow  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  any    br-mailcow  anywhere             anywhere            

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain MAILCOW (2 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain nixos-fw (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   19  1596 nixos-fw-accept  all  --  lo     any     anywhere             anywhere            
 2025 1968K nixos-fw-accept  all  --  any    any     anywhere             anywhere             ctstate RELATED,ESTABLISHED
    0     0 nixos-fw-accept  tcp  --  any    any     anywhere             anywhere             tcp dpt:ssh
    0     0 nixos-fw-accept  tcp  --  wg0    any     anywhere             anywhere             tcp dpt:smtp
    0     0 nixos-fw-accept  tcp  --  wg0    any     anywhere             anywhere             tcp dpt:http
    0     0 nixos-fw-accept  tcp  --  wg0    any     anywhere             anywhere             tcp dpt:https
    0     0 nixos-fw-accept  tcp  --  wg0    any     anywhere             anywhere             tcp dpt:urd
    0     0 nixos-fw-accept  tcp  --  wg0    any     anywhere             anywhere             tcp dpt:imaps
    0     0 nixos-fw-accept  udp  --  any    any     anywhere             anywhere             udp dpts:60000:61000
    0     0 DROP       ipv6-icmp --  any    any     anywhere             anywhere             ipv6-icmp redirect
    0     0 DROP       ipv6-icmp --  any    any     anywhere             anywhere             ipv6-icmptype 139
  366 23232 nixos-fw-accept  ipv6-icmp --  any    any     anywhere             anywhere            
    0     0 nixos-fw-accept  udp  --  any    any     anywhere             fe80::/64            udp dpt:dhcpv6-client
  380 44594 nixos-fw-log-refuse  all  --  any    any     anywhere             anywhere            

Chain nixos-fw-accept (11 references)
 pkts bytes target     prot opt in     out     source               destination         
 2410 1992K ACCEPT     all  --  any    any     anywhere             anywhere            

Chain nixos-fw-log-refuse (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   12   928 LOG        tcp  --  any    any     anywhere             anywhere             tcp flags:FIN,SYN,RST,ACK/SYN LOG level info prefix "refused connection: "
  368 43666 nixos-fw-refuse  all  --  any    any     anywhere             anywhere             PKTTYPE != unicast
   12   928 nixos-fw-refuse  all  --  any    any     anywhere             anywhere            

Chain nixos-fw-refuse (2 references)
 pkts bytes target     prot opt in     out     source               destination         
  380 44594 DROP       all  --  any    any     anywhere             anywhere

DocFraggle

I guess the nixos chains are your problem. As per mailcow documentation you should disable all other firewalls

esackbauer

And NixOS is not a supported platform…

maybl8

esackbauer And NixOS is not a supported platform…

On this subject is there any talk of adding other platforms in the future that will be supported?
Sorry I don’t mean to hijack this post.

esackbauer

maybl8 I think the efforts to do so are too high. Especially that NixOS is a quite unique distro which might not work properly for mailcow.

maybl8

esackbauer I have been running Mailcow on Arch for over a year now with no issues. But I know it is not supported. I use the lts kernel . So was just wondering. I know Arch will never be supported because no one recommends it for a server but it can work .

artemislena

T.: So, I tried doing networking.firewall.enable = false, that didn’t seema change the issue (as before’t didn’t work before restarting the service n did work after), so from what I can tell, it’s not the nixos-firewall tables. Ik NixOS’s not officially supported but’t does decrease admin workload in general from my experience n I thought asking bout’t in the community forum might be just fine. Tbc, the issue could or could not be NixOS-specific (altho tbf generally issues we’ve had w mailcow in the past did not seem NixOS-specific), n I’m also not sure whether it’s relateda the firewall rules at all, we just noticed some strange behaviour there.

artemislena

T.: I just tried running Docker w ip6tables: false. That also seemsa fix the issue, but I feel like it’s prolly less secure that way so I’d rather not set that’f possible (unless netfilter-mailcow already handles everything that’d catch? I don’t see any rules in the MAILCOW chain, tho, maybe I gotta look at a different table?). So, I’d suspect a bug somewhere in Docker itself, Compose, or netfilter-mailcow.

Ftr, Docker version is 28.5.1, Compose version is 2.39.4. Mailcow is 2025-12.

artemislena

T.: Ah, setting ip6tables to false in the Docker config breaks outbound IPv6 :/ So Ig we gotta live w the restart for now. ’f we don’t find a solution here I s’pose we’ll just leave that as a hack until Docker 29 releases; maybe the nftables backend’ll work better.

mlcwuser

I know this doesn’t directly help with the specific problem you’re having right now. But what I really don’t understand is why one wouldn’t simply use a supported distribution like Debian or Ubuntu for this particular use case.

Mailcow is essentially an appliance. You don’t need to manually configure individual services, nor do you need packages from the distro’s repositories. So the main advantages of NixOS (configuration management) or a rolling-release model like Arch in @maybl8’s case, don’t really apply here at all, since the only purpose of the distro is to run the Mailcow Docker stack propperly.

Unless, of course, you want to install additional services or programs on the same machine/VM, which leads us to point 2 …
Don’t install any additional services or programs on the Mailcow server! 😉

A mail server should run as isolated and stable as possible. Anything else increases the chances of conflicts, update issues, or security problems.

artemislena

mlcwuser

T.: It’s run within a VM that’s configured via microvm-nix/microvm.nix, which only supports NixOS for the guests, that’s y (we only got one physical machine n don’t want more). N also there’s still some shared system configuration we wanna apply like https://codeberg.org/artemislena/nixos-server-configs/src/branch/main/shared/hardening.nix; using NixOS everywhere just makes stuff easier.

artemislena

T.: Think we can rule out Netfilter as the culprit, at least. In fact we think just specifying a custom network via Compose was enough for reproducing the behaviour; the same thing happened w this config (but not in a compose.yml that didn’t specify a custom network):

services:
  meow:
    image: toolbelt/netcat
    entrypoint: sh -c "while true; do echo MEOW | nc -6lvp 80; done;"
    ports:
      - 80:80
    networks:
      mailcow-network:
        aliases:
          - nginx

networks:
  mailcow-network:
    driver: bridge
    driver_opts:
      com.docker.network.bridge.name: br-mailcow
    enable_ipv6: ${ENABLE_IPV6:-true}
    ipam:
      driver: default
      config:
        - subnet: ${IPV4_NETWORK:-172.22.1}.0/24
        - subnet: ${IPV6_NETWORK:-fd4d:6169:6c63:6f77::/64}