Network is unreachable when running update.sh

DHO

I have had mailcow running well for over a year.

Yesterday, I decided it was time to update. I ssh into the server. sudo su, cd /opt/mailcow-dockerized, and ./update.sh. I consistently receive this error:

Detecting if your IP is listed on Spamhaus Bad ASN List...
Check failed! Maybe a DNS or Network problem?
Checking internet connection... OK
Detecting which build your mailcow runs on...
You are receiving stable updates (master).
To change that run the update.sh Script one time with the --nightly parameter to switch to nightly builds.
Checking for newer update script...
fatal: unable to access 'https://github.com/mailcow/mailcow-dockerized/': Could not resolve host: github.com
Updated 0 paths from 12754a3b
Are you sure you want to update mailcow: dockerized? All containers will be stopped. [y/N] y
Great! Native IPv6 NAT is active.
Validating docker-compose stack configuration...
Checking for conflicting bridges...
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Saving diff to update_diffs/diff_before_update_2024-02-21-12-47-43...
Prefetching images...
fatal: unable to access 'https://github.com/mailcow/mailcow-dockerized/': Could not resolve host: github.com
1.21: Pulling from mailcow/unbound
619be1103602: Pull complete 
1f4254c93fbb: Pull complete 
cd03c5664d5d: Pull complete 
ff6211bd3a5f: Pull complete 
39075a9d6a11: Pull complete 
Digest: sha256:a2ae19356feab1aa8dbc26e3dcd1c7b6fd87a07763316e5b5dc52b4b32722920
Status: Downloaded newer image for mailcow/unbound:1.21
docker.io/mailcow/unbound:1.21
10.5: Pulling from library/mariadb
Digest: sha256:f5a27c861ffbcbeddcd6855dcfde5237df16640f3f97f1736cb7046d3a9b85ba
Status: Image is up to date for mariadb:10.5
docker.io/library/mariadb:10.5
Error response from daemon: Get "https://registry-1.docker.io/v2/library/redis/manifests/sha256:1b503bb77079ba644371969e06e1a6a1670bb34c2251107c0fc3a21ef9fdaeca": dial tcp [2600:1f18:2148:bc02:445d:9ace:d20b:c303]:443: connect: network is unreachable

Error pulling redis:7-alpine, retrying...
Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp [2600:1f18:2148:bc01:571f:e759:a87a:2961]:443: connect: network is unreachable

Error pulling redis:7-alpine, retrying...
Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp [2600:1f18:2148:bc01:571f:e759:a87a:2961]:443: connect: network is unreachable

Error pulling redis:7-alpine, retrying...
Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp [2600:1f18:2148:bc02:445d:9ace:d20b:c303]:443: connect: network is unreachable

Error pulling redis:7-alpine, retrying...

Too many failed retries, exiting

The thing that jump out at me is the IPv6. My network has no IPv6 support. Typing ping6 ipv6.google.com will always result in no network found. I found this odd since I have updated many times previously…. So I waited a few hour and nothing changed, so I started troubleshooting.

First thought was to edit grub to disable IPv6. And docker would not start, even after removing the known docker network cards.
So I then edited the /etc/gai.conf to use ::ffff:0:0/96 100. And net.ipv6.conf.all.disable_ipv6 = 1. Docker seems to ignore theses completely.

So I turned to mail cow and edited the docker-compose.yml to ipv6=failse and commented out the subnet. This boots and as far as I can tell does nothing.

I added the override with “ipv6nat disabled in compose.override.yml” And get this error:

sudo docker compose up -d
[+] Running 1/1
 ✘ ipv6nat-mailcow Error                                                                                                     0.5s 
Error response from daemon: Get "https://registry-1.docker.io/v2/library/bash/manifests/sha256:8e45c8ffe44db8784197f7c849a22292b446d76895f65646717b5a2152114d6e": dial tcp [2600:1f18:2148:bc00:8d61:9b62:40aa:8bb8]:443: connect: network is unreachable

Remove the override and boot again.
To look for taint I ran an update –gc. No change.

I have tried a couple house, and maybe 100% looking in wrong place, but that’s why I’m asking.
Thanks you for any help.

Ubuntu 22.04.4 LTS
Mailcow: Version: 2023-05a
Docker Compose version v2.10.2
Docker version 25.0.3, build 4debf41

esackbauer

DHO Checking for newer update script…
fatal: unable to access ‘mailcow/mailcow-dockerized': Could not resolve host: github.com

Did that ever succeed? Downloading the latest update.sh script? If not, than no wonder you have problems…

DHO

It’s hit or miss for github.
Pings fine on IPv4.

Here’s one where github connection was okay:

./update.sh 
Detecting if your IP is listed on Spamhaus Bad ASN List...
Check completed! Your IP is clean
Checking internet connection... OK
Detecting which build your mailcow runs on...
You are receiving stable updates (master).
To change that run the update.sh Script one time with the --nightly parameter to switch to nightly builds.
Checking for newer update script...
Updated 0 paths from 12754a3b
Are you sure you want to update mailcow: dockerized? All containers will be stopped. [y/N] y
Great! Native IPv6 NAT is active.
Validating docker-compose stack configuration...
Checking for conflicting bridges...
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Saving diff to update_diffs/diff_before_update_2024-02-21-14-04-50...
Prefetching images...
1.21: Pulling from mailcow/unbound
Digest: sha256:a2ae19356feab1aa8dbc26e3dcd1c7b6fd87a07763316e5b5dc52b4b32722920
Status: Image is up to date for mailcow/unbound:1.21
docker.io/mailcow/unbound:1.21
10.5: Pulling from library/mariadb
Digest: sha256:f5a27c861ffbcbeddcd6855dcfde5237df16640f3f97f1736cb7046d3a9b85ba
Status: Image is up to date for mariadb:10.5
docker.io/library/mariadb:10.5
Error response from daemon: Get "https://registry-1.docker.io/v2/library/redis/manifests/sha256:1b503bb77079ba644371969e06e1a6a1670bb34c2251107c0fc3a21ef9fdaeca": dial tcp [2600:1f18:2148:bc00:8d61:9b62:40aa:8bb8]:443: connect: network is unreachable

Error pulling redis:7-alpine, retrying...
Error response from daemon: Get "https://registry-1.docker.io/v2/": dial tcp [2600:1f18:2148:bc00:8d61:9b62:40aa:8bb8]:443: connect: network is unreachable

DocFraggle

You can try to disable IPv6 for mailcow if you don’t use it at all

https://docs.mailcow.email/post_installation/firststeps-disable_ipv6

DHO

DocFraggle

Following that guide. On step #2 when I try to docker up, I get this error:

sudo docker compose up -d
[+] Running 1/1
 ✘ ipv6nat-mailcow Error                                                                                                     0.5s 
Error response from daemon: Get "https://registry-1.docker.io/v2/library/bash/manifests/sha256:8e45c8ffe44db8784197f7c849a22292b446d76895f65646717b5a2152114d6e": dial tcp [2600:1f18:2148:bc00:8d61:9b62:40aa:8bb8]:443: connect: network is unreachable

DocFraggle

Hmm… so you are on Docker 25.0.3 at least? And you commented out the IPv6 subnet? And set enable_ipv6 to false?

Your docker-compose.override.yml is at the right place?

DHO

DocFraggle

Doc yep.
IPv6 false + commented out subnet.
Docker version 25.0.3, build 4debf41

And I know override is working, b/c only get error when it is in place. And the update script recognizes it. So double confirmation.

networks:
  mailcow-network:
    driver: bridge
    driver_opts:
      com.docker.network.bridge.name: br-mailcow
    enable_ipv6: false
    ipam:
      driver: default
      config:
        - subnet: ${IPV4_NETWORK:-172.22.1}.0/24
#        - subnet: ${IPV6_NETWORK:-fd4d:6169:6c63:6f77::/64}

DocFraggle

OK but where is the override for the ipv6nat-mailcow?

DHO

DocFraggle

It’s here:
/opt/mailcow-dockerized/docker-compose.override.yml

version: '2.1'
services:

    ipv6nat-mailcow:
      image: bash:latest
      restart: "no"
      entrypoint: ["echo", "ipv6nat disabled in compose.override.yml"]

DocFraggle

I just had a look at my docker-compose.yml, the whole ipv6nat-mailcow part is missing. I guess it’s stripped in case you don’t need it, but I can’t find the stripping part in a script at the moment…

Please have a look at your main file, if it’s also not in there that may be the problem with the override file, so just delete it again

DHO

DocFraggle

I dont see the ipv6nat in the main config either.
But only having the enable_ipv6 set to false doesn’t seem to do anything by itself.

DocFraggle

Well, back to the start then… are you sure there is no IPv6 config left? It seems that your host OS tries to resolve FQDNs both with IPv4 and IPv6, that’s the reason why it sometimes works (if the returned IPv4 address is used) and sometimes it doesn’t (if the returned IPv6 address is used).

What’s the output of a simple

dig github.com

on your host system? My local VM which doesn’t have any IPv6 address except for the default link local address only returns IPv4 addresses

DHO

There a couple mentions of IPv6 in environments for rspam and fpm
IPV6_NETWORK=${IPV6_NETWORK:-fd4d:6169:6c63:6f77::/64}
I have them commented out, and doesn’t seem to change anything.

In terms of the host, just a stock ubuntu box with mailcow. No other mods or installs.
I tried disabling IPv6 via grub, but docker completely refused to start. I can do again later for errors.
And all local syscntrl prop edits seem to be completely ignored by docker, others have posted the same.

; <<>> DiG 9.18.18-0ubuntu0.22.04.2-Ubuntu <<>> github.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13683
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;github.com.			IN	A

;; ANSWER SECTION:
github.com.		0	IN	A	140.82.113.4

;; Query time: 8 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Thu Feb 22 12:32:53 UTC 2024
;; MSG SIZE  rcvd: 55

Interesting finding, looking at the IPV6 on host.
sudo sysctl -a | grep ipv6 | less

I found every time I set any of these to 0
net.ipv6.conf.all.autoconf = 1
net.ipv6.conf.default.autoconf = 1
net.ipv6.conf.br-mailcow.autoconf = 1
etc

sudo sysctl -w net.ipv6.conf.br-mailcow.autoconf=0

It gets reverted to 1 on reboot. Probably be a few hours before i can properly research this.

Interesting finding, looking at the IPV6 on host.
sudo sysctl -a | grep ipv6 | less

I found every time I set any of these to 0
net.ipv6.conf.all.autoconf = 1
net.ipv6.conf.default.autoconf = 1
net.ipv6.conf.br-mailcow.autoconf = 1
etc

sudo sysctl -w net.ipv6.conf.br-mailcow.autoconf=0

It gets reverted to 1 on reboot. Probably be a few hours before i can properly research this.

DocFraggle

BTW, you’re not alone:

docker/hub-feedback2349

Maybe try adding the hosts entries as a workaround while figuring this out

DHO

Host file is a creative fix. That worked.

For me (my region) the host file is:

54.236.113.205 registry-1.docker.io
54.198.86.24 auth.docker.io
104.16.102.207 production.cloudflare.docker.com

I’m not sure if related, but the host file has the follow. Not sure where it came from or if it’s stock

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Thanks!!!