Docker network error after upgrading to 2023-08

ic9000

I did have a fully work Mailcow (version 2023-05a) setup until I did an upgrade a few moments ago, which took mailcow to 2023-08.

I’m running mailcow on a dedicated VM running Debian 12 with a basic install and docker. Apart from Mailcow I am also running portainer-agent.

Everything was running fine until this afternoon when I did a apt upgrade and a mailcow update.sh

Both completed without errors, but now when I do a ‘docker compose up -d’ I get the below error

Error response from daemon: user specified IP address is supported only when connecting to networks with user configured subnets

I checked my mailcow.conf and I have got IPV4_NETWORK=172.22.1 set.

However when I check the networks the compose is creating a network called “mailcowdockerized_mailcow-network” with address range 172.18.0.0/16

I can confirm there is nothing using 172.22.1 I tried changing IPV4_NETWORK to another range as I read someone had a similar problem several years ago but that didn’t fix my issue.

Is this a mailcow issue or a docker issue? I did notice the apt upgrade took my docker up to version 24.0.5

Any ideas?

ic9000

I’ve gone back to docker version 24.0.4 (which is what I had installed prior to the apt upgrade) and still the same error.

So my question now is. How can I resolve this? or How do I revert back to the previous version of Mailcow?

esackbauer

Are you using the latest docker-compose version as well?
I can say Docker 24.0.5 is running fine here, I have pretty much the same setup as you.

ic9000

esackbauer

docker-compose is v.2.20.2

esackbauer

why did you use then “docker compose up -d” (plugin variant) instead of “docker-compose up -d” (standalone variant)?
An update doesnt require to do that.
Which one did you use to install mailcow?

ic9000

I installed docker with

apt install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

When I originally ran the generate_config.sh it told me to use docker compose (no -)

That is what I have been using

# docker compose up -d   
[+] Running 17/20   
✔ Network mailcowdockerized_mailcow-network        Created
✔ Container mailcowdockerized-solr-mailcow-1       Started
⠿ Container mailcowdockerized-redis-mailcow-1      Starting
✔ Container mailcowdockerized-dockerapi-mailcow-1  Started
✔ Container mailcowdockerized-memcached-mailcow-1  Started
✔ Container mailcowdockerized-watchdog-mailcow-1   Started
✔ Container mailcowdockerized-olefy-mailcow-1      Started
⠿ Container mailcowdockerized-sogo-mailcow-1       Starting
⠿ Container mailcowdockerized-unbound-mailcow-1    Starting
✔ Container mailcowdockerized-php-fpm-mailcow-1    Created
✔ Container mailcowdockerized-clamd-mailcow-1      Created
✔ Container mailcowdockerized-mysql-mailcow-1      Created
✔ Container mailcowdockerized-nginx-mailcow-1      Created
✔ Container mailcowdockerized-postfix-mailcow-1    Created
✔ Container mailcowdockerized-dovecot-mailcow-1    Created
✔ Container mailcowdockerized-acme-mailcow-1       Created
✔ Container mailcowdockerized-rspamd-mailcow-1     Created
✔ Container mailcowdockerized-ofelia-mailcow-1     Created
✔ Container mailcowdockerized-netfilter-mailcow-1  Created
✔ Container mailcowdockerized-ipv6nat-mailcow-1    Created
Error response from daemon: user specified IP address is supported only when connecting to networks with user configured subnets

I tried docker compose up -d –force-recreate as well but same thing.

esackbauer

seems ok so far, but stay away from docker-compose (with dash, the standalone variant).
Maybe you mixed it up or installed both?
And pro tip: Always do a VM snapshot before upgrading…

ic9000

I usually do take a snapshot, but schoolboy error today.

I do have a backup from using the backup_and_restore.sh however I can’t restore it as I can’t get docker compose to come up with out the network error.

I completely removed docker and installed again, same error.

I am now in the process of bring up a brand new VM and will hopeful restore on that,

D4niel

That’s just an error for the mailcow docker network. There’s no need to restore anything. Did you try to stop all containers and remove the network by hand? It should be automatically created by docker compose again.

ic9000

D4niel

Yes.

I did a docker compose down and that removed the network as well

# docker compose down
# docker network ls
NETWORK ID     NAME      DRIVER    SCOPE
0b21777f6f89   bridge    bridge    local
6d972e79cf43   host      host      local
a9e359fdf655   none      null      local

I then do a docker compose up

# docker compose up -d   
[+] Running 17/20   
✔ Network mailcowdockerized_mailcow-network        Created
✔ Container mailcowdockerized-solr-mailcow-1       Started
⠿ Container mailcowdockerized-redis-mailcow-1      Starting
✔ Container mailcowdockerized-dockerapi-mailcow-1  Started
✔ Container mailcowdockerized-memcached-mailcow-1  Started
✔ Container mailcowdockerized-watchdog-mailcow-1   Started
✔ Container mailcowdockerized-olefy-mailcow-1      Started
⠿ Container mailcowdockerized-sogo-mailcow-1       Starting
⠿ Container mailcowdockerized-unbound-mailcow-1    Starting
✔ Container mailcowdockerized-php-fpm-mailcow-1    Created
✔ Container mailcowdockerized-clamd-mailcow-1      Created
✔ Container mailcowdockerized-mysql-mailcow-1      Created
✔ Container mailcowdockerized-nginx-mailcow-1      Created
✔ Container mailcowdockerized-postfix-mailcow-1    Created
✔ Container mailcowdockerized-dovecot-mailcow-1    Created
✔ Container mailcowdockerized-acme-mailcow-1       Created
✔ Container mailcowdockerized-rspamd-mailcow-1     Created
✔ Container mailcowdockerized-ofelia-mailcow-1     Created
✔ Container mailcowdockerized-netfilter-mailcow-1  Created
✔ Container mailcowdockerized-ipv6nat-mailcow-1    Created
Error response from daemon: user specified IP address is supported only when connecting to networks with user configured subnets

I then checked the network again and got the below

docker network ls
NETWORK ID     NAME                                DRIVER    SCOPE
0b21777f6f89   bridge                              bridge    local
6d972e79cf43   host                                host      local
1b980d5b7570   mailcowdockerized_mailcow-network   bridge    local
a9e359fdf655   none                                null      local

When I inspect the mailcow network

# docker network inspect 1b98
[
    {
        "Name": "mailcowdockerized_mailcow-network",
        "Id": "1b980d5b7570b27da2d5367760342fe52e520d90392c2ea2aef265d80e7553ce",
        "Created": "2023-08-03T19:16:17.816494536+01:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.21.0.0/16",
                    "Gateway": "172.21.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "21dd083ac4603fa6105b884cb4566c18bc27042f8b0fcf95b16e70fee74875d3": {
                "Name": "mailcowdockerized-watchdog-mailcow-1",
                "EndpointID": "1c6c3a6c85d64563f130a61ba29dcc4e0bbd87ee57a2ab718f07ddf7f3d87bdb",
                "MacAddress": "02:42:ac:15:00:05",
                "IPv4Address": "172.21.0.5/16",
                "IPv6Address": ""
            },
            "83f4b07f85517fd53d03840e44d9c322b268259bf3d7fc756329898062f5d379": {
                "Name": "mailcowdockerized-olefy-mailcow-1",
                "EndpointID": "e2fa7027e42f0ae602f8f8f34d028d3ade81669acda7f4e15350f59f19fdd475",
                "MacAddress": "02:42:ac:15:00:02",
                "IPv4Address": "172.21.0.2/16",
                "IPv6Address": ""
            },
            "a6c81716149c3115a36535b0807c7b7a4b35121da8f85a009f39b2715c674853": {
                "Name": "mailcowdockerized-memcached-mailcow-1",
                "EndpointID": "4676ea10fcd71563b09a2d3254a10acddc2338db1490f36824a8be595ebb8539",
                "MacAddress": "02:42:ac:15:00:04",
                "IPv4Address": "172.21.0.4/16",
                "IPv6Address": ""
            },
            "da5944dbe470047f6ed9fae2d5bfc15b78b08e9b8c1053f4d077d254b0da52be": {
                "Name": "mailcowdockerized-dockerapi-mailcow-1",
                "EndpointID": "7fe4cb6ec16846716a52cbe49d36277a9c5b01bd4a29eb77c0b944902551e383",
                "MacAddress": "02:42:ac:15:00:06",
                "IPv4Address": "172.21.0.6/16",
                "IPv6Address": ""
            },
            "dcc1d36520d4232c2af9b8584226927e765b251ce636c4d00286e4c17a8df7f1": {
                "Name": "mailcowdockerized-solr-mailcow-1",
                "EndpointID": "9b4e5c416614253aa03b811b5056681b74a17f2ae2cee058b4219e3364031d00",
                "MacAddress": "02:42:ac:15:00:03",
                "IPv4Address": "172.21.0.3/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "mailcow-network",
            "com.docker.compose.project": "mailcowdockerized",
            "com.docker.compose.version": "2.20.2"
        }
    }
]

In my mailcow.conf I have IPV4_NETWORK=172.22.1

The 172.22.1 network isn’t being used anywhere else. docker0 is using 172.17.0.1/16 and nic ens18 is assigned a 10.x.x.x address.

I also have IPv6 disabled

#cat docker-compose.override.yml
version: '2.1'
services:

  # Disable IPv6
  #
  # Ref - https://docs.mailcow.email/post_installation/firststeps-disable_ipv6
  ipv6nat-mailcow:
    image: bash:latest
    restart: "no"
    entrypoint: ["echo", "ipv6nat disabled in compose.override.yml"]

# Disable IPv6
networks:
  mailcow-network:
    enable_ipv6: false

I thought when I did a docker network ls I would see the network called “br-mailcow” and not “mailcowdockerized_mailcow-network”

What am I missing?

D4niel

ic9000 I thought when I did a docker network ls I would see the network called “br-mailcow” and not “mailcowdockerized_mailcow-network”

br-mailcow is the name for the physical interface on the host, not the name of the docker network.

You should probably compare your docker-compose file to the one on the repo. Either that is borked or something in your mailcow.conf

ic9000

both my docker-compose.yml and the repo version are identical.

can’t compare mailcow.conf as thats generated. So I run the generate script and overwrote the existing mailcow.conf file

Made no difference, still the same error.

ic9000

Ok, I’ve found the problem, just not sure of the resolution.

This is also repeatable from a fresh installation.

Setup a VM with base Debian 11
Install Docker using the plugin (native) docker compose
Clone the repo and run generate config
Edit the mailcow.conf, in this case the only thing I changed was SKIP_LETS_ENCRYPT=y

Fire up mailcow with “docker compose up -d” and all working fine, yay!

Now. I want to disable IPv6

So I did a “docker compose down”

Updated the /opt/mailcow-dockerized/data/conf/unbound/unbound.conf with “do-ip6: no”

Added a docker-compose.override.yml

version: '2.1'
services:

  # Disable IPv6
  ipv6nat-mailcow:
    image: bash:latest
    restart: "no"
    entrypoint: ["echo", "ipv6nat disabled in compose.override.yml"]

# Disable IPv6
networks:
  mailcow-network:
    enable_ipv6: false

Now when I do a docker compose up I get the error.

Error response from daemon: user specified IP address is supported only when connecting to networks with user configured subnets

Can you spot any errors in my docker-compose.override.yml file??

esackbauer

ic9000
You did not read the docs properly:
https://docs.mailcow.email/post_installation/firststeps-disable_ipv6/?h=disable
Its a bit misleading:
“Instead of editing docker-compose.yml directly, it is preferable to create an override file for it and implement your changes to the service there. Unfortunately, this right now only seems to work for services, not for network settings.”
So you HAVE to edit the docker-compose.yml!

hasanmerkit

Same problem! 1000+ mails not sending today! Please help!

ic9000

hasanmerkit

My work around “for now” was to leave IPv6 enabled. Didn’t think it mattered to much in my case as the VM is on a IPv4 network only.

However still searching for a solution, because I do want to disable IPv6.

ic9000

esackbauer

Strange, because until yesterday it was working with the override file as is.

I have other docker based projects that override the network settings without any issues.

Solved it…. For now at least

I’ve not changed the docker-compose.yml as suggested. Instead I’ve added a bit more into my docker-compose.override.yml file.

I’ve gone from this

version: '2.1'
services:

  # Disable IPv6
  ipv6nat-mailcow:
    image: bash:latest
    restart: "no"
    entrypoint: ["echo", "ipv6nat disabled in compose.override.yml"]

# Disable IPv6
networks:
  mailcow-network:
    enable_ipv6: false

to this

version: '2.1'
services:

  # Disable IPv6
  ipv6nat-mailcow:
    image: bash:latest
    restart: "no"
    entrypoint: ["echo", "ipv6nat disabled in compose.override.yml"]

# Disable IPv6
networks:
  mailcow-network:
    driver: bridge
    driver_opts:
      com.docker.network.bridge.name: br-mailcow
    enable_ipv6: false
    ipam:
      driver: default
      config:
        - subnet: ${IPV4_NETWORK:-172.22.1}.0/24
        - subnet: ${IPV6_NETWORK:-fd4d:6169:6c63:6f77::/64}

So basically taking the network section from the docker-compose.yml into the override and changing the enable_ipv6 to false.

Having it in the override file keeps the mailcow unchanged, just need to remember to maintain the override file at upgrade time.