When running update.sh I get the following output:

Detecting if your IP is listed on Spamhaus Bad ASN List...
Check completed! Your IP is clean
Checking internet connection... OK
Detecting which build your mailcow runs on...
You are receiving stable updates (master).
To change that run the update.sh Script one time with the --nightly parameter to switch to nightly builds.
Checking for newer update script...
Updated 0 paths from a1586f47
Are you sure you want to update mailcow: dockerized? All containers will be stopped. [y/N] y
Great! Native IPv6 NAT is active.
Validating docker-compose stack configuration...
Checking for conflicting bridges...
Saving diff to update_diffs/diff_before_update_2024-11-13-10-15-25...
Prefetching images...
1.23: Pulling from mailcow/unbound
Digest: sha256:161135ce3a3165430455e6191a5f88d9d9faab9f0c6b49689e0cd84f4fe98882
Status: Image is up to date for mailcow/unbound:1.23
docker.io/mailcow/unbound:1.23
10.5: Pulling from library/mariadb
Digest: sha256:575327d46f436430330e363aa151d34e730fa5e94d85e5effa87cc960714d1b7
Status: Image is up to date for mariadb:10.5
docker.io/library/mariadb:10.5
Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Error pulling redis:7-alpine, retrying...
Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Error pulling redis:7-alpine, retrying...
Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Error pulling redis:7-alpine, retrying...
Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Error pulling redis:7-alpine, retrying...

Too many failed retries, exiting

This happens even if attempting to update for the first time in over 12 hours - the Docker page states that the limit is 100 pulls per 6 hours.

  • esackbauer replied to this.

  • OK this is now resolved.

    I’m not sure what has changed recently but here’s what I discovered:

    Docker’s rate limiting when using IPv6 is per subnet, not per address.

    I ran the following commands from here:

    TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest 2>&1

    The response was:

    HTTP/1.1 200 OK
content-length: 527
content-type: application/vnd.docker.distribution.manifest.v2+json
docker-content-digest: sha256:c2d41d2ba6d8b7b4a3ffec621578eb4d9a0909df29dfa2f6fd8a2e5fd0836aed
docker-distribution-api-version: registry/2.0
etag: "sha256:c2d41d2ba6d8b7b4a3ffec621578eb4d9a0909df29dfa2f6fd8a2e5fd0836aed"
date: Wed, 13 Nov 2024 13:00:20 GMT
strict-transport-security: max-age=31536000
ratelimit-limit: 100;w=21600
ratelimit-remaining: 0;w=21600
docker-ratelimit-source: 2a03:b0c0:1:e0::

    See the docker-ratelimit-source - something must have changed recently and Docker now lumps together entire IPv6 subnets when rate limiting. I’ve never had reason to check rate limits before so can’t confirm what this was previously.

    The solution was to run sudo docker login - I had previously run docker login without sudo, so the Docker commands in the update script running as root were not authenticated. I then can use my own account rate limit rather than the shared IP-linked limit.

    Maybe you have other services constantly pulling images?

      Have something to say?

      Join the community by quickly registering to participate in this discussion. We'd like to see you joining our great moo-community!

      DocFraggle

      There’s nothing else on the server running except Mailcow.

      I tried authenticating with my Docker account to increase the pull limit but I got the same error immediately after. This is the only thing I use Docker for so I’m not sure why I’m hitting the limit.

        colincameron the Docker page states that the limit is 100 pulls per 6 hours.

        That is per IP address! So in case you are using a proxy and someone else is also using that proxy, it could be quickly depleted.
        Also if you are on a Carrier grate NAT (CGNAT, typically with IPv6 connections from a home/mobile network), may be the same.

          OK this is now resolved.

          I’m not sure what has changed recently but here’s what I discovered:

          Docker’s rate limiting when using IPv6 is per subnet, not per address.

          I ran the following commands from Docker Icon here

          Docker Icon Docker
          Checking Your Current Docker Pull Rate Limits and Status
          Learn from Docker experts to simplify and advance your app development and management with Docker. Stay up to date on Docker events and new version
          The Docker logo on a blue wave patterned background.Docker
          :

          TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest 2>&1

          The response was:

          HTTP/1.1 200 OK
content-length: 527
content-type: application/vnd.docker.distribution.manifest.v2+json
docker-content-digest: sha256:c2d41d2ba6d8b7b4a3ffec621578eb4d9a0909df29dfa2f6fd8a2e5fd0836aed
docker-distribution-api-version: registry/2.0
etag: "sha256:c2d41d2ba6d8b7b4a3ffec621578eb4d9a0909df29dfa2f6fd8a2e5fd0836aed"
date: Wed, 13 Nov 2024 13:00:20 GMT
strict-transport-security: max-age=31536000
ratelimit-limit: 100;w=21600
ratelimit-remaining: 0;w=21600
docker-ratelimit-source: 2a03:b0c0:1:e0::

          See the docker-ratelimit-source - something must have changed recently and Docker now lumps together entire IPv6 subnets when rate limiting. I’ve never had reason to check rate limits before so can’t confirm what this was previously.

          The solution was to run sudo docker login - I had previously run docker login without sudo, so the Docker commands in the update script running as root were not authenticated. I then can use my own account rate limit rather than the shared IP-linked limit.

          Seems like a week ago they changed something in their subscription plans:
          Docker Icon Announcing Upgraded Docker Plans: Simpler, More Value, Better Development and Productivity 

          Docker Icon Docker
          Announcing Upgraded Docker Plans: Simpler, More Value, Better Development and Productivity 
          Docker announces significant upgrades to its subscription plans, delivering more value, flexibility, and tools for customers of all sizes.
          dockerDocker

          Further changes are expected in December.

          I am also way way under their throttling threshold but I guess I’m an IP pool that depletes the free quota :-(

          No one is typing