Hello community,
I’m running a mailcow dockerized server on a Contabo VPS server since December.
I now received Let’s Encrypt notifications via mail saying my certificates will run out in 6 days and I tried to renew them today. Also I thought this is an automatic process?
I’ve used the following commands from the official documentation to force the cert renewal.
cd /opt/mailcow-dockerized
touch data/assets/ssl/force_renew
docker compose restart acme-mailcow
docker compose logs --tail=200 -f acme-mailcow
The output was the following - acme seems to skip the renewal even though I did force it.
I also confirmed that the force_renew file gets created before restarting acme-mailcow. Also I confirmed it got deleted afterwards.
acme-mailcow-1 | Sun Feb 23 11:13:56 CET 2025 - Waiting for Docker API...
acme-mailcow-1 | Sun Feb 23 11:13:56 CET 2025 - Docker API OK
acme-mailcow-1 | Sun Feb 23 11:13:56 CET 2025 - Waiting for Postfix...
acme-mailcow-1 | Sun Feb 23 11:13:56 CET 2025 - Postfix OK
acme-mailcow-1 | Sun Feb 23 11:13:57 CET 2025 - Waiting for Dovecot...
acme-mailcow-1 | Sun Feb 23 11:13:57 CET 2025 - Dovecot OK
acme-mailcow-1 | Sun Feb 23 11:13:57 CET 2025 - Waiting for database...
acme-mailcow-1 | Sun Feb 23 11:14:01 CET 2025 - Database OK
acme-mailcow-1 | Sun Feb 23 11:14:01 CET 2025 - Waiting for Nginx...
acme-mailcow-1 | Sun Feb 23 11:14:02 CET 2025 - Nginx OK
acme-mailcow-1 | Sun Feb 23 11:14:02 CET 2025 - Waiting for resolver...
acme-mailcow-1 | Sun Feb 23 11:14:03 CET 2025 - Resolver OK
acme-mailcow-1 | Sun Feb 23 11:14:03 CET 2025 - Waiting for domain table...
acme-mailcow-1 | OK
acme-mailcow-1 | Sun Feb 23 11:14:04 CET 2025 - Initializing, please wait...
acme-mailcow-1 | Could not find certificate from <stdin>
acme-mailcow-1 | Could not find certificate from <stdin>
acme-mailcow-1 | Could not find certificate from <stdin>
acme-mailcow-1 | Could not find certificate from <stdin>
acme-mailcow-1 | Sun Feb 23 11:14:04 CET 2025 - Using existing domain rsa key /var/lib/acme/acme/key.pem
acme-mailcow-1 | Sun Feb 23 11:14:04 CET 2025 - Using existing Lets Encrypt account key /var/lib/acme/acme/account.pem
acme-mailcow-1 | Sun Feb 23 11:14:05 CET 2025 - Found AAAA record for mta-sts.mydomain.com: 2a02:c202:XXXX:XXXX::1 - skipping A record check
acme-mailcow-1 | (skipping check, returning 0)
acme-mailcow-1 | Sun Feb 23 11:14:05 CET 2025 - Confirmed AAAA record with IP 2a02:c202:XXXX
acme-mailcow-1 | Sun Feb 23 11:14:05 CET 2025 - Found AAAA record for autodiscover.mydomain.com: 2a02:c202:XXXX:XXXX::1 - skipping A record check
acme-mailcow-1 | (skipping check, returning 0)
acme-mailcow-1 | Sun Feb 23 11:14:06 CET 2025 - Confirmed AAAA record with IP 2a02:c202:XXXX:XXXX
acme-mailcow-1 | Sun Feb 23 11:14:06 CET 2025 - Found AAAA record for autoconfig.mydomain.com: 2a02:c202:XXXX:XXXX:1 - skipping A record check
acme-mailcow-1 | (skipping check, returning 0)
acme-mailcow-1 | Sun Feb 23 11:14:06 CET 2025 - Confirmed AAAA record with IP 2a02:c202:XXXX:XXXX
acme-mailcow-1 | Sun Feb 23 11:14:07 CET 2025 - Found AAAA record for mail.mydomain.com 2a02:c202:XXXX:XXXX::1 - skipping A record check
acme-mailcow-1 | (skipping check, returning 0)
acme-mailcow-1 | Sun Feb 23 11:14:07 CET 2025 - Confirmed AAAA record with IP 2a02:c202:XXXX:XXXX
acme-mailcow-1 | Sun Feb 23 11:14:07 CET 2025 - Certificate /var/lib/acme/mail.mydomain.com/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow-1 | Sun Feb 23 11:14:07 CET 2025 - Certificates were successfully validated, no changes or renewals required, sleeping for another day.
After this I figured out maybe I need to update mailcow since I didn’t do it after the initial installation.
Below is just part of the output. However what’s import is the “Warning: You seem to have modified the /etc/docker/daemon.json configuration…” part which made we think this could have something to do with my issue.
/opt/mailcow-dockerized# ./update.sh
Detecting if your IP is listed on Spamhaus Bad ASN List...
Check completed! Your IP is clean
Adding new option "REDISPASS" to mailcow.conf
Checking internet connection... OK
Detecting which build your mailcow runs on...
You are receiving stable updates (master).
To change that run the update.sh Script one time with the --nightly parameter to switch to nightly builds.
Checking for newer update script...
Updated 0 paths from 5c1c579e
Are you sure you want to update mailcow: dockerized? All containers will be stopped. [y/N] y
Warning: You seem to have modified the /etc/docker/daemon.json configuration by yourself and not fully/correctly activated the native IPv6 NAT implementation.
You will need to merge your existing configuration manually or fix/delete the existing daemon.json configuration before trying the update process again.
Please merge the following content and restart the Docker daemon:
{"ipv6":true,"fixed-cidr-v6":"fd00:dead:beef:c0::/80","experimental":true,"ip6tables":true}
Validating docker-compose stack configuration...
Checking for conflicting bridges...
Saving diff to update_diffs/diff_before_update_2025-02-23-14-12-46...
Prefetching images...
1.23: Pulling from mailcow/unbound
So afterwards I edited the /etc/docker/daemon.json.
Before it was
{"ipv6":true,"fixed-cidr-v6":"2001:XXX:1::/64"}
And I changed it back to
{"ipv6":true,"fixed-cidr-v6":"fd00:dead:beef:c0::/80","experimental":true,"ip6tables":true}
Afterwards I’ve restarted the docker service and checked forced another cert renewal but nothing had changed from the earlier output.
Then I ran another update.sh and now the script asked if I wanted to enable Native IPv6 Implementation which I confirmed with yes.
root@XXXXX:/opt/mailcow-dockerized# ./update.sh
Detecting if your IP is listed on Spamhaus Bad ASN List...
Check completed! Your IP is clean
Checking internet connection... OK
Detecting which build your mailcow runs on...
You are receiving stable updates (master).
To change that run the update.sh Script one time with the --nightly parameter to switch to nightly builds.
Checking for newer update script...
Updated 0 paths from 5c1c579e
Are you sure you want to update mailcow: dockerized? All containers will be stopped. [y/N] y
Native IPv6 implementation available.
This will enable experimental features in the Docker daemon and configure Docker to do the IPv6 NATing instead of ipv6nat-mailcow.
!!! This step is recommended !!!
mailcow will try to roll back the changes if starting Docker fails after modifying the daemon.json configuration file.
Should we try to enable the native IPv6 implementation in Docker now (recommended)? [y/N] y
Great! Native IPv6 NAT is active.
Validating docker-compose stack configuration...
Checking for conflicting bridges...
Saving diff to update_diffs/diff_before_update_2025-02-23-15-05-09...
Prefetching images...
1.23: Pulling from mailcow/unbound
Afterwards I’ve tried to do another forced cert renewal and now receive the following output.
acme-mailcow-1 | Sun Feb 23 15:11:19 CET 2025 - Waiting for Docker API...
acme-mailcow-1 | Sun Feb 23 15:11:19 CET 2025 - Docker API OK
acme-mailcow-1 | Sun Feb 23 15:11:19 CET 2025 - Waiting for Postfix...
acme-mailcow-1 | Sun Feb 23 15:11:19 CET 2025 - Postfix OK
acme-mailcow-1 | Sun Feb 23 15:11:19 CET 2025 - Waiting for Dovecot...
acme-mailcow-1 | Sun Feb 23 15:11:19 CET 2025 - Dovecot OK
acme-mailcow-1 | Sun Feb 23 15:11:19 CET 2025 - Waiting for database...
acme-mailcow-1 | Sun Feb 23 15:11:19 CET 2025 - Database OK
acme-mailcow-1 | Sun Feb 23 15:11:19 CET 2025 - Waiting for Nginx...
acme-mailcow-1 | Sun Feb 23 15:11:19 CET 2025 - Nginx OK
acme-mailcow-1 | Sun Feb 23 15:11:19 CET 2025 - Waiting for resolver...
acme-mailcow-1 | Sun Feb 23 15:11:19 CET 2025 - Resolver OK
acme-mailcow-1 | Sun Feb 23 15:11:19 CET 2025 - Waiting for domain table...
acme-mailcow-1 | OK
acme-mailcow-1 | Sun Feb 23 15:11:19 CET 2025 - Initializing, please wait...
acme-mailcow-1 | Sun Feb 23 15:11:20 CET 2025 - Using existing domain rsa key /var/lib/acme/acme/key.pem
acme-mailcow-1 | Sun Feb 23 15:11:20 CET 2025 - Using existing Lets Encrypt account key /var/lib/acme/acme/account.pem
acme-mailcow-1 | Sun Feb 23 15:11:20 CET 2025 - Found AAAA record for mta-sts.mydomain.com: 2a02:c202:XXXX:XXXX::1 - skipping A record check
acme-mailcow-1 | (skipping check, returning 0)
acme-mailcow-1 | Sun Feb 23 15:11:20 CET 2025 - Confirmed AAAA record with IP 2a02:c202:XXXX:XXXX
acme-mailcow-1 | Sun Feb 23 15:11:21 CET 2025 - Found AAAA record for autodiscover.mydomain.com: 2a02:c202:XXXX:XXXX::1 - skipping A record check
acme-mailcow-1 | (skipping check, returning 0)
acme-mailcow-1 | Sun Feb 23 15:11:21 CET 2025 - Confirmed AAAA record with IP 2a02:c202:XXXX:XXXX
acme-mailcow-1 | Sun Feb 23 15:11:21 CET 2025 - Found AAAA record for autoconfig.mydomain.com: 2a02:c202:XXXX:XXXX::1 - skipping A record check
acme-mailcow-1 | (skipping check, returning 0)
acme-mailcow-1 | Sun Feb 23 15:11:21 CET 2025 - Confirmed AAAA record with IP 2a02:c202:XXXX:XXXX
acme-mailcow-1 | Sun Feb 23 15:11:21 CET 2025 - Found AAAA record for mail.mydomain.com: 2a02:c202:XXXX:XXXX::1 - skipping A record check
acme-mailcow-1 | (skipping check, returning 0)
acme-mailcow-1 | Sun Feb 23 15:11:21 CET 2025 - Confirmed AAAA record with IP 2a02:c202:XXXX:XXXX
acme-mailcow-1 | Sun Feb 23 15:11:21 CET 2025 - Certificate /var/lib/acme/mail.mydomain.com/cert.pem doesn't exist yet or forced renewal - start obtaining
acme-mailcow-1 | Sun Feb 23 15:11:21 CET 2025 - Creating backups in /var/lib/acme/backups/mail.mydomain.com/2025-02-23_15_11_21 ...
acme-mailcow-1 | Sun Feb 23 15:11:21 CET 2025 - Checking resolver...
acme-mailcow-1 | Sun Feb 23 15:11:21 CET 2025 - Resolver OK
acme-mailcow-1 | Sun Feb 23 15:11:21 CET 2025 - Using command acme-tiny --account-key /var/lib/acme/acme/account.pem --disable-check --csr /var/lib/acme/mail.mydomain.com/acme.csr --acme-dir /var/www/acme/
acme-mailcow-1 | Parsing account key...
acme-mailcow-1 | Parsing CSR...
acme-mailcow-1 | Found domains: autoconfig.mydomain.com, mta-sts.mydomain.com, mail.mydomain.com, autodiscover.mydomain.com
acme-mailcow-1 | Getting directory...
acme-mailcow-1 | Directory found!
acme-mailcow-1 | Registering account...
acme-mailcow-1 | Already registered! Account ID: https://acme-v02.api.letsencrypt.org/acme/acct/XXXX
acme-mailcow-1 | Creating new order...
acme-mailcow-1 | Traceback (most recent call last):
acme-mailcow-1 | File "/usr/bin/acme-tiny", line 8, in <module>
acme-mailcow-1 | sys.exit(main())
acme-mailcow-1 | ^^^^^^
acme-mailcow-1 | File "/usr/lib/python3.12/site-packages/acme_tiny.py", line 195, in main
acme-mailcow-1 | signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca, disable_check=args.disable_check, directory_url=args.directory_url, contact=args.contact, check_port=args.check_port)
acme-mailcow-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
acme-mailcow-1 | File "/usr/lib/python3.12/site-packages/acme_tiny.py", line 120, in get_crt
acme-mailcow-1 | order, _, order_headers = _send_signed_request(directory['newOrder'], order_payload, "Error creating new order")
acme-mailcow-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
acme-mailcow-1 | File "/usr/lib/python3.12/site-packages/acme_tiny.py", line 60, in _send_signed_request
acme-mailcow-1 | return _do_request(url, data=data.encode('utf8'), err_msg=err_msg, depth=depth)
acme-mailcow-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
acme-mailcow-1 | File "/usr/lib/python3.12/site-packages/acme_tiny.py", line 46, in _do_request
acme-mailcow-1 | raise ValueError("{0}:\nUrl: {1}\nData: {2}\nResponse Code: {3}\nResponse: {4}".format(err_msg, url, data, code, resp_data))
acme-mailcow-1 | ValueError: Error creating new order:
acme-mailcow-1 | Url: https://acme-v02.api.letsencrypt.org/acme/new-order
acme-mailcow-1 | Data: b'{"protected": "XXXXXXXX"}'
acme-mailcow-1 | Response Code: 429
acme-mailcow-1 | Response: {'type': 'urn:ietf:params:acme:error:rateLimited', 'detail': 'too many certificates (5) already issued for this exact set of domains in the last 168h0m0s, retry after 2025-02-24 21:28:22 UTC: see https://letsencrypt.org/docs/rate-limits/#new-certificates-per-exact-set-of-hostnames', 'status': 429}
acme-mailcow-1 | Sun Feb 23 15:11:24 CET 2025 - Failed to obtain certificate /var/lib/acme/mail.mydomain.com/cert.pem for domains 'mail.mydomain.com autoconfig.mydomain.com autodiscover.mydomain.com mta-sts.mydomain.com'
acme-mailcow-1 | OK
acme-mailcow-1 | Sun Feb 23 15:11:24 CET 2025 - Some errors occurred, retrying in 30 minutes...
acme-mailcow-1 | OK
At this point I need to ask for your help since I have no idea how to proceed from here. I really don’t understand why acme isn’t renewing the certs.
Thank you.