Autoconfig Fails

OSSVirt

I’m using Thunderbird 115.9.0 and have seen Autoconfig work only for mailboxes on the same domain as the mailcow server domain. For any other domain setup on the server the Autoconfig does not work and errors out.

I can see that there are no SANs added to the LE TLS certificate, it’s just the mail.example.com domain in the cert. I’ve read the docs and I believe there should be SAN entries in the cert for:

autodiscover.maildomain1.com
autoconfig.maildomain1.com
autodiscover.maildomain2.com
autoconfig.maildomain2.com

These are not present in the LE TLS cert.

I’ve read https://docs.mailcow.email/post_installation/firststeps-ssl/ and tried restarting the acme-mailcow container with the data/assets/ssl/force_renew flag file present. The logs show that acme-tiny seems to only be requesting a cert for the main mail.example.com and not the additional domain SANs.

I’m out of ideas now - can anyone suggest where to look please? Thanks in advance for help.

DocFraggle

If you plan to use a server name that is not MAILCOW_HOSTNAME to access the mailcow UI (for example by adding mail.* to ADDITIONAL_SAN make sure to populate that name in mailcow.conf via ADDITIONAL_SERVER_NAMES. Names must be separated by commas and must not contain spaces. If you skip this step, mailcow may respond with an incorrect site.

OSSVirt

Ok, thanks. Does that also address Autoconfiguration?

DocFraggle

I don’t think so, the original hostname is always set via auto discover, so mail.example.com as server for example2.com

OSSVirt

Ok yes, so accordingly, if I load:

https://autoconfig.maildomain1.com/.well-known/autoconfig/mail/config-v1.1.xml

Then I see a browser security warning for the wrong TLS cert, because that domain does not exist as a SAN in the LE cert. If I accept the security risk and continue it successfully loads the XML file.

So I think this means that the only thing stopping Thunderbird doing the Autoconfiguration is the absence of the correct TLS certificate on the HTTPS connection.

Is that right?

DocFraggle

autoconfig.* and autodiscover.* is automatically generated for all configured domains in the mailcow UI. The ACME container will do that automatically and it should work out of the box.

You can check the current SANs of your ceritificate by running this command in /opt/mailcow-dockerized:

openssl x509 -text -noout -in data/assets/ssl/cert.pem | grep DNS | sed -e 's/,/\n/g' | awk -F: '{print $2}' | sort

OSSVirt

Ah great, thank you.

When I run that command it shows only the main domain of the mailserver:

mail.domain.com

No SANs are configured in the certificate. So the ACME container is failing to do that automatically. What could cause this to happen?

esackbauer

Probably not properly configured
ADDITIONAL_SAN and ADDITIONAL_SERVER_NAMES im mailcow.conf
assumed that you also have created the DNS entries for ALL those hostnames. And waited until they are replicated across all DNS servers.
You could also share your letsencrypt logfiles.

IdrisK

I had the same issue. It is mandatory to create a SAN in mailcow.conf, autoconfig.domain.com.

Also I discovered, the autoconfig.* is not working in SAN. It must be a FQDN, may be a Bug.

OSSVirt

No - this doesn’t work for me. I’ve tried the following in ADDITIONAL_SAN= and ADDITIONAL_SERVER_NAMES= in mailcow.conf:

Left completely blank, as default
Wildcard entries added manually in ADDITIONAL_SAN=
e.g. autoconfig.*,autodiscover.*
Wildcard entries added manually in ADDITIONAL_SAN= and ADDITIONAL_SERVER_NAMES=
e.g. autoconfig.*,autodiscover.*
All domains entered in full in ADDITIONAL_SAN=
e.g. autoconfig.maildomain1.com,autodiscover.maildomain1.com
All domains entered in full in ADDITIONAL_SAN= and ADDITIONAL_SERVER_NAMES=
e.g. autoconfig.maildomain1.com,autodiscover.maildomain1.com

None of these make any difference. Neither does either running with the Force flag added or not.

I have 5 domains configured and working for mail, as well as my main domain - mail.example.com so that’s 6 domains in total. For each mail domain I have CNAME records as follows:

autoconfig 300 IN CNAME mail.example.com.
autodiscover 300 IN CNAME mail.example.com.

When tested with dig every domain resolves to the A and AAAA record correctly. I think these 2 DNS records are all that is required for an ACME HTTP-01 challenge to succeed with each SAN domain.

Every single time I restart acme-mailcow the acme-tiny system runs, finds that there are no new certs to create, and ends:

Certificates were successfully validated, no changes or renewals required, sleeping for another day.

Or, if a forced renewal:

Certificates were successfully renewed where required, sleeping for another day.

Different users report different issues - IdrisK found that it was necessary to write out the whole FQDN. maddler found that entering the wildcards worked. Both of these are contrary to the documentation that states that these 2 wildcards are automatically entered. So there is clearly an issue.

Does someone know what is really going on at all? Maybe there is some config stuck in the database, or an issue with the local Unbound DNS server passing the domain table? Anything else anyone can suggest to troubleshoot please?

OSSVirt

Ok, thanks both @esackbauer and @IdrisK for your replies. Right so it seems quite tricky this issue.

Here is an acme log result from today:

Mon Apr 22 11:46:24 BST 2024 - Waiting for Docker API...
Mon Apr 22 11:46:24 BST 2024 - Docker API OK
Mon Apr 22 11:46:24 BST 2024 - Waiting for Postfix...
Mon Apr 22 11:46:24 BST 2024 - Postfix OK
Mon Apr 22 11:46:24 BST 2024 - Waiting for Dovecot...
Mon Apr 22 11:46:24 BST 2024 - Dovecot OK
Mon Apr 22 11:46:24 BST 2024 - Waiting for database...
Mon Apr 22 11:46:24 BST 2024 - Database OK
Mon Apr 22 11:46:24 BST 2024 - Waiting for Nginx...
Mon Apr 22 11:46:24 BST 2024 - Nginx OK
Mon Apr 22 11:46:24 BST 2024 - Waiting for resolver...
Mon Apr 22 11:46:24 BST 2024 - Resolver OK
Mon Apr 22 11:46:24 BST 2024 - Waiting for domain table...
OK
Mon Apr 22 11:46:24 BST 2024 - Initializing, please wait...
Mon Apr 22 11:46:24 BST 2024 - Using existing domain rsa key /var/lib/acme/acme/key.pem
Mon Apr 22 11:46:24 BST 2024 - Using existing Lets Encrypt account key /var/lib/acme/acme/account.pem
Mon Apr 22 11:46:24 BST 2024 - Detecting IP addresses...
Mon Apr 22 11:46:24 BST 2024 - OK: 1.2.3.4, 2001::0000
Mon Apr 22 11:46:25 BST 2024 - Found AAAA record for mail.domain.com.: 2001::0000 - skipping A record check
Mon Apr 22 11:46:25 BST 2024 - Confirmed AAAA record with IP 2001::0000
Mon Apr 22 11:46:25 BST 2024 - Certificate /var/lib/acme/mail.domain.com./cert.pem doesn't exist yet or forced renewal - start obtaining
Mon Apr 22 11:46:25 BST 2024 - Creating backups in /var/lib/acme/backups/mail.domain.com./2024-04-22_11_46_25 ...
Mon Apr 22 11:46:25 BST 2024 - Checking resolver...
Mon Apr 22 11:46:25 BST 2024 - Resolver OK
Mon Apr 22 11:46:25 BST 2024 - Using command acme-tiny   --account-key /var/lib/acme/acme/account.pem --disable-check --csr /var/lib/acme/mail.domain.com./acme.csr --acme-dir /var/www/acme/
Parsing account key...
Parsing CSR...
Found domains: mail.domain.com.
Getting directory...
Directory found!
Registering account...
Already registered! Account ID: https://acme-v02.api.letsencrypt.org/acme/acct/1359439836
Creating new order...
Order created!
Already verified: mail.domain.com., skipping...
Signing certificate...
Certificate signed!
Mon Apr 22 11:46:32 BST 2024 - Deploying certificate /var/lib/acme/mail.domain.com./cert.pem...
Mon Apr 22 11:46:32 BST 2024 - Verified hashes.
Mon Apr 22 11:46:32 BST 2024 - Certificate successfully obtained
Mon Apr 22 11:46:32 BST 2024 - Reloading or restarting services... (1)
Reloading Nginx...
Restarting 84b5ae145e96adbfaac52d6c2af89617416d2232189534ce9743b92ddae5a4ed...
command completed successfully
Restarting 62e8476df215c3d2b605d03d9d67a6839cd88c4741151d1c76270f7ea8bee5fa...
command completed successfully
Mon Apr 22 11:46:38 BST 2024 - Waiting for containers to settle...
Mon Apr 22 11:46:48 BST 2024 - Certificates were successfully renewed where required, sleeping for another day.

Every set of log entries looks identical to this, with different date/time stamps. This particular one is from after I manually added:

/opt/mailcow-dockerized/data/assets/ssl/force_renew

And ran:

docker compose restart acme-mailcow

But the log set looks the same whether I force renew or not.

Also I have added the following into /opt/mailcow-dockerized/mailcow.conf:

ADDITIONAL_SAN=serverhostname.domain.com,autoconfig.maildomain1.org.uk,autodiscover.maildomain1.org.uk

ADDITIONAL_SERVER_NAMES=serverhostname.domain.com,autoconfig.maildomain1.org.uk,autodiscover.maildomain1.org.uk

This may be overkill, but the acme system is not responding to any of it. Similarly I previously tried:

ADDITIONAL_SAN=autoconfig.*,autodiscover.*

And that also did nothing different. Please ask me more questions or for more log details etc.

I’m guessing this is an acme-tiny bug?

DocFraggle

Did you somehow edit files you should not edit manually? I.e. your docker-compose.yml file or other config files?

OSSVirt

No, I’m very careful with editing config files. The only time I accessed the docker-compose.yml file was to check the solr settings with the previous issue. I think it was yourself who actually told me to never edit the docker-compose.yml directly so I haven’t touched it since then. I definitely haven’t edited anything in docker-compose.yml.

One thing occurs to me though - I have one domain in Mailcow which is deactivated and not in use. The mx, autoconfig and autodiscovery DNS records for this domain are no longer pointing at my Mailcow server. Would this one domain not having DNS records stop ACME creating records for all the other domains? I really hope not, but I’m trying to think of everything now.

esackbauer

OSSVirt Would this one domain not having DNS records stop ACME creating records for all the other domains?

I would bet so. Also you logs are not from inside the ACME container, and not from a force renewal.
That would give more clarity.

OSSVirt

Doh! Right well I’ll try deleting that domain then 😖

esackbauer Also you logs are not from inside the ACME container, and not from a force renewal.
That would give more clarity.

Ok, do you mean to attach a shell to the acme container and view logs from within it?

esackbauer

OSSVirt do you mean to attach a shell to the acme container and view logs from within it?

“That is the way”

The Mandalorian

OSSVirt

Ok, so I am doing all this on a production server, because in this case ACME can only use HTTP challenge so it has to be the live server, which makes me nervous.

So is it as simple as:

docker container attach mailcowdockerized-acme-mailcow-1

And that shouldn’t have any affect on the running system?

esackbauer

correct. If you have Portainer you could also read the logs from there.
Just don’t enter reboot or sudo rm -rf / 😉

OSSVirt

esackbauer correct
👍️

esackbauer If you have Portainer you could also read the logs from there
Oh interesting, I’ll look at that later

esackbauer reboot or sudo rm -rf /
😱 yeah these are banned 😆

Ok I’ll try this stuff later on and report back. Thanks everyone for all the help, apologies in advance if it’s a rookie error of leaving unconfigured domain in situ. I was really hoping the config would deal with it because it is deactivated. We’ll see.

OSSVirt

Ok, so the following command gives me a bash shell inside the acme-mailcow container:

docker container exec -it mailcowdockerized-acme-mailcow-1 /bin/bash

But I cannot find where the logs live inside this container. /var/log only contains a folder called redis which is empty. I can’t find any hints in /etc/logrotate.d or anywhere else in /etc and in fact the normal LetsEncrypt folders are not present in /etc either. I guess ACME-Tiny runs in a Python env does it?

Where can I find the logs inside of the acme-mailcow container?

accolon

To my knowledge, the ACME container mainly logs to stdout which you can see (as you already did) via:
docker logs mailcowdockerized-acme-mailcow-1.

The container also logs to the central redis db which is run as a separate container. You can connect there via command line from inside any other container like this:
redis-cli -h redis -p 6379

You can show the logged info by executing:
lrange ACME_LOG 0 10000

This should show the whole redis log for ACME, based on the fact that it stores 10000 entries according to llen ACME_LOG.

I’m not sure if this is necessary, though – the same logs are shown in the mailcow UI (System/Information/Logs/ACME).

Looking at the scripts in mailcow/mailcow-dockerizedtree/36b5cccd186090d726de62b6b00d1e842e67aacd/data/Dockerfiles/acme I don’t see any additional locations for ACME logging.