HTTP validation failed

BornDeranged

Hello everyone
I have recently encountered that error which has been documented before, but I haven’t found a solution through the community discussions, the github issues nor the documentation.

I am on 2025-10a, on Debian running at Hetzner. No other symptoms except that the certificates get no longer renewed:

acme-mailcow-1  | Sun Nov  9 10:14:39 UTC 2025 - Initializing, please wait...
acme-mailcow-1  | Sun Nov  9 10:14:40 UTC 2025 - Using existing domain rsa key /var/lib/acme/acme/key.pem
acme-mailcow-1  | Sun Nov  9 10:14:40 UTC 2025 - Using existing Lets Encrypt account key /var/lib/acme/acme/account.pem
acme-mailcow-1  | Sun Nov  9 10:14:40 UTC 2025 - Detecting IP addresses...
acme-mailcow-1  | Sun Nov  9 10:14:40 UTC 2025 - OK: 5.75.244.224, 2a01:4f8:c17:58a7::1
acme-mailcow-1  | Sun Nov  9 10:14:40 UTC 2025 - Found AAAA record for autodiscover.usel.eu: 2a01:4f8:c17:58a7::1 - skipping A record check
acme-mailcow-1  | Sun Nov  9 10:14:41 UTC 2025 - Confirmed AAAA record with IP 2a01:04f8:0c17:58a7:0000:0000:0000:0001, but HTTP validation failed
acme-mailcow-1  | Sun Nov  9 10:14:41 UTC 2025 - Found AAAA record for autoconfig.usel.eu: 2a01:4f8:c17:58a7::1 - skipping A record check
acme-mailcow-1  | Sun Nov  9 10:14:42 UTC 2025 - Confirmed AAAA record with IP 2a01:04f8:0c17:58a7:0000:0000:0000:0001, but HTTP validation failed
acme-mailcow-1  | Sun Nov  9 10:14:42 UTC 2025 - No A or AAAA record found for hostname mta-sts.usel.eu
acme-mailcow-1  | Sun Nov  9 10:14:42 UTC 2025 - Found AAAA record for autodiscover.usel.lu: 2a01:4f8:c17:58a7::1 - skipping A record check
acme-mailcow-1  | Sun Nov  9 10:14:43 UTC 2025 - Confirmed AAAA record with IP 2a01:04f8:0c17:58a7:0000:0000:0000:0001, but HTTP validation failed
acme-mailcow-1  | Sun Nov  9 10:14:43 UTC 2025 - Found AAAA record for autoconfig.usel.lu: 2a01:4f8:c17:58a7::1 - skipping A record check
acme-mailcow-1  | Sun Nov  9 10:14:44 UTC 2025 - Confirmed AAAA record with IP 2a01:04f8:0c17:58a7:0000:0000:0000:0001, but HTTP validation failed
acme-mailcow-1  | Sun Nov  9 10:14:44 UTC 2025 - No A or AAAA record found for hostname mta-sts.usel.lu
acme-mailcow-1  | Sun Nov  9 10:14:44 UTC 2025 - Found AAAA record for autodiscover.usel.me: 2a01:4f8:c17:58a7::1 - skipping A record check
acme-mailcow-1  | Sun Nov  9 10:14:46 UTC 2025 - Confirmed AAAA record with IP 2a01:04f8:0c17:58a7:0000:0000:0000:0001, but HTTP validation failed
acme-mailcow-1  | Sun Nov  9 10:14:46 UTC 2025 - Found AAAA record for autoconfig.usel.me: 2a01:4f8:c17:58a7::1 - skipping A record check
acme-mailcow-1  | Sun Nov  9 10:14:47 UTC 2025 - Confirmed AAAA record with IP 2a01:04f8:0c17:58a7:0000:0000:0000:0001, but HTTP validation failed
acme-mailcow-1  | Sun Nov  9 10:14:47 UTC 2025 - No A or AAAA record found for hostname mta-sts.usel.me
acme-mailcow-1  | Sun Nov  9 10:14:47 UTC 2025 - Found AAAA record for autodiscover.usel.ovh: 2a01:4f8:c17:58a7::1 - skipping A record check
acme-mailcow-1  | Sun Nov  9 10:14:48 UTC 2025 - Confirmed AAAA record with IP 2a01:04f8:0c17:58a7:0000:0000:0000:0001, but HTTP validation failed
acme-mailcow-1  | Sun Nov  9 10:14:48 UTC 2025 - Found AAAA record for autoconfig.usel.ovh: 2a01:4f8:c17:58a7::1 - skipping A record check
acme-mailcow-1  | Sun Nov  9 10:14:49 UTC 2025 - Confirmed AAAA record with IP 2a01:04f8:0c17:58a7:0000:0000:0000:0001, but HTTP validation failed
acme-mailcow-1  | Sun Nov  9 10:14:49 UTC 2025 - No A or AAAA record found for hostname mta-sts.usel.ovh
acme-mailcow-1  | Sun Nov  9 10:14:49 UTC 2025 - Found AAAA record for hx2.usel.eu: 2a01:4f8:c17:58a7::1 - skipping A record check
acme-mailcow-1  | Sun Nov  9 10:14:50 UTC 2025 - Confirmed AAAA record with IP 2a01:04f8:0c17:58a7:0000:0000:0000:0001, but HTTP validation failed
acme-mailcow-1  | Sun Nov  9 10:14:50 UTC 2025 - Cannot validate any hostnames, skipping Let's Encrypt for 1 hour.
acme-mailcow-1  | Sun Nov  9 10:14:50 UTC 2025 - Use SKIP_LETS_ENCRYPT=y in mailcow.conf to skip it permanently.
acme-mailcow-1  | OK

IPv6 works, there’s no additional proxy, curl http://[2a01:4f8:c17:58a7::1] works:

root@hx2 /opt/mailcow-dockerized $  ip a show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 96:00:03:28:3e:c2 brd ff:ff:ff:ff:ff:ff
    altname enp1s0
    altname enx960003283ec2
    inet 5.75.244.224/32 brd 5.75.244.224 scope global dynamic eth0
       valid_lft 83625sec preferred_lft 83625sec
    inet6 2a01:4f8:c17:58a7::1/64 scope global
       valid_lft forever preferred_lft forever

ufw is active but appears OK, and so is the associated Hetzner firewall for port 80:

root@hx2 /opt/mailcow-dockerized $  ufw status
Status: active

To                         Action      From
--                         ------      ----
Anywhere                   ALLOW       80/tcp                     # http
Anywhere                   ALLOW       443/tcp                    # https
Anywhere (v6)              ALLOW       80/tcp (v6)                # http
Anywhere (v6)              ALLOW       443/tcp (v6)               # https

For some reason it appears to have worked on Nov 1st (filenames redacted):

root@hx2 /opt/mailcow-dockerized/data/web/.well-known/acme-challenge $  ll -ltr
total 20K
-rw-r--r-- 1 root root 14 Sep 20 03:21 aaaaa47622
-rw-r--r-- 1 root root 15 Sep 24 03:21 bbbbb629512
-rw-r--r-- 1 root root 16 Oct 18 03:21 ccccc7330577
-rw-r--r-- 1 root root 16 Oct 25 03:21 ddddd5916735
-rw-r--r-- 1 root root 14 Nov  1 03:21 eeeee33601

I removed the CAA record just to make sure, but that didn’t change anything (and it did work with the CAA record before).

I would love to avoid skipping http check but rather find the root cause.

Any clues appreciated!

esackbauer

BornDeranged No A or AAAA record found for hostname mta-sts.usel.me

“recently” seems after you enabled MTA-STS. It seems you missed the proper A and AAAA records for your STS hostnames.

esackbauer

BornDeranged That was mentioned in my first post

Mentioned was only “proxy” not reverse proxy 😉
Maybe its a docker network thing, make sure you updated docker compose plugin and Docker to the latest versions.

BornDeranged ufw is active but appears OK

“appears”… Please disable
https://docs.mailcow.email/getstarted/prerequisite-system/#firewall-ports
What do you expect ufw can solve what Mailcows docker network rules or Hetzner firewall cannot?

BornDeranged

esackbauer MTA-STS in not enabled for any domain according to the web interface, which is why I ignored that part.

esackbauer

BornDeranged
It seems then since 2025-09 version mailcow wants to add this hostname to your certificate. Or maybe you had it enabled for a site and then disabled it again?
Create a CNAME for it and point it to your mailcow hostname.

BornDeranged

esackbauer Good idea, unfortunately it still fails

acme-mailcow-1  | Sun Nov  9 11:00:05 UTC 2025 - Initializing, please wait...
acme-mailcow-1  | Sun Nov  9 11:00:05 UTC 2025 - Using existing domain rsa key /var/lib/acme/acme/key.pem
acme-mailcow-1  | Sun Nov  9 11:00:05 UTC 2025 - Using existing Lets Encrypt account key /var/lib/acme/acme/account.pem
acme-mailcow-1  | Sun Nov  9 11:00:05 UTC 2025 - Detecting IP addresses...
acme-mailcow-1  | Sun Nov  9 11:00:05 UTC 2025 - OK: 5.75.244.224, 2a01:4f8:c17:58a7::1
acme-mailcow-1  | Sun Nov  9 11:00:05 UTC 2025 - Found AAAA record for autodiscover.usel.eu: 2a01:4f8:c17:58a7::1 - skipping A record check
acme-mailcow-1  | Sun Nov  9 11:00:06 UTC 2025 - Confirmed AAAA record with IP 2a01:04f8:0c17:58a7:0000:0000:0000:0001, but HTTP validation failed
acme-mailcow-1  | Sun Nov  9 11:00:06 UTC 2025 - Found AAAA record for autoconfig.usel.eu: 2a01:4f8:c17:58a7::1 - skipping A record check
acme-mailcow-1  | Sun Nov  9 11:00:07 UTC 2025 - Confirmed AAAA record with IP 2a01:04f8:0c17:58a7:0000:0000:0000:0001, but HTTP validation failed
acme-mailcow-1  | Sun Nov  9 11:00:07 UTC 2025 - Found AAAA record for mta-sts.usel.eu: 2a01:4f8:c17:58a7::1 - skipping A record check
acme-mailcow-1  | Sun Nov  9 11:00:08 UTC 2025 - Confirmed AAAA record with IP 2a01:04f8:0c17:58a7:0000:0000:0000:0001, but HTTP validation failed
acme-mailcow-1  | Sun Nov  9 11:00:09 UTC 2025 - Found AAAA record for autodiscover.usel.lu: 2a01:4f8:c17:58a7::1 - skipping A record check
acme-mailcow-1  | Sun Nov  9 11:00:10 UTC 2025 - Confirmed AAAA record with IP 2a01:04f8:0c17:58a7:0000:0000:0000:0001, but HTTP validation failed
acme-mailcow-1  | Sun Nov  9 11:00:10 UTC 2025 - Found AAAA record for autoconfig.usel.lu: 2a01:4f8:c17:58a7::1 - skipping A record check
acme-mailcow-1  | Sun Nov  9 11:00:11 UTC 2025 - Confirmed AAAA record with IP 2a01:04f8:0c17:58a7:0000:0000:0000:0001, but HTTP validation failed
acme-mailcow-1  | Sun Nov  9 11:00:11 UTC 2025 - Found AAAA record for mta-sts.usel.lu: 2a01:4f8:c17:58a7::1 - skipping A record check
acme-mailcow-1  | Sun Nov  9 11:00:12 UTC 2025 - Confirmed AAAA record with IP 2a01:04f8:0c17:58a7:0000:0000:0000:0001, but HTTP validation failed
acme-mailcow-1  | Sun Nov  9 11:00:12 UTC 2025 - Found AAAA record for autodiscover.usel.me: 2a01:4f8:c17:58a7::1 - skipping A record check
acme-mailcow-1  | Sun Nov  9 11:00:13 UTC 2025 - Confirmed AAAA record with IP 2a01:04f8:0c17:58a7:0000:0000:0000:0001, but HTTP validation failed
acme-mailcow-1  | Sun Nov  9 11:00:13 UTC 2025 - Found AAAA record for autoconfig.usel.me: 2a01:4f8:c17:58a7::1 - skipping A record check
acme-mailcow-1  | Sun Nov  9 11:00:14 UTC 2025 - Confirmed AAAA record with IP 2a01:04f8:0c17:58a7:0000:0000:0000:0001, but HTTP validation failed
acme-mailcow-1  | Sun Nov  9 11:00:14 UTC 2025 - Found AAAA record for mta-sts.usel.me: 2a01:4f8:c17:58a7::1 - skipping A record check
acme-mailcow-1  | Sun Nov  9 11:00:15 UTC 2025 - Confirmed AAAA record with IP 2a01:04f8:0c17:58a7:0000:0000:0000:0001, but HTTP validation failed
acme-mailcow-1  | Sun Nov  9 11:00:15 UTC 2025 - Found AAAA record for autodiscover.usel.ovh: 2a01:4f8:c17:58a7::1 - skipping A record check
acme-mailcow-1  | Sun Nov  9 11:00:16 UTC 2025 - Confirmed AAAA record with IP 2a01:04f8:0c17:58a7:0000:0000:0000:0001, but HTTP validation failed
acme-mailcow-1  | Sun Nov  9 11:00:17 UTC 2025 - Found AAAA record for autoconfig.usel.ovh: 2a01:4f8:c17:58a7::1 - skipping A record check
acme-mailcow-1  | Sun Nov  9 11:00:18 UTC 2025 - Confirmed AAAA record with IP 2a01:04f8:0c17:58a7:0000:0000:0000:0001, but HTTP validation failed
acme-mailcow-1  | Sun Nov  9 11:00:18 UTC 2025 - Found AAAA record for mta-sts.usel.ovh: 2a01:4f8:c17:58a7::1 - skipping A record check
acme-mailcow-1  | Sun Nov  9 11:00:19 UTC 2025 - Confirmed AAAA record with IP 2a01:04f8:0c17:58a7:0000:0000:0000:0001, but HTTP validation failed
acme-mailcow-1  | Sun Nov  9 11:00:19 UTC 2025 - Found AAAA record for hx2.usel.eu: 2a01:4f8:c17:58a7::1 - skipping A record check
acme-mailcow-1  | Sun Nov  9 11:00:20 UTC 2025 - Confirmed AAAA record with IP 2a01:04f8:0c17:58a7:0000:0000:0000:0001, but HTTP validation failed
acme-mailcow-1  | Sun Nov  9 11:00:20 UTC 2025 - Cannot validate any hostnames, skipping Let's Encrypt for 1 hour.
acme-mailcow-1  | Sun Nov  9 11:00:20 UTC 2025 - Use SKIP_LETS_ENCRYPT=y in mailcow.conf to skip it permanently.
acme-mailcow-1  | OK

esackbauer

Oh, now Records look good. Do you have reverse proxy?
Port 80 is reachable from outside for both IPv4 and IPv6? (Maybe Hetzner Firewall does something?)

BornDeranged

esackbauer Thought so too, but it does find them, and https://dnschecker.org/#CNAME/mta-sts.usel.eu agrees.
What is the easiest way to disable IPv6 temporarily?

esackbauer

I think there is no easy way. You need to delete the AAAA record in your DNS and follow the docs to disable IPv6.
I guess its easier to check Hetzners Firewall, if it lets through the port 80 for IPv6.

BornDeranged

esackbauer Yes I did that:

esackbauer

https://community.mailcow.email/d/3998-acme-http-validation-failed
You still owe the answer if you are using a reverse proxy (or have a local nginx/apache running on your host)

BornDeranged

esackbauer That was mentioned in my first post - no reverse proxy.

BornDeranged

esackbauer Right, reverse proxy.😅

I have ufw for other ports related to the admin network interface.
The docker version I use is from docker themselves:

root@hx2 /etc/apt/sources.list.d $  cat docker.sources
Types: deb
URIs: https://download.docker.com/linux/debian/
Suites: trixie
Components: stable
Signed-By: /etc/apt/keyrings/docker.gpg

root@hx2 /opt/mailcow-dockerized $  docker --version
Docker version 28.5.2, build ecc6942

esackbauer

BornDeranged I have ufw for other ports related to the admin network interface.

That does not answer my question or solves the problem that ufw interferes with the docker networks.

BornDeranged

esackbauer
Those ufw rules haven’t changed in ages, but just to make sure I have just restarted the machine without ufw - and it worked 😱.
Going to dig into it in more detail.
Thanks for your help!