SSL Certificate Renewal Not Working - ACME Challenge File Missing

blue

Running mailcow-dockerized on Ubuntu 24.04. SSL certificate renewal stopped working recently.
The error log suggests a firewall issue, but that’s not the case - nginx works fine and the webUI is accessible. The real problem seems to be that the ACME challenge file isn’t being created anywhere.

Error from logs:

ValueError: Challenge did not pass for mail.mydomain.io: {'identifier': {'type': 'dns', 'value': 'mail.mydomain.io'}, 'status': 'invalid', 'expires': '2024-10-29T07:54:05Z', 'challenges': [{'type': 'http-01', 'url': 'https://acme-v02.api.letsencrypt.org/acme/chall-v3/419591644157/kGIG3w', 'status': 'invalid', 'validated': '2024-10-22T07:54:07Z', 'error': {'type': 'urn:ietf:params:acme:error:connection', 'detail': 'Timeout during connect (likely firewall problem)'}]}

pastebin:
https://pastebin.com/fzcPSLRk

It says likely a firewall problem but I think thats nonsense, ngnix works fine and webui is accessible alright.

Key points:

/var/lib/acme is empty (no domain subdirectory exists)
Can’t find or access the ACME challenge file anywhere
Tried curl and browser, challenge file isn’t accessible

Only custom config is this volume override: (I didnt find ACME Challange files there)

version: '2.1'
volumes:
  vmail-vol-1:
    driver_opts:
      type: none
      device: /mnt/raid_storage/mailcow_vmail
      o: bind

Any ideas where to look for the missing ACME challenge file?
How to fix this and ensure ACME works as intended?

DocFraggle

blue /var/lib/acme

You are looking inside the ACME container?

/opt/mailcow-dockerized# docker compose exec acme-mailcow /bin/bash 
222cafdca3f3:/# ls /var/lib/acme/
acme            backups         cert.pem        dhparams.pem    key.pem         mail.domain.tld

Which version are you running? stable or nightly?

And did you check if port 80 is opened in your firewall?

blue

DocFraggle

Ah thanks. All the files are here!
I have checked firewall again and I dont have anything up but I see something weird in iptables

target     prot opt source               destination
DROP       tcp  --  anywhere             anywhere             /* mailcow isolation */
``` I will try to flush and report back 

@"DocFraggle"#p19408 

No it wasnt that. :(

My server is accessible on port 80 and I can use webui in browser from external network.
I have put a text file in ```/var/www/acme/test.txt``` and I can see it in the browser @ ```http://myipaddress/.well-known/acme-challenge/test.txt```

@"DocFraggle"#p19408 

No it wasnt that. :(

My server is accessible on port 80 and I can use webui in browser from external network.
I have put a text file in ```/var/www/acme/test.txt``` and I can see it in the browser @ ```http://myipaddress/.well-known/acme-challenge/test.txt```

(Formatting in previous message got wrecked and I can't edit it for some reason) 

@"DocFraggle"#p19408 

No it wasnt that. :(

My server is accessible on port 80 and I can use webui in browser from external network.
I have put a text file in ```/var/www/acme/test.txt``` and I can see it in the browser @ ```http://myipaddress/.well-known/acme-challenge/test.txt```

(Formatting in previous message got wrecked and I cant edit it for some reason)

No it wasnt that. :(

My server is accessible on port 80 and I can use webui in browser from external network.
I have put a text file in ```/var/www/acme/test.txt``` and I can see it in the browser @ ```http://myipaddress/.well-known/acme-challenge/test.txt```

Update 2:

I have confirmed that my server is reachable on port 80, there is no firewall and no weird iptables rules. I can type in the address of challange token into the browser bar and inspect it. but acme still fails to renew for some point and complains about timeout 🙁

DocFraggle

Hmm, I would have bet it’s ufw… does it work with the hostname as well?

so not IP address, but http://your.domain.tld/.well-known/acme-challenge/test.txt

blue

Okay actually I am unable to reach my ubuntu on port 80, only 443. I cannot find the reason for that, something terrible must have happened. I tried tcpdump on port 80 and I can only see outgoing traffic, no firewall rules, no iptables. Maybe my ISP stealtlhy blocked port 80?

DocFraggle

To be absolutely sure check the output of

ufw status
systemctl status firewalld

and to be 100% certain

systemctl list-units --type=service | grep firewall

If they are all negative, check your router, it also has a firewall (I understand you are using mailcow at home)

blue

DocFraggle

-ufw inactive
-no firewalld
-nothing under systemctl list-units –type=service | grep firewall

I tested port 80 by forwarding to different computer on my network - works and is reachable from outside world.

It just seems that mailcow ABSOLUTELY refuses to let me connect on port 80 and I’ve no clue why.

On another note

sudo ss -lptn 'sport = :80'
State          Recv-Q         Send-Q                 Local Address:Port                 Peer Address:Port         Process
LISTEN         0              4096                         0.0.0.0:80                        0.0.0.0:*             users:(("docker-proxy",pid=1116544,fd=4))
LISTEN         0              4096                            [::]:80                           [::]:*             users:(("docker-proxy",pid=1116552,fd=4))

sudo ss -lptn 'sport = :443'
State          Recv-Q         Send-Q                 Local Address:Port                 Peer Address:Port         Process
LISTEN         0              4096                         0.0.0.0:443                       0.0.0.0:*             users:(("docker-proxy",pid=1116516,fd=4))
LISTEN         0              4096                            [::]:443                          [::]:*             users:(("docker-proxy",pid=1116524,fd=4))

Which seems correct?

DocFraggle

Maybe just restart everything…

docker compose down

reboot

then

docker compose up -d

blue

DocFraggle

Unfortunately didnt help.
Do you have any way to check whats blocking port 80? I have other services running on the same server so there might be some interference but I couldn’t find anything with my linux skills.

DocFraggle

What exactly is the output of

ss -lptn 'sport = :80'

on your mailcow host? I couldn’t make out if you were running the command on your mailcow host or “On another note”

Ah sorry, so the above was the output of your mailcow server. Whats the output if you use curl on your mailcow host (replace your.mailcow.fwdn whith your mailcow FQDN)? It should look like this:

curl -v -H "Host: your.mailcow.fqdn" http://127.0.0.1:80
*   Trying 127.0.0.1:80...
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: your.mailcow.fqdn
> User-Agent: curl/7.88.1
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.27.2
< Date: Tue, 22 Oct 2024 12:07:31 GMT
< Content-Type: text/html
< Content-Length: 169
< Connection: keep-alive
< Location: https://your.mailcow.fqdn/
< 
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.27.2</center>
</body>
</html>
* Connection #0 to host 127.0.0.1 left intact

blue

DocFraggle

Im doing something wrong because result of my curl is the HTML of admin panel.

On another note I can get curl for the following situations:
port 443 - outside network and inside local network
port 80 - only inside local network

Which is very bizarre! I have no firewall to block port 80, my mailcow is behind a NAT so it shouldn’t even know whats outside network traffic, and I tested my NAT and port 80 with another computer and it worked fine.

-When Im on local network I can resolve mailcow curl via (its local ip address, its domain, its publc ip address)

DocFraggle

Whcih admin panel? The mailcow admin UI? Are you using some kind of reverse proxy? Nginx proxy manager or stuff like that?

blue

DocFraggle

No proxy.

Yeah I think its admin UI starts off like this

< Server: nginx
< Date: Tue, 22 Oct 2024 12:43:50 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Vary: Accept-Encoding
< Set-Cookie: PHPSESSID=e82f7a5c83a4a2ca6fad2e7e843f0406; path=/; HttpOnly
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Strict-Transport-Security: max-age=15768000;
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< X-Robots-Tag: none
< X-Download-Options: noopen
< X-Frame-Options: SAMEORIGIN
< X-Permitted-Cross-Domain-Policies: none
< Referrer-Policy: strict-origin

To summarise:

I can resolve my server from FQND and public ip as long as its on any port other than 80 - for example 443
I can resolve any other server on my network, just not the one with mailcow - something is broken.
I can resolve mailcow on port 80 as long as its from local network.
No proxy
It worked fine before and broke for unknown reason.
No firewall on my router, no firewall on my mailcow server - I am able to resolve locally.
I am behind NAT so mailcow should have no idea whats ‘outside’ traffic yet for some reason local network can reach it but not outside network.

Puzzling indeed.

DocFraggle

ah ok, it’s different because I redirect HTTP to HTTPS 🙂

https://docs.mailcow.email/manual-guides/u_e-80_to_443/

Last idea: what did you set these to in mailcow.conf?

HTTP_PORT=80
HTTP_BIND=
HTTPS_PORT=443
HTTPS_BIND=

blue

DocFraggle

Yeah its exactly that. I;m losing hope I can ever fix this 🙁

Just to be sure sudo ss -lptn 'sport = :80' is supposed to show two entries?

LISTEN               0                    4096                                     0.0.0.0:80                                      0.0.0.0:*                   users:(("docker-proxy",pid=3316,fd=4))

LISTEN               0                    4096                                        [::]:80                                         [::]:*                   users:(("docker-proxy",pid=3344,fd=4))

blue

Okay after enough rebooting, destroying and flushing iptables the issue was solved by restarting the router…

blue

Edit:

Turns out it was actually borked SPI firewall rules.

The whole thing makes sense. The port 80 worked on other machines on the network because they had different SPI rules, connections within same network also worked because they bypassed SPI. Either via direct connection or NAT loopback.

DocFraggle

blue Turns out it was actually borked SPI firewall rules.

So my hunch regarding a firewall issue was right in the end 🙂

Great that you figured this out, have fun