SSL Certificate Renewal Not Working - ACME Challenge File Missing

Running mailcow-dockerized on Ubuntu 24.04. SSL certificate renewal stopped working recently.
The error log suggests a firewall issue, but that’s not the case - nginx works fine and the webUI is accessible. The real problem seems to be that the ACME challenge file isn’t being created anywhere.

Error from logs:

ValueError: Challenge did not pass for mail.mydomain.io: {'identifier': {'type': 'dns', 'value': 'mail.mydomain.io'}, 'status': 'invalid', 'expires': '2024-10-29T07:54:05Z', 'challenges': [{'type': 'http-01', 'url': 'https://acme-v02.api.letsencrypt.org/acme/chall-v3/419591644157/kGIG3w', 'status': 'invalid', 'validated': '2024-10-22T07:54:07Z', 'error': {'type': 'urn:ietf:params:acme:error:connection', 'detail': 'Timeout during connect (likely firewall problem)'}]}

pastebin:
acme-mailcow-1 | Tue Oct 22 07:54:00 UTC 2024 - Found A record for mail.mydomai - Pastebin.com

Pastebin

acme-mailcow-1 | Tue Oct 22 07:54:00 UTC 2024 - Found A record for mail.mydomai - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

It says likely a firewall problem but I think thats nonsense, ngnix works fine and webui is accessible alright.

Key points:

• /var/lib/acme is empty (no domain subdirectory exists)
• Can’t find or access the ACME challenge file anywhere
• Tried curl and browser, challenge file isn’t accessible

Only custom config is this volume override: (I didnt find ACME Challange files there)

version: '2.1'
volumes:
  vmail-vol-1:
    driver_opts:
      type: none
      device: /mnt/raid_storage/mailcow_vmail
      o: bind

Any ideas where to look for the missing ACME challenge file?
How to fix this and ensure ACME works as intended?

DocFraggle

Ah thanks. All the files are here!
I have checked firewall again and I dont have anything up but I see something weird in iptables

target     prot opt source               destination
DROP       tcp  --  anywhere             anywhere             /* mailcow isolation */
``` I will try to flush and report back 

@"DocFraggle"#p19408 

No it wasnt that. :(

My server is accessible on port 80 and I can use webui in browser from external network.
I have put a text file in ```/var/www/acme/test.txt``` and I can see it in the browser @ ```http://myipaddress/.well-known/acme-challenge/test.txt```

@"DocFraggle"#p19408 

No it wasnt that. :(

My server is accessible on port 80 and I can use webui in browser from external network.
I have put a text file in ```/var/www/acme/test.txt``` and I can see it in the browser @ ```http://myipaddress/.well-known/acme-challenge/test.txt```

(Formatting in previous message got wrecked and I can't edit it for some reason) 

@"DocFraggle"#p19408 

No it wasnt that. :(

My server is accessible on port 80 and I can use webui in browser from external network.
I have put a text file in ```/var/www/acme/test.txt``` and I can see it in the browser @ ```http://myipaddress/.well-known/acme-challenge/test.txt```

(Formatting in previous message got wrecked and I cant edit it for some reason)

No it wasnt that. :(

My server is accessible on port 80 and I can use webui in browser from external network.
I have put a text file in ```/var/www/acme/test.txt``` and I can see it in the browser @ ```http://myipaddress/.well-known/acme-challenge/test.txt```

Update 2:

I have confirmed that my server is reachable on port 80, there is no firewall and no weird iptables rules. I can type in the address of challange token into the browser bar and inspect it. but acme still fails to renew for some point and complains about timeout

DocFraggle

Im doing something wrong because result of my curl is the HTML of admin panel.

On another note I can get curl for the following situations:
port 443 - outside network and inside local network
port 80 - only inside local network

Which is very bizarre! I have no firewall to block port 80, my mailcow is behind a NAT so it shouldn’t even know whats outside network traffic, and I tested my NAT and port 80 with another computer and it worked fine.

-When Im on local network I can resolve mailcow curl via (its local ip address, its domain, its publc ip address)

DocFraggle

Yeah its exactly that. I;m losing hope I can ever fix this

Just to be sure sudo ss -lptn 'sport = :80' is supposed to show two entries?

LISTEN               0                    4096                                     0.0.0.0:80                                      0.0.0.0:*                   users:(("docker-proxy",pid=3316,fd=4))

LISTEN               0                    4096                                        [::]:80                                         [::]:*                   users:(("docker-proxy",pid=3344,fd=4))

Edit:

Turns out it was actually borked SPI firewall rules.

The whole thing makes sense. The port 80 worked on other machines on the network because they had different SPI rules, connections within same network also worked because they bypassed SPI. Either via direct connection or NAT loopback.