Running mailcow-dockerized on Ubuntu 24.04. SSL certificate renewal stopped working recently.
The error log suggests a firewall issue, but that’s not the case - nginx works fine and the webUI is accessible. The real problem seems to be that the ACME challenge file isn’t being created anywhere.

Error from logs:

ValueError: Challenge did not pass for mail.mydomain.io: {'identifier': {'type': 'dns', 'value': 'mail.mydomain.io'}, 'status': 'invalid', 'expires': '2024-10-29T07:54:05Z', 'challenges': [{'type': 'http-01', 'url': 'https://acme-v02.api.letsencrypt.org/acme/chall-v3/419591644157/kGIG3w', 'status': 'invalid', 'validated': '2024-10-22T07:54:07Z', 'error': {'type': 'urn:ietf:params:acme:error:connection', 'detail': 'Timeout during connect (likely firewall problem)'}]}

pastebin:
Pastebin Icon acme-mailcow-1 | Tue Oct 22 07:54:00 UTC 2024 - Found A record for mail.mydomai - Pastebin.com

Pastebin Icon Pastebin
acme-mailcow-1 | Tue Oct 22 07:54:00 UTC 2024 - Found A record for mail.mydomai - Pastebin.com
Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.
Pastebin

It says likely a firewall problem but I think thats nonsense, ngnix works fine and webui is accessible alright.

Key points:

  • /var/lib/acme is empty (no domain subdirectory exists)
  • Can’t find or access the ACME challenge file anywhere
  • Tried curl and browser, challenge file isn’t accessible

Only custom config is this volume override: (I didnt find ACME Challange files there)

version: '2.1'
volumes:
  vmail-vol-1:
    driver_opts:
      type: none
      device: /mnt/raid_storage/mailcow_vmail
      o: bind

Any ideas where to look for the missing ACME challenge file?
How to fix this and ensure ACME works as intended?

    blue /var/lib/acme

    You are looking inside the ACME container?

    /opt/mailcow-dockerized# docker compose exec acme-mailcow /bin/bash 
222cafdca3f3:/# ls /var/lib/acme/
acme            backups         cert.pem        dhparams.pem    key.pem         mail.domain.tld

    Which version are you running? stable or nightly?

    And did you check if port 80 is opened in your firewall?

    • blue replied to this.

        Have something to say?

        Join the community by quickly registering to participate in this discussion. We'd like to see you joining our great moo-community!

        DocFraggle

        Ah thanks. All the files are here!
        I have checked firewall again and I dont have anything up but I see something weird in iptables

        target     prot opt source               destination
DROP       tcp  --  anywhere             anywhere             /* mailcow isolation */
``` I will try to flush and report back 

@"DocFraggle"#p19408 

No it wasnt that. :(

My server is accessible on port 80 and I can use webui in browser from external network.
I have put a text file in ```/var/www/acme/test.txt``` and I can see it in the browser @ ```http://myipaddress/.well-known/acme-challenge/test.txt```

@"DocFraggle"#p19408 

No it wasnt that. :(

My server is accessible on port 80 and I can use webui in browser from external network.
I have put a text file in ```/var/www/acme/test.txt``` and I can see it in the browser @ ```http://myipaddress/.well-known/acme-challenge/test.txt```

(Formatting in previous message got wrecked and I can't edit it for some reason) 

@"DocFraggle"#p19408 

No it wasnt that. :(

My server is accessible on port 80 and I can use webui in browser from external network.
I have put a text file in ```/var/www/acme/test.txt``` and I can see it in the browser @ ```http://myipaddress/.well-known/acme-challenge/test.txt```

(Formatting in previous message got wrecked and I cant edit it for some reason)

No it wasnt that. :(

My server is accessible on port 80 and I can use webui in browser from external network.
I have put a text file in ```/var/www/acme/test.txt``` and I can see it in the browser @ ```http://myipaddress/.well-known/acme-challenge/test.txt```

        Update 2:

        I have confirmed that my server is reachable on port 80, there is no firewall and no weird iptables rules. I can type in the address of challange token into the browser bar and inspect it. but acme still fails to renew for some point and complains about timeout 🙁

          Okay actually I am unable to reach my ubuntu on port 80, only 443. I cannot find the reason for that, something terrible must have happened. I tried tcpdump on port 80 and I can only see outgoing traffic, no firewall rules, no iptables. Maybe my ISP stealtlhy blocked port 80?

          To be absolutely sure check the output of

          ufw status
systemctl status firewalld

          and to be 100% certain

          systemctl list-units --type=service | grep firewall

          If they are all negative, check your router, it also has a firewall (I understand you are using mailcow at home)

          • blue replied to this.

            DocFraggle

            -ufw inactive
            -no firewalld
            -nothing under systemctl list-units –type=service | grep firewall

            I tested port 80 by forwarding to different computer on my network - works and is reachable from outside world.

            It just seems that mailcow ABSOLUTELY refuses to let me connect on port 80 and I’ve no clue why.

            On another note

            sudo ss -lptn 'sport = :80'
State          Recv-Q         Send-Q                 Local Address:Port                 Peer Address:Port         Process
LISTEN         0              4096                         0.0.0.0:80                        0.0.0.0:*             users:(("docker-proxy",pid=1116544,fd=4))
LISTEN         0              4096                            [::]:80                           [::]:*             users:(("docker-proxy",pid=1116552,fd=4))

sudo ss -lptn 'sport = :443'
State          Recv-Q         Send-Q                 Local Address:Port                 Peer Address:Port         Process
LISTEN         0              4096                         0.0.0.0:443                       0.0.0.0:*             users:(("docker-proxy",pid=1116516,fd=4))
LISTEN         0              4096                            [::]:443                          [::]:*             users:(("docker-proxy",pid=1116524,fd=4))

            Which seems correct?

              Maybe just restart everything…

              docker compose down

reboot

              then

              docker compose up -d

              • blue replied to this.

                DocFraggle

                Unfortunately didnt help.
                Do you have any way to check whats blocking port 80? I have other services running on the same server so there might be some interference but I couldn’t find anything with my linux skills.

                  What exactly is the output of

                  ss -lptn 'sport = :80'

                  on your mailcow host? I couldn’t make out if you were running the command on your mailcow host or “On another note”

                  Ah sorry, so the above was the output of your mailcow server. Whats the output if you use curl on your mailcow host (replace your.mailcow.fwdn whith your mailcow FQDN)? It should look like this:

                  curl -v -H "Host: your.mailcow.fqdn" http://127.0.0.1:80
*   Trying 127.0.0.1:80...
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: your.mailcow.fqdn
> User-Agent: curl/7.88.1
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.27.2
< Date: Tue, 22 Oct 2024 12:07:31 GMT
< Content-Type: text/html
< Content-Length: 169
< Connection: keep-alive
< Location: https://your.mailcow.fqdn/
< 
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.27.2</center>
</body>
</html>
* Connection #0 to host 127.0.0.1 left intact
                  • blue replied to this.

                    DocFraggle

                    Im doing something wrong because result of my curl is the HTML of admin panel.

                    On another note I can get curl for the following situations:
                    port 443 - outside network and inside local network
                    port 80 - only inside local network

                    Which is very bizarre! I have no firewall to block port 80, my mailcow is behind a NAT so it shouldn’t even know whats outside network traffic, and I tested my NAT and port 80 with another computer and it worked fine.

                    -When Im on local network I can resolve mailcow curl via (its local ip address, its domain, its publc ip address)

                      Whcih admin panel? The mailcow admin UI? Are you using some kind of reverse proxy? Nginx proxy manager or stuff like that?

                      • blue replied to this.

                        DocFraggle

                        No proxy.

                        Yeah I think its admin UI starts off like this

                        < Server: nginx
< Date: Tue, 22 Oct 2024 12:43:50 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Vary: Accept-Encoding
< Set-Cookie: PHPSESSID=e82f7a5c83a4a2ca6fad2e7e843f0406; path=/; HttpOnly
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Strict-Transport-Security: max-age=15768000;
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< X-Robots-Tag: none
< X-Download-Options: noopen
< X-Frame-Options: SAMEORIGIN
< X-Permitted-Cross-Domain-Policies: none
< Referrer-Policy: strict-origin

                        To summarise:

                        1. I can resolve my server from FQND and public ip as long as its on any port other than 80 - for example 443
                        2. I can resolve any other server on my network, just not the one with mailcow - something is broken.
                        3. I can resolve mailcow on port 80 as long as its from local network.
                        4. No proxy
                        5. It worked fine before and broke for unknown reason.
                        6. No firewall on my router, no firewall on my mailcow server - I am able to resolve locally.
                        7. I am behind NAT so mailcow should have no idea whats ‘outside’ traffic yet for some reason local network can reach it but not outside network.

                        Puzzling indeed.

                          ah ok, it’s different because I redirect HTTP to HTTPS 🙂

                          docs.mailcow.email Icon Redirect HTTP to HTTPS - mailcow: dockerized documentation

                          docs.mailcow.email Icon docs.mailcow.email
                          Redirect HTTP to HTTPS - mailcow: dockerized documentation
                          None
                          docs.mailcow.email

                          Last idea: what did you set these to in mailcow.conf?

                          HTTP_PORT=80
HTTP_BIND=
HTTPS_PORT=443
HTTPS_BIND=
                          • blue replied to this.

                            DocFraggle

                            Yeah its exactly that. I;m losing hope I can ever fix this 🙁

                            Just to be sure sudo ss -lptn 'sport = :80' is supposed to show two entries?

                            LISTEN               0                    4096                                     0.0.0.0:80                                      0.0.0.0:*                   users:(("docker-proxy",pid=3316,fd=4))

LISTEN               0                    4096                                        [::]:80                                         [::]:*                   users:(("docker-proxy",pid=3344,fd=4))

                              Okay after enough rebooting, destroying and flushing iptables the issue was solved by restarting the router…

                              Edit:

                              Turns out it was actually borked SPI firewall rules.

                              The whole thing makes sense. The port 80 worked on other machines on the network because they had different SPI rules, connections within same network also worked because they bypassed SPI. Either via direct connection or NAT loopback.

                                blue Turns out it was actually borked SPI firewall rules.

                                So my hunch regarding a firewall issue was right in the end 🙂

                                Great that you figured this out, have fun

                                  No one is typing