DocFraggle

Ah thanks. All the files are here!
I have checked firewall again and I dont have anything up but I see something weird in iptables

target     prot opt source               destination
DROP       tcp  --  anywhere             anywhere             /* mailcow isolation */
``` I will try to flush and report back 

@"DocFraggle"#p19408 

No it wasnt that. :(

My server is accessible on port 80 and I can use webui in browser from external network.
I have put a text file in ```/var/www/acme/test.txt``` and I can see it in the browser @ ```http://myipaddress/.well-known/acme-challenge/test.txt```

@"DocFraggle"#p19408 

No it wasnt that. :(

My server is accessible on port 80 and I can use webui in browser from external network.
I have put a text file in ```/var/www/acme/test.txt``` and I can see it in the browser @ ```http://myipaddress/.well-known/acme-challenge/test.txt```

(Formatting in previous message got wrecked and I can't edit it for some reason) 

@"DocFraggle"#p19408 

No it wasnt that. :(

My server is accessible on port 80 and I can use webui in browser from external network.
I have put a text file in ```/var/www/acme/test.txt``` and I can see it in the browser @ ```http://myipaddress/.well-known/acme-challenge/test.txt```

(Formatting in previous message got wrecked and I cant edit it for some reason)

No it wasnt that. :(

My server is accessible on port 80 and I can use webui in browser from external network.
I have put a text file in ```/var/www/acme/test.txt``` and I can see it in the browser @ ```http://myipaddress/.well-known/acme-challenge/test.txt```

Update 2:

I have confirmed that my server is reachable on port 80, there is no firewall and no weird iptables rules. I can type in the address of challange token into the browser bar and inspect it. but acme still fails to renew for some point and complains about timeout 🙁

    Okay actually I am unable to reach my ubuntu on port 80, only 443. I cannot find the reason for that, something terrible must have happened. I tried tcpdump on port 80 and I can only see outgoing traffic, no firewall rules, no iptables. Maybe my ISP stealtlhy blocked port 80?

    To be absolutely sure check the output of

    ufw status
systemctl status firewalld

    and to be 100% certain

    systemctl list-units --type=service | grep firewall

    If they are all negative, check your router, it also has a firewall (I understand you are using mailcow at home)

    • blue replied to this.

      DocFraggle

      -ufw inactive
      -no firewalld
      -nothing under systemctl list-units –type=service | grep firewall

      I tested port 80 by forwarding to different computer on my network - works and is reachable from outside world.

      It just seems that mailcow ABSOLUTELY refuses to let me connect on port 80 and I’ve no clue why.

      On another note

      sudo ss -lptn 'sport = :80'
State          Recv-Q         Send-Q                 Local Address:Port                 Peer Address:Port         Process
LISTEN         0              4096                         0.0.0.0:80                        0.0.0.0:*             users:(("docker-proxy",pid=1116544,fd=4))
LISTEN         0              4096                            [::]:80                           [::]:*             users:(("docker-proxy",pid=1116552,fd=4))

sudo ss -lptn 'sport = :443'
State          Recv-Q         Send-Q                 Local Address:Port                 Peer Address:Port         Process
LISTEN         0              4096                         0.0.0.0:443                       0.0.0.0:*             users:(("docker-proxy",pid=1116516,fd=4))
LISTEN         0              4096                            [::]:443                          [::]:*             users:(("docker-proxy",pid=1116524,fd=4))

      Which seems correct?

        Maybe just restart everything…

        docker compose down

reboot

        then

        docker compose up -d

        • blue replied to this.

          DocFraggle

          Unfortunately didnt help.
          Do you have any way to check whats blocking port 80? I have other services running on the same server so there might be some interference but I couldn’t find anything with my linux skills.

            What exactly is the output of

            ss -lptn 'sport = :80'

            on your mailcow host? I couldn’t make out if you were running the command on your mailcow host or “On another note”

            Ah sorry, so the above was the output of your mailcow server. Whats the output if you use curl on your mailcow host (replace your.mailcow.fwdn whith your mailcow FQDN)? It should look like this:

            curl -v -H "Host: your.mailcow.fqdn" http://127.0.0.1:80
*   Trying 127.0.0.1:80...
* Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: your.mailcow.fqdn
> User-Agent: curl/7.88.1
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.27.2
< Date: Tue, 22 Oct 2024 12:07:31 GMT
< Content-Type: text/html
< Content-Length: 169
< Connection: keep-alive
< Location: https://your.mailcow.fqdn/
< 
<html>
<head><title>301 Moved Permanently</title></head>
<body>
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.27.2</center>
</body>
</html>
* Connection #0 to host 127.0.0.1 left intact
            • blue replied to this.

              DocFraggle

              Im doing something wrong because result of my curl is the HTML of admin panel.

              On another note I can get curl for the following situations:
              port 443 - outside network and inside local network
              port 80 - only inside local network

              Which is very bizarre! I have no firewall to block port 80, my mailcow is behind a NAT so it shouldn’t even know whats outside network traffic, and I tested my NAT and port 80 with another computer and it worked fine.

              -When Im on local network I can resolve mailcow curl via (its local ip address, its domain, its publc ip address)

                Whcih admin panel? The mailcow admin UI? Are you using some kind of reverse proxy? Nginx proxy manager or stuff like that?

                • blue replied to this.

                  DocFraggle

                  No proxy.

                  Yeah I think its admin UI starts off like this

                  < Server: nginx
< Date: Tue, 22 Oct 2024 12:43:50 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Vary: Accept-Encoding
< Set-Cookie: PHPSESSID=e82f7a5c83a4a2ca6fad2e7e843f0406; path=/; HttpOnly
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Strict-Transport-Security: max-age=15768000;
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< X-Robots-Tag: none
< X-Download-Options: noopen
< X-Frame-Options: SAMEORIGIN
< X-Permitted-Cross-Domain-Policies: none
< Referrer-Policy: strict-origin

                  To summarise:

                  1. I can resolve my server from FQND and public ip as long as its on any port other than 80 - for example 443
                  2. I can resolve any other server on my network, just not the one with mailcow - something is broken.
                  3. I can resolve mailcow on port 80 as long as its from local network.
                  4. No proxy
                  5. It worked fine before and broke for unknown reason.
                  6. No firewall on my router, no firewall on my mailcow server - I am able to resolve locally.
                  7. I am behind NAT so mailcow should have no idea whats ‘outside’ traffic yet for some reason local network can reach it but not outside network.

                  Puzzling indeed.

                    ah ok, it’s different because I redirect HTTP to HTTPS 🙂

                    docs.mailcow.email Icon Redirect HTTP to HTTPS - mailcow: dockerized documentation

                    docs.mailcow.email Icon docs.mailcow.email
                    Redirect HTTP to HTTPS - mailcow: dockerized documentation
                    None
                    docs.mailcow.email

                    Last idea: what did you set these to in mailcow.conf?

                    HTTP_PORT=80
HTTP_BIND=
HTTPS_PORT=443
HTTPS_BIND=
                    • blue replied to this.

                      DocFraggle

                      Yeah its exactly that. I;m losing hope I can ever fix this 🙁

                      Just to be sure sudo ss -lptn 'sport = :80' is supposed to show two entries?

                      LISTEN               0                    4096                                     0.0.0.0:80                                      0.0.0.0:*                   users:(("docker-proxy",pid=3316,fd=4))

LISTEN               0                    4096                                        [::]:80                                         [::]:*                   users:(("docker-proxy",pid=3344,fd=4))

                        Okay after enough rebooting, destroying and flushing iptables the issue was solved by restarting the router…

                        Edit:

                        Turns out it was actually borked SPI firewall rules.

                        The whole thing makes sense. The port 80 worked on other machines on the network because they had different SPI rules, connections within same network also worked because they bypassed SPI. Either via direct connection or NAT loopback.

                          blue Turns out it was actually borked SPI firewall rules.

                          So my hunch regarding a firewall issue was right in the end 🙂

                          Great that you figured this out, have fun

                            No one is typing