Okay actually I am unable to reach my ubuntu on port 80, only 443. I cannot find the reason for that, something terrible must have happened. I tried tcpdump on port 80 and I can only see outgoing traffic, no firewall rules, no iptables. Maybe my ISP stealtlhy blocked port 80?

To be absolutely sure check the output of

ufw status
systemctl status firewalld

and to be 100% certain

systemctl list-units --type=service | grep firewall

If they are all negative, check your router, it also has a firewall (I understand you are using mailcow at home)

  • blue replied to this.

    DocFraggle

    -ufw inactive
    -no firewalld
    -nothing under systemctl list-units –type=service | grep firewall

    I tested port 80 by forwarding to different computer on my network - works and is reachable from outside world.

    It just seems that mailcow ABSOLUTELY refuses to let me connect on port 80 and I’ve no clue why.

    On another note

    sudo ss -lptn 'sport = :80'
    State          Recv-Q         Send-Q                 Local Address:Port                 Peer Address:Port         Process
    LISTEN         0              4096                         0.0.0.0:80                        0.0.0.0:*             users:(("docker-proxy",pid=1116544,fd=4))
    LISTEN         0              4096                            [::]:80                           [::]:*             users:(("docker-proxy",pid=1116552,fd=4))
    
    sudo ss -lptn 'sport = :443'
    State          Recv-Q         Send-Q                 Local Address:Port                 Peer Address:Port         Process
    LISTEN         0              4096                         0.0.0.0:443                       0.0.0.0:*             users:(("docker-proxy",pid=1116516,fd=4))
    LISTEN         0              4096                            [::]:443                          [::]:*             users:(("docker-proxy",pid=1116524,fd=4))

    Which seems correct?

    Maybe just restart everything…

    docker compose down
    
    reboot

    then

    docker compose up -d

    • blue replied to this.

      DocFraggle

      Unfortunately didnt help.
      Do you have any way to check whats blocking port 80? I have other services running on the same server so there might be some interference but I couldn’t find anything with my linux skills.

      What exactly is the output of

      ss -lptn 'sport = :80'

      on your mailcow host? I couldn’t make out if you were running the command on your mailcow host or “On another note”

      Ah sorry, so the above was the output of your mailcow server. Whats the output if you use curl on your mailcow host (replace your.mailcow.fwdn whith your mailcow FQDN)? It should look like this:

      curl -v -H "Host: your.mailcow.fqdn" http://127.0.0.1:80
      *   Trying 127.0.0.1:80...
      * Connected to 127.0.0.1 (127.0.0.1) port 80 (#0)
      > GET / HTTP/1.1
      > Host: your.mailcow.fqdn
      > User-Agent: curl/7.88.1
      > Accept: */*
      > 
      < HTTP/1.1 301 Moved Permanently
      < Server: nginx/1.27.2
      < Date: Tue, 22 Oct 2024 12:07:31 GMT
      < Content-Type: text/html
      < Content-Length: 169
      < Connection: keep-alive
      < Location: https://your.mailcow.fqdn/
      < 
      <html>
      <head><title>301 Moved Permanently</title></head>
      <body>
      <center><h1>301 Moved Permanently</h1></center>
      <hr><center>nginx/1.27.2</center>
      </body>
      </html>
      * Connection #0 to host 127.0.0.1 left intact
      • blue replied to this.

        DocFraggle

        Im doing something wrong because result of my curl is the HTML of admin panel.

        On another note I can get curl for the following situations:
        port 443 - outside network and inside local network
        port 80 - only inside local network

        Which is very bizarre! I have no firewall to block port 80, my mailcow is behind a NAT so it shouldn’t even know whats outside network traffic, and I tested my NAT and port 80 with another computer and it worked fine.

        -When Im on local network I can resolve mailcow curl via (its local ip address, its domain, its publc ip address)

        Whcih admin panel? The mailcow admin UI? Are you using some kind of reverse proxy? Nginx proxy manager or stuff like that?

        • blue replied to this.

          DocFraggle

          No proxy.

          Yeah I think its admin UI starts off like this

          < Server: nginx
          < Date: Tue, 22 Oct 2024 12:43:50 GMT
          < Content-Type: text/html; charset=utf-8
          < Transfer-Encoding: chunked
          < Connection: keep-alive
          < Vary: Accept-Encoding
          < Set-Cookie: PHPSESSID=e82f7a5c83a4a2ca6fad2e7e843f0406; path=/; HttpOnly
          < Expires: Thu, 19 Nov 1981 08:52:00 GMT
          < Cache-Control: no-store, no-cache, must-revalidate
          < Pragma: no-cache
          < Strict-Transport-Security: max-age=15768000;
          < X-Content-Type-Options: nosniff
          < X-XSS-Protection: 1; mode=block
          < X-Robots-Tag: none
          < X-Download-Options: noopen
          < X-Frame-Options: SAMEORIGIN
          < X-Permitted-Cross-Domain-Policies: none
          < Referrer-Policy: strict-origin

          To summarise:

          1. I can resolve my server from FQND and public ip as long as its on any port other than 80 - for example 443
          2. I can resolve any other server on my network, just not the one with mailcow - something is broken.
          3. I can resolve mailcow on port 80 as long as its from local network.
          4. No proxy
          5. It worked fine before and broke for unknown reason.
          6. No firewall on my router, no firewall on my mailcow server - I am able to resolve locally.
          7. I am behind NAT so mailcow should have no idea whats ‘outside’ traffic yet for some reason local network can reach it but not outside network.

          Puzzling indeed.

          ah ok, it’s different because I redirect HTTP to HTTPS 🙂

          docs.mailcow.email Icon Redirect HTTP to HTTPS - mailcow: dockerized documentation

          Last idea: what did you set these to in mailcow.conf?

          HTTP_PORT=80
          HTTP_BIND=
          HTTPS_PORT=443
          HTTPS_BIND=
          • blue replied to this.

            DocFraggle

            Yeah its exactly that. I;m losing hope I can ever fix this 🙁

            Just to be sure sudo ss -lptn 'sport = :80' is supposed to show two entries?

            LISTEN               0                    4096                                     0.0.0.0:80                                      0.0.0.0:*                   users:(("docker-proxy",pid=3316,fd=4))
            
            LISTEN               0                    4096                                        [::]:80                                         [::]:*                   users:(("docker-proxy",pid=3344,fd=4))

            Okay after enough rebooting, destroying and flushing iptables the issue was solved by restarting the router…

            Edit:

            Turns out it was actually borked SPI firewall rules.

            The whole thing makes sense. The port 80 worked on other machines on the network because they had different SPI rules, connections within same network also worked because they bypassed SPI. Either via direct connection or NAT loopback.

              blue Turns out it was actually borked SPI firewall rules.

              So my hunch regarding a firewall issue was right in the end 🙂

              Great that you figured this out, have fun

              No one is typing