Issues with secondary IPv6 adress @ Hetzner cloud

thy1985

Hello mailcow community,

I used forum search and “hetzner” resulted in a few hits with at least closely related issues but no solution yet.

I have a mailcow installation on a Hetzner cloud VM (Ubuntu) that is working fine since I finished setup roughly half a year ago.

However ACME/LE certifcate renewal is problematic. The reason is that the correct (out of two) IPv6 IPs is not detected (or I missed to configure it correctly).

Some basics about my setup:

1 VM that comes with:

Primary IPv4, I will call that “ipv4prim” in the following
Primary IPv6, I will call that “ipv6prim” in the following
Floating IPv4, I will call that ipv4float in the following
Floating IPv6, I will call that ipv6float in the following

The floating IPs are those that are supposed to be used for the mail services (as the primary ones are prone to change when I destroy and recreated the server).

DNS is setup correctly for “mail.example.com” (real name redacted). Most importantly forward and reverse DNS on “mail.example.com” are working for both ipv4float/ipv6float consistently.

The problem with ACME is visible in the following log output:

Tue May 18 20:27:48 CEST 2021 - Found AAAA record for mail.example.com: ipv6float - skipping A record check
Tue May 18 20:27:48 CEST 2021 - Cannot match your IP ipv6prim against hostname mail.example.com (DNS returned ipv6float)
Tue May 18 20:27:48 CEST 2021 - Cannot validate any hostnames, skipping Let’s Encrypt for 1 hour.

Another symptom of the same issue is that when I check DNS via the admin UI:
“Konfiguration” -> “E-Mail-Setup” -> Tab “Domains”, click “?DNS” on the right hand side of the primary domain (example.com) it shows (among other stuff):

Name Typ Korrekte Daten Aktueller Status
mail.example.com A ipv4float green checkmark
taolf4vpi.in-addr.arpa PTR mail.example.com green checkmark
mail.example.com AAA ipv6prim ipv6float
………

As you can see the A record is checked against ipv4float (correct data, that’s great! It correctly ignores ip4vprim).
You can also see that the it considers ipv6prim to be the correct data for the AAA record (but the current status with ipv6float is actually correct/what I want/need).

So my question is how I have to configure my instance so that it uses/expects the ipv6float (in “Korrekte Daten”) instead of ipv6prim when dealing with DNS/ACME - just as it does with IPv4 (where it uses ipv4float and not ipv4prim)?

Any help, pointer, input, suggestion would be very much appreciated. 🙂
Also do not hesitate for more information if I left out anything relevant!

Thanks!

thy1985

Answering myself as I fixed both issue myself now. Hope it might help s/o in a similar situation.

I first digged into how the IPs are detected in the ACME container and found it:
https://github.com/mailcow/mailcow-dockerized/blob/56a085b6324ea48f192c7152dc01605403965075/data/Dockerfiles/acme/functions.sh#L54

Based on that I did tests with the get_ipv6 and get_ipv4 functions on the host itself and inside the acme container.
Funny enough outside and inside the container get_ipv4 was returning the same IP (which is my secondary/floating ip).

get_ipv6 was returning the primary IPv6 both inside and outside of the container. Looking at the implementation of the get_ipX fucntion it got obvious that they rely on the public egress IP (no idea why it is automagically correct for ipv4 here, I would have expected to have the same effect here…?) of the docker containers.

As docker doesn’t let you (up to my knowledge, let me know if you know a way) change that egress IP directly I decided to solve that via ip6tables:
ip6tables -t nat -I POSTROUTING -p all -s {{ dockeripv6nw }} -j SNAT --to-source {{ ipv6 }}

I also had issues with HTTP validation afterwards (for some reason an all interfaces/IP bind obviously did not include my floating ipv6..?). Consequently I also had to add a docker-compose.override.yml to bind to my floating ipv6 explicitly as per https://mailcow.github.io/mailcow-dockerized-docs/firststeps-ip_bindings/.

thy1985

Is there a way to mark it as solved or edit the title of the OP? I seem to be too stupid to find it…

MAGIC

Here 🙂

thy1985

MAGIC Thanks

diekuh

ip6tables -t nat -I POSTROUTING -p all -s {{ dockeripv6nw }} -j SNAT --to-source {{ ipv6 }} is pretty much what SNAT_TO_IPV6 in mailcow.conf does. 🙂

thy1985

@diekuh Thanks a ton! That seems to be the better (from a maintainability point of view) approach.
I’ll go for it.

I found no indication of SNAT_TO_IPV6 but based on your answer I ended up here (which is obviously what you meant): https://mailcow.github.io/mailcow-dockerized-docs/firststeps-snat/
(R.I.P dead beef 😃 )

thy1985

thy1985 It also deals with the IPv6 binding in that case obviously so I had to remove the docker-compose.override.yml to resolve a bind conflict. Nice, even better. :-)

diekuh

Oh, sure, it’s SNAT6_TO_SOURCE. 🙂 Sorry.