Issue with ACME and DNS resolving

maverick

Hi all,

I’ve migrated my server recently and updated all DNS records accordingly.
If I query CloudFlare, OpenDNS, Google, the records come out correct.

When I run the ACME container it always finds wrong IP’s.
I’ve tried clearing unbound container cache, no success, the error keeps the same.

docker-compose exec unbound-mailcow unbound-control dump_cache

acme-mailcow_1       | Fri Jan 22 13:09:58 WET 2021 - Initializing, please wait...
acme-mailcow_1       | Fri Jan 22 13:09:59 WET 2021 - Using existing domain rsa key /var/lib/acme/acme/key.pem
acme-mailcow_1       | Fri Jan 22 13:09:59 WET 2021 - Using existing Lets Encrypt account key /var/lib/acme/acme/account.pem
acme-mailcow_1       | Fri Jan 22 13:09:59 WET 2021 - Detecting IP addresses...
acme-mailcow_1       | Fri Jan 22 13:10:18 WET 2021 - OK: 198.100.10.24, 0000:0000:0000:0000:0000:0000:0000:0000
acme-mailcow_1       | Fri Jan 22 13:10:18 WET 2021 - Found AAAA record for imap.domain.io: 2a02:c805:2023:535::1 - skipping A record check
acme-mailcow_1       | Fri Jan 22 13:10:18 WET 2021 - Cannot match your IP 0000:0000:0000:0000:0000:0000:0000:0000 against hostname imap.domain.io (DNS returned 2a02:c805:2023:0535:0000:0000:0000:0001)
acme-mailcow_1       | Fri Jan 22 13:10:18 WET 2021 - Found AAAA record for smtp.domain.io: 2a02:c805:2023:535::1 - skipping A record check
acme-mailcow_1       | Fri Jan 22 13:10:18 WET 2021 - Cannot match your IP 0000:0000:0000:0000:0000:0000:0000:0000 against hostname smtp.domain.io (DNS returned 2a02:c805:2023:0535:0000:0000:0000:0001)
acme-mailcow_1       | Fri Jan 22 13:10:18 WET 2021 - Found AAAA record for mail.domain.io: 2a02:c805:2023:535::1 - skipping A record check
acme-mailcow_1       | Fri Jan 22 13:10:18 WET 2021 - Cannot match your IP 0000:0000:0000:0000:0000:0000:0000:0000 against hostname mail.domain.io (DNS returned 2a02:c805:2023:0535:0000:0000:0000:0001)

These AAAA records haven’t existed for around 24 hours. I can’t get those records from anywhere on our network (or from outside) except on the mailcow machine that keeps returning wrong records and failing getting the certificates.

omexlu

Is your mailcow installation up-to-date? There was a fix few weeks ago 🙂
If not please update and look again in your logs.

maverick

Hi @omexlu thank you for such a quick reply.
Well, I imagine it’s up to date, I’ve installed it yesterday and cloned from the GitHub repo.

omexlu

So mailcow and your system is all up-to-date?
What says the DNS-Tests in the mailcow installation, there you can see if the DNS-Settings matches the requirements?

maverick

The system is Oracle Linux 8 and is up to date.

This was a system migration, we migrated from an old server to this.
Everything seems to be working: all the inboxes, all the emails are there, accounts, logins, logos… all is correct.

Going to the DNS option on Mail Setup/Domains I get the following:

$ORIGIN domain.io.
mx0	IN	A	198.100.10.24
_25._tcp.mx0	IN	TLSA	110: Operation timed out
_443._tcp.mx0	IN	TLSA	3 1 1 24f25e13b468d8f579bc9a594b5603d03e92ea14a9eeb2df0715129d140d5e8e
_110._tcp.mx0	IN	TLSA	110: Operation timed out
_143._tcp.mx0	IN	TLSA	110: Operation timed out
_465._tcp.mx0	IN	TLSA	110: Operation timed out
_587._tcp.mx0	IN	TLSA	110: Operation timed out
_993._tcp.mx0	IN	TLSA	110: Operation timed out
_995._tcp.mx0	IN	TLSA	110: Operation timed out
_4190._tcp.mx0	IN	TLSA	110: Operation timed out
@	IN	MX	mx0.domain.io.
autodiscover	IN	CNAME	mx0.domain.io.
_autodiscover._tcp	IN	SRV	mx0.domain.io. 443
autoconfig	IN	CNAME	mx0.domain.io.
@	IN	TXT	?
@	IN	TXT	v=spf1 a mx ip4:80.240.210.22 ip6:2a02:c805:2023:535::1 include:domain.io ~all
;_dmarc	IN	TXT	**TODO**
dkim._domainkey	IN	TXT	v=DKIM1;k=rsa;t=s;s=email;p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAtU0kAn2Lnz8cIG4gOfaaZI3tUoJtyb9re4si90C2bfDsOOtCz7MynBaEpTQqwLj9cmt8g9/YoRUEtVJKcP1t8HSopiRI1pTEJfWP+KsW7VCON4e3Nehh/uRR1ENtTqxlE8PPeMSs5aMDoruXlLkSsTBjPuwU0GCgfZGpHXE9CUnK7t1QdD6RwBi5Q7q+879z6nmLW8jbSJ0MHQ8e5he/e1np9uiflfnfPxbNPswWx7HRS/jcM6ZRlSltOpCUP8Xbp0aadkX5hLF3u/M/qAMB90N8viwChXFkqa1wsT4qtSEdrR2UZMPzkzCax3bq6XRSWN/EUO4gJQ3h/gYaVoYLMwIDAQAB

I don’t get the operation timed out…
the first A record is correct. SPF is incorrect, the SPF has been corrected with the new IP address but still showing the old one here.
If I make the query from the router or any other machine alongside the mailcow server, I get the correct answers.

omexlu

operation timed out: could be because blocking maybe by firewall?

For the others records maybe the DNS are not actually changed globaly, maybe try with a VPN from a region where you server is too.

diekuh

Firewalld probably. We don’t support that OS anyway.

maverick

diekuh Oracle Linux is just like CentOS.

If you support CentOS (although you no longer recommend it) its the same. By the way, if I put it on a Red Hat Enterprise Linux, is it supported?
Its a bit obnoxious when people deflect to distro, when the distro has no problems. IIRC mailcow is running on a containerised environment abstracted from the underlying system, which conforms to mailcow requirements.

maverick

diekuh Allow me to add something to my last reply;

I would like to remind that Enterprise Linux systems include:

Red Hat Enterprise Linux
CentOS
Amazon Linux
Oracle Linux
Fedora
CloudLinux
CoreOS

And in a mere sharing of opinion, that despising the users of the distros above may not be the best for business. I’ve worked with many giant tech companies, I’ve been in datacenters all around Europe configuring systems, most of them are not Debian nor debian based. Not in production servers for sure. People that pay 2000eur/year for Oracle Support may as well pay some for mailcow support. But maybe you’re not all that interested in those customers is what I may assume?

As I clearly stated above, firewalld is stopped.

# firewall-cmd --list-all
FirewallD is not running

Was, but is no more. Completely disabled. Issue persists.

maverick

Firewall isn’t blocking. We have pfSense firewalls upstream with HA and all the services and machines resolve everything correctly.
On the mailcow machine, even tho we are in 2021 I even disabled firewalld so its all up to iptables as recommended.

After disabling firewalld instead of the timeout error I get a new one:
0: php_network_getaddresses: getaddrinfo failed: Name does not resolve

I’m not sure what you meant by “for the other records maybe the DNS are not actually changed globally”

mxtoolbox resolves correctly.

Querying via WAN to the Authoritative name server:

% dig @100.110.10.1 _25._tcp.mx0.domain.io TLSA

; <<>> DiG 9.10.6 <<>> @100.110.10.1 _25._tcp.mx0.domain.io TLSA
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11308
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 3
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;_25._tcp.mx0.domain.io.		IN	TLSA

;; ANSWER SECTION:
_25._tcp.mx0.domain.io.	86400	IN	TLSA	3 1 1 24F25E13B468D8F579BC9A594B5603D03E92EA14A9EEB2DF0715129D 140D5E8E

;; AUTHORITY SECTION:
domain.io.		86400	IN	NS	ns01.domain.io.
domain.io.		86400	IN	NS	ns13.domain.io.

;; ADDITIONAL SECTION:
ns01.domain.io.		1200	IN	A	x.x.x.x
ns13.domain.io.		1200	IN	A	x.x.x.x

;; Query time: 51 msec
;; SERVER: 100.110.10.1#53(100.110.10.1)
;; WHEN: Fri Jan 22 14:16:57 WET 2021
;; MSG SIZE  rcvd: 166

Query to Cloudflare 1.1.1.1:

% dig @1.1.1.1 _25._tcp.mx0.domain.io TLSA 

; <<>> DiG 9.10.6 <<>> @1.1.1.1 _25._tcp.mx0.domain.io TLSA
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49485
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_25._tcp.mx0.domain.io.		IN	TLSA

;; ANSWER SECTION:
_25._tcp.mx0.domain.io.	86400	IN	TLSA	3 1 1 24F25E13B468D8F579BC9A594B5603D03E92EA14A9EEB2DF0715129D 140D5E8E

;; Query time: 960 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Fri Jan 22 14:17:17 WET 2021
;; MSG SIZE  rcvd: 96

Query to Google

% dig @8.8.4.4 _25._tcp.mx0.domain.io TLSA

; <<>> DiG 9.10.6 <<>> @8.8.4.4 _25._tcp.mx0.domain.io TLSA
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 14988
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;_25._tcp.mx0.domain.io.		IN	TLSA

;; ANSWER SECTION:
_25._tcp.mx0.domain.io.	21599	IN	TLSA	3 1 1 24F25E13B468D8F579BC9A594B5603D03E92EA14A9EEB2DF0715129D 140D5E8E

;; Query time: 83 msec
;; SERVER: 8.8.4.4#53(8.8.4.4)
;; WHEN: Fri Jan 22 14:22:57 WET 2021
;; MSG SIZE  rcvd: 96

Query to the router address in /etc/resolv.conf:

% dig @10.0.1.254 _25._tcp.mx0.domain.io TLSA

; <<>> DiG 9.10.6 <<>> @10.49.1.254 _25._tcp.mx0.domain.io TLSA
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19565
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;_25._tcp.mx0.domain.io.		IN	TLSA

;; ANSWER SECTION:
_25._tcp.mx0.domain.io.	86395	IN	TLSA	3 1 1 24F25E13B468D8F579BC9A594B5603D03E92EA14A9EEB2DF0715129D 140D5E8E

;; Query time: 50 msec
;; SERVER: 10.0.1.254#53(10.0.1.254)
;; WHEN: Fri Jan 22 14:24:03 WET 2021
;; MSG SIZE  rcvd: 96

To me it seems that the DNS servers around are working fine and give the correct answers.
Any machine connected to the same subnet resolves and makes queries without issues.

diekuh

maverick People that pay 2000eur/year for Oracle Support may as well pay some for mailcow support. But maybe you’re not all that interested in those customers is what I may assume?

Sigh.

It’s a network issue. mailcow still works on C8, it just happened to totally switch what it is and can break any time IMO. So no, I don’t recommend it.

The fact your system seems not to be able to query the root servers will be an underlying network issue. Happens with Hetzner for example, depending on the configuration.

It sounds like you WANT this to be a mailcow bug and are getting angry. That’s a bit confusing.

maverick

Testing name resolving from acme container:

# docker exec -it $(docker ps -qf name=acme-mailcow) dig +short autoconfig.domain.io @1.1.1.1
mail.domain.io.
198.100.10.24

# docker exec -it $(docker ps -qf name=acme-mailcow) dig +short autoconfig.domain.io @172.22.1.254
mail.domain.io.
80.240.210.22

Querying to predefined docker network address always returns bad results. Results you can’t get from anywhere else.

maverick

Hi @diekuh,

Let me tackle your reply per topics:

I’m not using C8 (imagine that’s CentOS 8). I clearly mentioned above “Oracle Linux 8” which is exactly alike Red Hat Enterprise Linux 8. I can also put mailcow on a RHEL server if that’s better for you. They’re both fully fledged production systems, not community driven Linux distros used in college;
I’m sure there’s an underlying issue. But the issue is with mailcow. Mailcow is able to return results from god knows where, when all queries inside the same network return the proper results.
I don’t “want” it to be a mailcow issue and getting angry. Lol getting angry 🙂 you’re wannabe smartass remarks are as ridiculous as most the official interventions around here.

It’s never a problem with mailcow:
It’s a problem with the distro, because you only know debian-based distros so everything else is a conundrum;
It’s a problem with firewalld, even if the issue has nothing to do with it but you’ve seen a user saying it had it on so the problem is automatically it; (its really really sad)
It’s a problem with the network, because despite the whole underlying infrastructure being well configured, and working for a plenitude of servers with a wide variety of provided services, despite only mailcow showing problems its automatically something else problem.

Looking around your forums and GitHub its very easy to spot the dozens of solicitations for some help and the obnoxious answers form you people. It’s never a mailcow problem, the problem is ALWAYS with something else.

So, angry: not at all. Sad at most, because its sad that a reasonably interesting software has such poor people behind, people that are unpleasant to those who reach out for help etc.

Our internal DNS resolvers also use unbound and query the roots directly. Never had issues resolving anything. Only mailcow. But please, please keep telling me how is nothing about mailcow.

diekuh

Okay Maverick, I got your point. Fix your network though. 🙂 Querying the roots from within mailcow over a NAT’ed connection is different from querying roots directly. And I am sure that’s where the problem is.

But hey, it’s a bug with mailcow. You will probably tell me what this bug is about, help me replicate it (I cannot, I tried) or create a PR to fix it, right? 🙂 Or do you expect me to do some free consulting? I’m sorry, but I can’t. Time is rare.

maverick

Network is working fine. Overriding mailcow unbound to use the internal resolvers works just fine. Our resolvers use unbound too, but hey, seems our unbound setup is inferior to yours, cause our works fine. Its such a burden to fix working networks, I just rather put the fix on that part you say aint broken. this forum misses facepalm emojis.

Edit: There’s a bunch of people around here on the forums and on GitHub complaining about the same issue. Go and debug them, I have better things to do than to amuse people such as you.

diekuh

It is working for every single machine in support or managed. Every single setup is working fine. If I was able to fix that on mailcows side, I would! There is just no way I can offer free remote hands to debug your servers; how would that work? Do you want to do my work while I’m helping you?

diekuh

Okay, then explain how it is working fine for us? 🙂 I’m really curious. You seem to be angry that you cannot solve your problem, so I will leave to let you cool down a bit.

maverick

Lol you keep missing the point 🙂

I’m not angry. At most I’d feel sad for you;
I’ve also put it to work with a few customisations. If the machines are managed by you, who tells me you haven’t such fixes in place?
What it tells me is that you guys deny any kind of support or solutions out in the open to push people for paid support, again that’s sad;

I find it funny cause you keep insisting like I still have any issue. I don’t, everything is fixed and working fine with a couple tweaks. I’m not asking for help, I’m sharing an issue.

This forum really lacks a facepalm emoji.

SnowWhite

yeah, this is what they call community support these days… LOL

maverick

SnowWhite hi man, currently we’re only using mailcow on one server for an charity project, all the others we have migrated to zimbra.
It’s not only the bad support, its that these people involved on this project are rude, lack basic education, and always push problems to “something else” its never their problem.

Recently I opened a bug on mailcow-dockerized github Issue 4158
I did all the steps asked to fill the bug report, searched other solutions and tried to apply them and posted the results. They simply ignored my issue. After I bumped the issue, some guy comes around telling its a domain configuration problem on my end. Every information I provided proved otherwise.

Finally that “andryyy” comes around and replies “sigh, you again” and closes the issue without any technical approach to the issue at hand.

This actually speaks more about the kind of people these guys are, their competence and the business they’re running. But well, its their problem actually. It’s just a reflection of the sad little people they are, and the inability to debate technical issues, as its always someone else’s fault. I have come to pity them, but life goes on and there are many email softwares out there.

SnowWhite

maverick my stuff got censored right away as I brought forth the gaping holes in what they call “documentation” around here. Well… there is always Mail-In-A-Box and iREDmail of course…