Lets Encrypt Error

cancan

I am seeing this error in the ACME logs now:

“Cannot validate any hostnames, skipping Let’s Encrypt for 1 hour.”

Any idea why this is happening?

01/20/2021, 01:06:13 PM mail.xxxxxx.com - Use SKIP_LETS_ENCRYPT=y in mailcow.conf to skip it permanently. 01/20/2021, 01:06:13 PM mail.xxxxxx.com - Cannot validate any hostnames, skipping Let's Encrypt for 1 hour. 01/20/2021, 01:06:13 PM mailxxxxxx.com - Confirmed A record with IP xxx.xxx.xxx.xxx, but HTTP validation failed 01/20/2021, 01:05:41 PM mail.xxxxxx.com - Found A record for mail.xxxxxx.com: xxx.xxx.xxx.xxx

MAGIC

What’s your setup?
Do you use a reverse proxy in front of your mailcow?

cancan

No I do not. Its a vanilla install on a VPS at Ramnode. ACME was working fine originally, it seems after one of the updates this happened.

Not sure how/where to start troubleshooting..

maverick

Hi there,
I’m having the exact same error. Reported here: https://community.mailcow.email/d/611-issue-with-acme-and-dns-resolving

Looking on google I also saw a bunch of people asking for the same problem on GitHub and even here on the forum some other posts without any solution.
One guy on GitHub said he used google public DNS gave me the shivers: mailcow/mailcow-dockerized3588
We don’t use public DNS servers. We have caching nameservers with unbound. Tried and tested my post querying our resolvers directly we get the correct answers.

I’ve even added to our docker-compose.override.yml as suggested by method B here: https://mailcow.github.io/mailcow-dockerized-docs/u_e-unbound-fwd/

And we got it “more or less” working. While looking at the container logs we see correct IP resolution, HTTP validation fails.
But that’s actually not what we consider more interesting. What we consider most interesting is that even with this new config (with the override) while on the Domain management in mailcow UI the “DNS” button keeps reporting 110: Operation timed out
What is funny is that if we remove the override, it will AGAIN give incorrect replies of DNS records that don’t exist.

We were considering paying support for mailcow but after looking carefully through some of the answers of some mods and even some developers on GitHub we realised we are not interested in paying support to mailcow, instead we have people looking for other email solutions.
We would rather have an inferior software application with an outstanding support than a slightly better software with a dreadful support. And what we have seen most around here is the dreadful support, many topics left unanswered and without solution, and unpleasant comments from mods on github too. I think its a shame, considering the replies are from official mailcow people, but that gives us a hint on what to expect.

cancan

Just wait till everyones certificates start expiring soon…

diekuh

cancan sorry but this will not happen. Perhaps you can give us more I Formation about the OS, firewall etc.

Waiting for other to run into this will not happen.

maverick

Actually is funny because you change a few parameters and test again and it always comes out wrong sometimes it says there’s only AAAA records - there are no current AAAA records for this server - or some IP that is not reflected anywhere. You clear the unbound cache all the times you want but it still gets poisoned records god knows where from.

In the following excerpt you can see the results returned from docker-dns as configured by default, (comes empty) for existing records.

As I showed on the other topic, querying public DNS servers always returns correct results: be it cloudflare, google, OpenDNS or any other you may remember about. Our DNS Resolvers, as shown below, also return correct results.

The following shows:

acme-mailcow log;
query from acme-mailcow container to the docker network address 172.22.1.254;
verification of what is the configured resolver on the system;
query from acme-mailcow container to the system configured resolver;

[root@mx0 mailcow-dockerized]# docker-compose logs --tail=200 -f acme-mailcow
Attaching to mailcowdockerized_acme-mailcow_1
acme-mailcow_1       | Fri Jan 22 19:41:36 WET 2021 - Waiting for Docker API...
acme-mailcow_1       | Fri Jan 22 19:41:36 WET 2021 - Docker API OK
acme-mailcow_1       | Fri Jan 22 19:41:36 WET 2021 - Waiting for Postfix...
acme-mailcow_1       | Fri Jan 22 19:41:36 WET 2021 - Postfix OK
acme-mailcow_1       | Fri Jan 22 19:41:36 WET 2021 - Waiting for Dovecot...
acme-mailcow_1       | Fri Jan 22 19:41:36 WET 2021 - Dovecot OK
acme-mailcow_1       | Fri Jan 22 19:41:36 WET 2021 - Waiting for database...
acme-mailcow_1       | Fri Jan 22 19:41:36 WET 2021 - Database OK
acme-mailcow_1       | Fri Jan 22 19:41:36 WET 2021 - Waiting for Nginx...
acme-mailcow_1       | Fri Jan 22 19:41:38 WET 2021 - Nginx OK
acme-mailcow_1       | Fri Jan 22 19:41:38 WET 2021 - Waiting for resolver...
acme-mailcow_1       | Fri Jan 22 19:41:39 WET 2021 - Resolver OK
acme-mailcow_1       | Fri Jan 22 19:41:39 WET 2021 - Waiting for domain table...
acme-mailcow_1       | OK
acme-mailcow_1       | Fri Jan 22 19:41:39 WET 2021 - Initializing, please wait...
acme-mailcow_1       | unable to load certificate
acme-mailcow_1       | 140344576671048:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
acme-mailcow_1       | unable to load certificate
acme-mailcow_1       | 140590739037512:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
acme-mailcow_1       | unable to load certificate
acme-mailcow_1       | 140369987378504:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
acme-mailcow_1       | unable to load certificate
acme-mailcow_1       | 140569254317384:error:0909006C:PEM routines:get_name:no start line:crypto/pem/pem_lib.c:745:Expecting: TRUSTED CERTIFICATE
acme-mailcow_1       | Fri Jan 22 19:41:39 WET 2021 - Using existing domain rsa key /var/lib/acme/acme/key.pem
acme-mailcow_1       | Fri Jan 22 19:41:39 WET 2021 - Using existing Lets Encrypt account key /var/lib/acme/acme/account.pem
acme-mailcow_1       | Fri Jan 22 19:41:39 WET 2021 - Detecting IP addresses...
acme-mailcow_1       | Fri Jan 22 19:41:59 WET 2021 - OK: 198.100.10.24, 0000:0000:0000:0000:0000:0000:0000:0000
acme-mailcow_1       | Fri Jan 22 19:41:59 WET 2021 - Found A record for imap.domain.io: 80.240.210.22
acme-mailcow_1       | Fri Jan 22 19:41:59 WET 2021 - Cannot match your IP 198.100.10.24 against hostname imap.domain.io (DNS returned 80.240.210.22)
acme-mailcow_1       | Fri Jan 22 19:41:59 WET 2021 - Found A record for smtp.domain.io: 80.240.210.22
acme-mailcow_1       | Fri Jan 22 19:41:59 WET 2021 - Cannot match your IP 198.100.10.24 against hostname smtp.domain.io (DNS returned 80.240.210.22)
acme-mailcow_1       | Fri Jan 22 19:41:59 WET 2021 - Found A record for mail.domain.io: 80.240.210.22
acme-mailcow_1       | Fri Jan 22 19:41:59 WET 2021 - Cannot match your IP 198.100.10.24 against hostname mail.domain.io (DNS returned 80.240.210.22)
acme-mailcow_1       | Fri Jan 22 19:41:59 WET 2021 - No A or AAAA record found for hostname mx.domain.io
acme-mailcow_1       | Fri Jan 22 19:41:59 WET 2021 - No A or AAAA record found for hostname webmail.domain.io
acme-mailcow_1       | Fri Jan 22 19:41:59 WET 2021 - Found A record for autodiscover.domain.io: 80.240.210.22
acme-mailcow_1       | Fri Jan 22 19:41:59 WET 2021 - Cannot match your IP 198.100.10.24 against hostname autodiscover.domain.io (DNS returned 80.240.210.22)
acme-mailcow_1       | Fri Jan 22 19:41:59 WET 2021 - Found A record for autoconfig.domain.io: 80.240.210.22
acme-mailcow_1       | Fri Jan 22 19:41:59 WET 2021 - Cannot match your IP 198.100.10.24 against hostname autoconfig.domain.io (DNS returned 80.240.210.22)
acme-mailcow_1       | Fri Jan 22 19:41:59 WET 2021 - Found A record for mx0.domain.io: 198.100.10.24
acme-mailcow_1       | (skipping check, returning 0)
acme-mailcow_1       | Fri Jan 22 19:41:59 WET 2021 - Confirmed A record 198.100.10.24
acme-mailcow_1       | Fri Jan 22 19:41:59 WET 2021 - Certificate /var/lib/acme/mx0.domain.io/cert.pem validation done, neither changed nor due for renewal.
acme-mailcow_1       | Fri Jan 22 19:41:59 WET 2021 - Certificates were successfully validated, no changes or renewals required, sleeping for another day.
^CERROR: Aborting.
[root@mx0 mailcow-dockerized]# docker exec -it $(docker ps -qf name=acme-mailcow) dig +short webmail.domain.io @172.22.1.254
[root@mx0 mailcow-dockerized]# cat /etc/resolv.conf 
# Generated by NetworkManager
search domain.io
nameserver 10.0.1.254
[root@mx0 mailcow-dockerized]# docker exec -it $(docker ps -qf name=acme-mailcow) dig +short webmail.domain.io @10.0.1.254
mail.domain.io.
198.100.10.24
[root@mx0 mailcow-dockerized]#

I remember a few months back a colleague around with some debian system reported issues with getaddrinfo() function or something like that. I have never had such problems in EL systems. I really would LOVE to know where are the wrong records coming from.

tbrummell

So I just brought up a Debian 10 box to replace an older CentOS 7 box, did a mail migration, and now ACME is failing as shown above:
acme-mailcow_1 | Fri Feb 5 13:35:04 EST 2021 - Found A record for mailcow.my.domain: 172.22.1.11 acme-mailcow_1 | Fri Feb 5 13:35:04 EST 2021 - Cannot match your IP 23.95.x.x against hostname mailcow.my.domain (DNS returned 172.22.1.11)Other records such as autodiscover and autoconfig all get resolved/matched fine. It’s just the primary name of the box that fails. Idea’s?
This is a bone stock Debian10 install on RackNerd, nothing in front or behind, just Debain10.

Edit: my workaround was to set Skip IP check in mailcow.conf. At least my browser is happy again.

maverick

tbrummell
Hi there,
Just saw your post and I was thinking to myself…

I think this container has problems and needs some review. But mailcow devs just say that no, everything’s alright, its always the users fault. Because you have firewalld enabled, because its an “unsupported distro”, because “on their install is working”, because… hell there’s all kinds of excuses specially on GitHub for things not get followed up. So I gave up on that.
I’m also not fond of debian based distros. Whenever we run stuff on Debian distros we get issues we’ve never seen before (and of which stack overflow and other technical forums are full of) some with the network stack… but I don’t want to digress…

Since Red Hat now allows you to have a free licence and up to 16 production servers, we now are shifting towards a Red Hat + Oracle Linux mixed environment. Oracle Linux is free, their Linux team is doing a great job bringing more packages to the yum repos, and they also have a tool that allows you to directly convert your system from CentOS to Oracle Linux (from experience, the lesser third party repos, the better). But we’ve used it successfully on a few servers and others we migrated the apps like from CentOS7 to OL8 as usual.

Going back to mailcow, all the containers are built on debian. I’m not sure where the issue lies, although I have my suspicions. But anyway, we have our own internal resolvers (that actually also run unbound, but our unbound machines all work fine) so we simply add the following to data/conf/unbound/unbound.conf

forward-zone:
  name: "."
  forward-addr: 10.53.10.11
  forward-addr: 10.78.2.53

replace the ip’s with the external resolvers you want to use and restart the unbound container.
-> fixed

tbrummell

Yeah, but I want to use unbound, not Google or anything. And my personal DNS (using Unbound) isn’t public facing. I guess I could make it so it was, but I’d rather Unbound on Mailcow just query root servers itself.

maverick

tbrummell
our resolvers aren’t public resolvers either, they’re only accessible from within the network.
And they are unbound too.

Our unbound (vanilla setup) works fine. Mailcow’s doesn’t.
So we point mallow’s unbound container to our unbound. Not public facing, and problem solved.

Good luck to you.