Unbound issue, probably causing slowness, DNS simplification?

jonathanbrickman0000

Was noticing SMTP submissions a bit slow. Below is a snippet from docker compose logs --tail=200 unbound-mailcow. I imagine this is due to something specific to Ubuntu Server 24. What is the best way to simplify? I’m guessing that it may have something to do with Ubuntu’s 127.0.0.1/127.0.0.53 DNS complexity.

unbound-mailcow-1 | [1742390222] unbound[17:0] info: generate keytag query _ta-4f66. NULL IN unbound-mailcow-1 | 2025-03-19 08:17:10: Healthcheck: DNS Resolution Failed on attempt 1 for hub.docker.com! Trying again... unbound-mailcow-1 | 2025-03-19 10:56:45: Healthcheck: DNS Resolution Failed on attempt 1 for github.com! Trying again... unbound-mailcow-1 | 2025-03-19 12:19:46: Healthcheck: DNS Resolution Failed on attempt 1 for hub.docker.com! Trying again... unbound-mailcow-1 | 2025-03-19 14:38:09: Healthcheck: Failed to ping 1.1.1.1 on attempt 1. Trying again... unbound-mailcow-1 | 2025-03-19 14:38:21: Healthcheck: Failed to ping 1.1.1.1 on attempt 2. Trying again... unbound-mailcow-1 | 2025-03-19 14:38:33: Healthcheck: Failed to ping 1.1.1.1 on attempt 3. Trying again... unbound-mailcow-1 | 2025-03-19 14:38:33: Healthcheck: Couldn't ping 1.1.1.1 after 3 attempts. Marking this IP as failed. root@smtp-relay:/opt/mailcow-dockerized/data/conf/unbound# ping 1.1.1.1 PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data. 64 bytes from 1.1.1.1: icmp_seq=1 ttl=63 time=1.08 ms 64 bytes from 1.1.1.1: icmp_seq=2 ttl=63 time=0.749 ms 64 bytes from 1.1.1.1: icmp_seq=3 ttl=63 time=0.565 ms 64 bytes from 1.1.1.1: icmp_seq=4 ttl=63 time=0.778 ms 64 bytes from 1.1.1.1: icmp_seq=5 ttl=63 time=0.824 ms

esackbauer

So you have configured DNS forwarders? That is not best practice.
unbound should use the root servers (default config).
Either way you need to open port 53 TCP and UDP outbound, for both IPv4 and IPv6.

Your ping check just tells us that ICMP is passing through. Try investigate with dig and use the trace option.

jonathanbrickman0000

DNS forwarders are not configured. And yet Unbound cannot seem to do lookups (below as above). I have tested with external DNS and it works. But I want to use the recommended. I ‘dig’ or ‘nslookup’ fuzzy.mailcow.email, or github.com, from the system command line, works just fine. I’m theorizing that this is Ubuntu’s systemd resolv system getting in the way, but am not sure how best to handle it.

unbound-mailcow-1 | 2025-03-20 09:14:06,894 INFO success: unbound-healthcheck entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) unbound-mailcow-1 | [1742480053] unbound[18:0] info: generate keytag query _ta-4f66. NULL IN unbound-mailcow-1 | 2025-03-20 09:14:13: Healthcheck: DNS Resolution Failed on attempt 1 for fuzzy.mailcow.email! Trying again... unbound-mailcow-1 | 2025-03-20 09:14:17: Healthcheck: DNS Resolution Failed on attempt 1 for github.com! Trying again...

DocFraggle

Do you have ufw and/or selinux enabled? If so, disable both completly, restart mailcow and try again. Maybe even reboot after disabling

jonathanbrickman0000

Well, that was interesting. I disabled AppArmor and selinux altogether, rebooted. Unbound still shows errors (see below), but connect by TLS/465 works consistently, but STARTTLS/587 doesn’t work at all (see postfix further below).

Unbound:
unbound-mailcow-1 | Receiving anchor key... unbound-mailcow-1 | Receiving root hints... ######################################################################## 100.0% unbound-mailcow-1 | setup in directory /etc/unbound unbound-mailcow-1 | removing artifacts unbound-mailcow-1 | Setup success. Certificates created. Enable in unbound.conf file to use unbound-mailcow-1 | 2025-03-20 10:49:05,595 INFO Set uid to user 0 succeeded unbound-mailcow-1 | 2025-03-20 10:49:05,610 INFO supervisord started with pid 1 unbound-mailcow-1 | 2025-03-20 10:49:06,612 INFO spawned: 'processes' with pid 16 unbound-mailcow-1 | 2025-03-20 10:49:06,618 INFO spawned: 'syslog-ng' with pid 17 unbound-mailcow-1 | 2025-03-20 10:49:06,621 INFO spawned: 'unbound' with pid 18 unbound-mailcow-1 | 2025-03-20 10:49:06,630 INFO spawned: 'unbound-healthcheck' with pid 19 unbound-mailcow-1 | [1742485746] unbound[18:0] notice: init module 0: validator unbound-mailcow-1 | [1742485746] unbound[18:0] notice: init module 1: iterator unbound-mailcow-1 | [1742485746] unbound[18:0] info: start of service (unbound 1.20.0). unbound-mailcow-1 | Mar 20 10:49:06 3d3321b35b0a syslog-ng[17]: syslog-ng starting up; version='4.7.1' unbound-mailcow-1 | 2025-03-20 10:49:07,937 INFO success: processes entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) unbound-mailcow-1 | 2025-03-20 10:49:07,938 INFO success: syslog-ng entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) unbound-mailcow-1 | 2025-03-20 10:49:07,938 INFO success: unbound entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) unbound-mailcow-1 | 2025-03-20 10:49:07,938 INFO success: unbound-healthcheck entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) unbound-mailcow-1 | [1742485748] unbound[18:0] info: generate keytag query _ta-4f66. NULL IN unbound-mailcow-1 | 2025-03-20 10:49:14: Healthcheck: DNS Resolution Failed on attempt 1 for fuzzy.mailcow.email! Trying again... unbound-mailcow-1 | 2025-03-20 10:49:18: Healthcheck: DNS Resolution Failed on attempt 1 for hub.docker.com! Trying again...

postfix:
postfix-mailcow-1 | Mar 20 10:53:28 b337686ca9e3 postfix/submission/smtpd[375]: connect from unknown[172.16.1.254] postfix-mailcow-1 | Mar 20 10:53:28 b337686ca9e3 postfix/submission/smtpd[375]: improper command pipelining after CONNECT from unknown[172.16.1.254]: \026\003\001\002\000\001\000\001\374\003\003U\356z\002\037\344\376\242\0250\203\306\261\345\315\332\240\227~\177\355"\344\340@?&q\214\363\277h \257\235\027\0318\034\032\274\321\001\204J\373\340\255\207'\276<\273\250\312\256\216\316\r3\2440#\342J\000"\023\001\023\003\023\002\300+\300/\314\251\314\250\300,\3000\300\n\300\t postfix-mailcow-1 | Mar 20 10:55:08 b337686ca9e3 postfix/submission/smtpd[375]: lost connection after CONNECT from unknown[172.16.1.254] postfix-mailcow-1 | Mar 20 10:55:08 b337686ca9e3 postfix/submission/smtpd[375]: disconnect from unknown[172.16.1.254] commands=0/0 postfix-mailcow-1 | Mar 20 10:58:28 b337686ca9e3 postfix/anvil[370]: statistics: max connection rate 1/60s for (smtps:172.16.1.254) at Mar 20 10:52:54 postfix-mailcow-1 | Mar 20 10:58:28 b337686ca9e3 postfix/anvil[370]: statistics: max connection count 1 for (smtps:172.16.1.254) at Mar 20 10:52:54 postfix-mailcow-1 | Mar 20 10:58:28 b337686ca9e3 postfix/anvil[370]: statistics: max cache size 2 at Mar 20 10:53:28 postfix-mailcow-1 | Mar 20 10:58:53 b337686ca9e3 postfix/submission/smtpd[387]: connect from unknown[20.169.106.93] postfix-mailcow-1 | Mar 20 10:58:53 b337686ca9e3 postfix/submission/smtpd[387]: improper command pipelining after CONNECT from unknown[20.169.106.93]: MGLNDD_12.236.200.190_587\n postfix-mailcow-1 | Mar 20 10:58:53 b337686ca9e3 postfix/submission/smtpd[387]: lost connection after UNKNOWN from unknown[20.169.106.93] postfix-mailcow-1 | Mar 20 10:58:53 b337686ca9e3 postfix/submission/smtpd[387]: disconnect from unknown[20.169.106.93] unknown=0/1 commands=0/1

DocFraggle

Did you disable ufw as well?

Try running

dig +trace fuzzy.mailcow.email

inside the unbound container and paste the output

jonathanbrickman0000

Looks like ‘dig’ is not available in that container? Also not ‘bash’.

`
mailcowdockerized-unbound-mailcow-1
1a0fb7c1e98e memcached:alpine “docker-entrypoint.s…” 20 hours ago Up 42 minutes 11211/tcp mailcowdockerized-memcached-mailcow-1
root@smtp-relay:/opt/mailcow-dockerized# docker exec -it 1a0fb7c1e98e
docker: ‘docker exec’ requires at least 2 arguments

Usage: docker exec [OPTIONS] CONTAINER COMMAND [ARG…]

See ‘docker exec –help’ for more information
root@smtp-relay:/opt/mailcow-dockerized# docker exec -it 1a0fb7c1e98e bash
OCI runtime exec failed: exec failed: unable to start container process: exec: “bash”: executable file not found in $PATH: unknown
root@smtp-relay:/opt/mailcow-dockerized# docker exec -it 1a0fb7c1e98e /bin/bash
OCI runtime exec failed: exec failed: unable to start container process: exec: “/bin/bash”: stat /bin/bash: no such file or directory: unknown
root@smtp-relay:/opt/mailcow-dockerized# docker exec -it 1a0fb7c1e98e /bin/sh
/ $ dig +trace fuzzy.mailcow.email
/bin/sh: dig: not found
/ $
`

But oddly, ‘nslookup’ was there and worked, by way of busybox:

`/ $ nslookup
BusyBox v1.37.0 (2025-01-17 18:12:01 UTC) multi-call binary.

Usage: nslookup [-type=QUERY_TYPE] [-debug] HOST [DNS_SERVER]

Query DNS about HOST

QUERY_TYPE: soa,ns,a,aaaa,cname,mx,txt,ptr,srv,any
/ $ nslookup fuzzy.mailcow.email
Server: 127.0.0.11
Address: 127.0.0.11:53

Non-authoritative answer:

Non-authoritative answer:
Name: fuzzy.mailcow.email
Address: 95.217.129.125

/ $
`

DocFraggle

docker compose exec unbound-mailcow /bin/bash

jonathanbrickman0000

Worked. Have no idea what I did wrong, but obviously the name is best. Here’s the result of the dig command. IPv6 is disabled on this network:

`; <<>> DiG 9.18.34 <<>> +trace fuzzy.mailcow.email
;; global options: +cmd
. 24051 IN NS d.root-servers.net.
. 24051 IN NS f.root-servers.net.
. 24051 IN NS m.root-servers.net.
. 24051 IN NS k.root-servers.net.
. 24051 IN NS b.root-servers.net.
. 24051 IN NS j.root-servers.net.
. 24051 IN NS c.root-servers.net.
. 24051 IN NS e.root-servers.net.
. 24051 IN NS g.root-servers.net.
. 24051 IN NS i.root-servers.net.
. 24051 IN NS h.root-servers.net.
. 24051 IN NS a.root-servers.net.
. 24051 IN NS l.root-servers.net.
. 24051 IN RRSIG NS 8 0 518400 20250402050000 20250320040000 26470 . kXWTkZI01/tlOZMZEysFJT2ulAYYmjegEcosHcvlWvMcoT4uXvnLY/GS kl/QZuGSmTYgiyrlNl1YZOlyfpbCl8rGCPkLe/wnPlmVhxMs6L4wLenK LSatVCuPlG2r+8KfKfjdk1E/8pYxwR0PRInE+PVxwZMjlSpFbtW7YlTF 4fzvgA3YROHCSw1OFwOcSK+o6/fF2JIYHFkBLW8R89/eXM5AId2QKrbu 5×63hczMF5osQcfroJVbH2zC8dJ33thWYJKrsTBUS5rWS2sISRvHmS0Z eOhyX1XygV+CgOuwH5iJGQcu8c5U+CO/YCNIh1d5HKv/RU2RvToSmLnQ bVLzrA==
;; Received 525 bytes from 127.0.0.11#53(127.0.0.11) in 26 ms

email. 172800 IN NS v0n0.nic.email.
email. 172800 IN NS v0n1.nic.email.
email. 172800 IN NS v0n2.nic.email.
email. 172800 IN NS v0n3.nic.email.
email. 172800 IN NS v2n0.nic.email.
email. 172800 IN NS v2n1.nic.email.
email. 86400 IN DS 62422 8 2 92A9DA686674B9BC30EE72224130A7C761D5B01902DA9A5A62038C79 6A16BD33
email. 86400 IN RRSIG DS 8 1 86400 20250402050000 20250320040000 26470 . nQtLnc7Ex8LGIkD7wAq+59LCPNqq0uJI13RwOOvAKz2IAy365jIVPMnK FhOjcrpWbgK8GJfuu2AeVNgwZKSb9b9ACQYSXJc16de1voQR8Dxo+g8G aYfCMGeVnKlHaqMm6620qmCaIsDE5wwxC0oPBKZU6FYhYN5+tb4GL5uj PzyQA341gqgGc55uFmlV1t37aSDIvKXI5wJ+q8X48mAfehepA+P8F4Rh qfEpzbQtPpESGtIYugdWF+HvCHW4yI+E5GxQsfaNhwZ+kucmZjoBYY3O z++rYvfqTShTYxPqK9nHoGYUVp42JGWy65q2h6H3vgUDkZ+qpQdaYre2 +6fD7Q==
;; Received 765 bytes from 199.7.91.13#53(d.root-servers.net) in 51 ms

mailcow.email. 3600 IN NS angus.ns.servercow.de.
mailcow.email. 3600 IN NS dexter.ns.servercow.net.
mailcow.email. 3600 IN NS galloway.ns.servercow.com.
mailcow.email. 3600 IN DS 41879 13 2 7427B02EF2AB136E84740A14D3D2F7CC84AB821D55AB0EFEFCF4B950 4DC55FC6
mailcow.email. 3600 IN RRSIG DS 8 2 3600 20250407153830 20250317143830 20013 email. c3kDPhwa1HiSxLHfurHSXytq/f5BNNf5Q0LSkagrwoeZuJu3kEfXoXso MEwN6cxpvakMmg+F0NF/rQEDVle3UuPXbbrTmOEwpmG3T6IsmrBB0V2Z Z3fs0N8cYHnkiBayJ/GbszaqBYeNM7FOhdt0Ag0tR43mUzTGN+MKpMi5 k/s=
;; Received 372 bytes from 161.232.12.35#53(v0n3.nic.email) in 51 ms

;; communications error to 2a00:f820:417:153::#53: timed out
;; communications error to 2a00:f820:417:153::#53: timed out
;; communications error to 2a00:f820:417:153::#53: timed out
fuzzy.mailcow.email. 64800 IN RRSIG A 13 3 64800 20250403000000 20250313000000 41879 mailcow.email. I3eyVoeP79Pm/2420jt4omdL5iFelOTZIwEE/C5Rc803YQzzJcNkGJEe bss6w4WFTrIHFb5Ey5nAMYJHCMexcw==
fuzzy.mailcow.email. 64800 IN A 95.217.129.125
;; Received 173 bytes from 45.85.48.12#53(dexter.ns.servercow.net) in 128 ms
`

And just in case, I restarted unbound-mailcow and checked logs again. Dig seems to get everything, but unbound is failing? No forwarding!!!

unbound-mailcow-1 | 2025-03-20 12:18:37,124 INFO success: syslog-ng entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) unbound-mailcow-1 | 2025-03-20 12:18:37,124 INFO success: unbound entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) unbound-mailcow-1 | 2025-03-20 12:18:37,124 INFO success: unbound-healthcheck entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) unbound-mailcow-1 | [1742491120] unbound[18:0] info: generate keytag query _ta-4f66. NULL IN unbound-mailcow-1 | 2025-03-20 12:18:44: Healthcheck: DNS Resolution Failed on attempt 1 for fuzzy.mailcow.email! Trying again... unbound-mailcow-1 | 2025-03-20 12:18:48: Healthcheck: DNS Resolution Failed on attempt 1 for hub.docker.com! Trying again...

And just in case, I restarted unbound-mailcow and checked logs again. Dig seems to get everything, but unbound is failing? No forwarding!!!

DocFraggle

Please run these commands inside the container:

dig +short +timeout=2 +tries=1 "fuzzy.mailcow.email" @127.0.0.1
echo $?

and paste the output

jonathanbrickman0000

Here it is:

root@smtp-relay:/opt/mailcow-dockerized/data/conf/unbound# docker compose exec unbound-mailcow /bin/bash
3d3321b35b0a:/# dig +short +timeout=2 +tries=1 “fuzzy.mailcow.email” @127.0.0.1
echo $?
95.217.129.125
0
3d3321b35b0a:/#

DocFraggle

Strange… that’s exactly the command the unbound container is running for it’s healthcheck… so if you restart your mailcow with

docker compose down
docker compose up -d

it still doesn’t work?

jonathanbrickman0000

That is correct. And indeed, very strange result. I have been wondering about vagaries in Ubuntu 24 built-in DNS, but then I saw that 127.0.0.11 is what the docker container defaults to, which I would think would work around that.