Another unbound not working post

Safeguard2711

Hey
I’ve just updated last night to 2025-09b and unbound became unhealthy

I ended up disabling health checks in the config and it was working again however then gmail gets errors back when trying to send email to my mailcow server

From Mail Delivery Subsystem mailer-daemon@googlemail.com

The response from the remote server was:
450 4.1.8 : Sender address rejected: Domain not found

I’ve been searching for info on unbound being unhealthy and this seems to be a common issue going back years with very little engagement

This has been the most useful post I can find so far
https://community.mailcow.email/d/2977-container-unbound-unhealthy/39

Following troubleshooting from that post

If i jump into the unbound docker container
sudo docker compose exec unbound-mailcow /bin/bash

nslookup fuzzy.mailcow.email succeeds
nslookup fuzzy.mailcow.email 127.0.0.1 fails
nslookup fuzzy.mailcow.email 127.0.0.11 succeeds

One of the later replies on that post said to add the docker network to the unbound.conf and mine was not listed in the config, so I did that but it didn’t help
access-control: 172.17.0.0/16 allow
I’ve since noticed mailcow runs its own docker network too which was also not in the allow list
access-control: 172.22.0.0/24 allow
I’ve added this too but again did not help

One of the other replies said they swapped out the unbound container to a different one
image: mvance/unbound:latest

I tried that and the contents of the container are quite different, I can’t find the healthcheck.sh file for instance but once i swapped to this everything is working and google can send email to my mailserver again

Within that new container, after running
apt install dnsutils

nslookup fuzzy.mailcow.email succeeds
nslookup fuzzy.mailcow.email 127.0.0.1 succeeds
nslookup fuzzy.mailcow.email 127.0.0.11 succeeds

Ok so this is great, this seems to prove there is something wrong with the bundled unbound containers config as shipped with mailcow but its not ideal for future upgrades

Does anyone know how to get mailcows unbound to work correctly? or perhaps i should raise this as a github issue since the vanilla unbound works correctly

Safeguard2711

Ok this is solved luckily through about a 3 hour grok session

/opt/mailcow-dockerized$ sudo docker logs -f mailcowdockerized-unbound-mailcow-1
Setting console permissions...
Receiving anchor key...
Receiving root hints...
######################################################################## 100.0%
setup in directory /etc/unbound
Certificate request self-signature ok
subject=CN=unbound-control
removing artifacts
Setup success. Certificates created. Enable in unbound.conf file to use
2025-10-02 18:04:35,345 INFO Set uid to user 0 succeeded
2025-10-02 18:04:35,390 INFO supervisord started with pid 1
2025-10-02 18:04:36,420 INFO spawned: 'processes' with pid 23
2025-10-02 18:04:36,496 INFO spawned: 'syslog-ng' with pid 24
2025-10-02 18:04:36,597 INFO spawned: 'unbound' with pid 25
2025-10-02 18:04:36,699 INFO spawned: 'unbound-healthcheck' with pid 26
[1759392277] unbound[25:0] notice: init module 0: subnetcache
[1759392277] unbound[25:0] notice: init module 1: validator
[1759392277] unbound[25:0] notice: init module 2: iterator
[1759392277] unbound[25:0] info: start of service (unbound 1.22.0).
Oct  2 18:04:37 9034913e48fe syslog-ng[24]: syslog-ng starting up; version='4.8.1'
2025-10-02 18:04:37,717 INFO success: processes entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-10-02 18:04:37,720 INFO success: syslog-ng entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-10-02 18:04:37,728 INFO success: unbound entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-10-02 18:04:37,731 INFO success: unbound-healthcheck entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-10-02 18:04:43: Healthcheck: DNS Resolution Failed on attempt 1 for fuzzy.mailcow.email! Trying again...
2025-10-02 18:04:44: Healthcheck: DNS Resolution Failed on attempt 2 for fuzzy.mailcow.email! Trying again...
2025-10-02 18:04:44: Healthcheck: DNS Resolution Failed on attempt 3 for fuzzy.mailcow.email! Trying again...
2025-10-02 18:04:44: Healthcheck: DNS Resolution not possible after 3 attempts for fuzzy.mailcow.email... Gave up!
2025-10-02 18:04:45: Healthcheck: DNS Resolution Failed on attempt 1 for github.com! Trying again...
2025-10-02 18:04:45: Healthcheck: DNS Resolution Failed on attempt 2 for github.com! Trying again...
2025-10-02 18:04:46: Healthcheck: DNS Resolution Failed on attempt 3 for github.com! Trying again...
2025-10-02 18:04:46: Healthcheck: DNS Resolution not possible after 3 attempts for github.com... Gave up!
2025-10-02 18:04:46: Healthcheck: DNS Resolution Failed on attempt 1 for hub.docker.com! Trying again...
2025-10-02 18:04:47: Healthcheck: DNS Resolution Failed on attempt 2 for hub.docker.com! Trying again...
2025-10-02 18:04:48: Healthcheck: DNS Resolution Failed on attempt 3 for hub.docker.com! Trying again...
2025-10-02 18:04:48: Healthcheck: DNS Resolution not possible after 3 attempts for hub.docker.com... Gave up!
2025-10-02 18:04:48: Healthcheck: Too many DNS failures (1 failures allowed, you got 3 failures), marking Healthcheck as unhealthy...

Editing unbound.conf

# logfile: /dev/console
log-servfail: yes
verbosity: 2
  logfile: unbound.log

Gave me these log results

vi /etc/unbound/unbound.log
[1759397324] unbound[25:0] notice: init module 0: subnetcache
[1759397324] unbound[25:0] notice: init module 1: validator
[1759397324] unbound[25:0] notice: init module 2: iterator
[1759397324] unbound[25:0] info: start of service (unbound 1.22.0).
[1759397324] unbound[25:0] info: resolving . DNSKEY IN
[1759397324] unbound[25:0] error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed
[1759397324] unbound[25:0] error: ssl handshake cert error: self-signed certificate in certificate chain
[1759397324] unbound[25:0] notice: ssl handshake failed 1.1.1.1 port 853
[1759397324] unbound[25:0] error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed
[1759397324] unbound[25:0] error: ssl handshake cert error: self-signed certificate in certificate chain
[1759397324] unbound[25:0] notice: ssl handshake failed 1.1.1.1 port 853
[1759397324] unbound[25:0] error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed
[1759397324] unbound[25:0] error: ssl handshake cert error: self-signed certificate in certificate chain
[1759397324] unbound[25:0] notice: ssl handshake failed 1.0.0.1 port 853
[1759397324] unbound[25:0] error: ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed
[1759397324] unbound[25:0] error: ssl handshake cert error: self-signed certificate in certificate chain
[1759397324] unbound[25:0] notice: ssl handshake failed 1.0.0.1 port 853

Updating the config to

server:
    harden-dnssec-stripped: no
    harden-referral-path: no
    do-not-query-localhost: no
forward-zone:
    name: "."
    forward-tls-upstream: no
    forward-addr: 1.1.1.1
    forward-addr: 8.8.8.8

Got the calls to start succeeding again

Now to reenable tls if possible
On the host

Create docker-compose.override.yml

services:
  unbound-mailcow:
    volumes:
      - /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt:ro

Change the unbound.conf to

server:
    harden-dnssec-stripped: no
    harden-referral-path: no
    do-not-query-localhost: no
    tls-cert-bundle: "/etc/ssl/certs/ca-certificates.crt"
forward-zone:
    name: "."
    forward-tls-upstream: yes
    forward-addr: 1.1.1.1@853#cloudflare-dns.com
    forward-addr: 1.0.0.1@853#cloudflare-dns.com

Update the root.hints and trusted-key.key files

sudo docker exec -it mailcowdockerized-unbound-mailcow-1 bash
wget -O /etc/unbound/root.hints https://www.internic.net/domain/named.root
unbound-anchor -a /etc/unbound/trusted-key.key
chmod 644 /etc/unbound/root.hints /etc/unbound/trusted-key.key
chown unbound:unbound /etc/unbound/root.hints /etc/unbound/trusted-key.key

Restart everything and its back up and running, healthchecks are succeeding again

I’m not able to run dig @1.1.1.1 -p 853 but everything else is working

Based on the logs showed crashes with “ssl handshake failed crypto error:0A000086:SSL routines::certificate verify failed” when using DNS-over-TLS (DoT) to Cloudflare (1.1.1.1:853, 1.0.0.1:853).

Unbound’s issues seem to be caused by a missing or inaccessible CA certificate possibly combined with bad config
I was able to resolve it in 3 different ways

Using an alternate unbound container without the same config
Turning off tls in the existing config
Copying the hosts cert file into the container and refreshing the root.hints and trusted-key.key files and massaging the tls config a fair bit

I didn’t mention it before but my mailcow server is running in a vm hosted on proxmox
The vm is ubuntu server 24.04
The install was vanilla following the installation instructions and its been running for about 8 months i think

Not sure how to mark this as solved now