Issue with `watchdog-mailcow` whereas `unbound-mailcow` is healthy

emericv

Hello,

I have an issue with watchdog-mailcow that says my unbound-mailcow is unhealthy whereas docker ps command says that this container is healthy…

There is a way to debug this issue ?

Thanks 🙂

Technical info:

docker-compose logs -f watchdog-mailcow | grep Unbound outputs :

watchdog-mailcow-1  | Sun May 19 17:28:46 CEST 2024 Unbound health level: 80% (4/5), health trend: -1
watchdog-mailcow-1  | Sun May 19 17:29:50 CEST 2024 Unbound health level: 60% (3/5), health trend: -1
watchdog-mailcow-1  | Sun May 19 17:30:48 CEST 2024 Unbound health level: 40% (2/5), health trend: -1
watchdog-mailcow-1  | Sun May 19 17:31:17 CEST 2024 Unbound health level: 20% (1/5), health trend: -1
watchdog-mailcow-1  | Sun May 19 17:32:01 CEST 2024 Unbound health level: 0% (0/5), health trend: -1
watchdog-mailcow-1  | Sun May 19 17:32:02 CEST 2024 Unbound hit error limit
watchdog-mailcow-1  | Sun May 19 17:32:12 CEST 2024 Unbound health level: 80% (4/5), health trend: -1
watchdog-mailcow-1  | Sun May 19 17:33:28 CEST 2024 Unbound health level: 60% (3/5), health trend: -1
watchdog-mailcow-1  | Sun May 19 17:34:20 CEST 2024 Unbound health level: 40% (2/5), health trend: -1
watchdog-mailcow-1  | Sun May 19 17:34:50 CEST 2024 Unbound health level: 20% (1/5), health trend: -1
watchdog-mailcow-1  | Sun May 19 17:35:37 CEST 2024 Unbound health level: 0% (0/5), health trend: -1
watchdog-mailcow-1  | Sun May 19 17:35:38 CEST 2024 Unbound hit error limit

docker-compose ps unbound-mailcow ouputs:

NAME                        IMAGE                  COMMAND                  SERVICE           CREATED          STATUS                   PORTS
mailcow-unbound-mailcow-1   mailcow/unbound:1.21   "/docker-entrypoint.…"   unbound-mailcow   15 minutes ago   Up 6 minutes (healthy)   53/tcp, 53/udp

docker-compose exec unbound-mailcow bash -ex /healthcheck.sh outputs:

+ [[ n =~ ^([yY][eE][sS]|[yY])+$ ]]
+ [[ n == \y ]]
+ check_ping
+ ipstoping=('1.1.1.1' '8.8.8.8' '9.9.9.9')
+ declare -a ipstoping
+ for ip in "${ipstoping[@]}"
+ ping -q -c 3 -w 5 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes

--- 1.1.1.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.996/1.011/1.024 ms
+ '[' 0 -ne 0 ']'
+ for ip in "${ipstoping[@]}"
+ ping -q -c 3 -w 5 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 1.482/1.525/1.599 ms
+ '[' 0 -ne 0 ']'
+ for ip in "${ipstoping[@]}"
+ ping -q -c 3 -w 5 9.9.9.9
PING 9.9.9.9 (9.9.9.9): 56 data bytes

--- 9.9.9.9 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 1.153/1.160/1.171 ms
+ '[' 0 -ne 0 ']'
+ log_to_file 'Healthcheck: Ping Checks WORKING properly!'
++ date '+%Y-%m-%d %H:%M:%S'
+ echo '2024-05-19 17:45:41: Healthcheck: Ping Checks WORKING properly!'
+ return 0
+ '[' 0 -ne 0 ']'
+ check_dns
+ domains=('mailcow.email' 'github.com' 'hub.docker.com')
+ declare -a domains
+ for domain in "${domains[@]}"
+ (( i=1 ))
+ (( i<=3 ))
+ dig +short +timeout=2 +tries=1 mailcow.email @127.0.0.1
+ '[' 0 -ne 0 ']'
+ (( i++ ))
+ (( i<=3 ))
+ dig +short +timeout=2 +tries=1 mailcow.email @127.0.0.1
+ '[' 0 -ne 0 ']'
+ (( i++ ))
+ (( i<=3 ))
+ dig +short +timeout=2 +tries=1 mailcow.email @127.0.0.1
+ '[' 0 -ne 0 ']'
+ (( i++ ))
+ (( i<=3 ))
+ for domain in "${domains[@]}"
+ (( i=1 ))
+ (( i<=3 ))
+ dig +short +timeout=2 +tries=1 github.com @127.0.0.1
+ '[' 0 -ne 0 ']'
+ (( i++ ))
+ (( i<=3 ))
+ dig +short +timeout=2 +tries=1 github.com @127.0.0.1
+ '[' 0 -ne 0 ']'
+ (( i++ ))
+ (( i<=3 ))
+ dig +short +timeout=2 +tries=1 github.com @127.0.0.1
+ '[' 0 -ne 0 ']'
+ (( i++ ))
+ (( i<=3 ))
+ for domain in "${domains[@]}"
+ (( i=1 ))
+ (( i<=3 ))
+ dig +short +timeout=2 +tries=1 hub.docker.com @127.0.0.1
+ '[' 0 -ne 0 ']'
+ (( i++ ))
+ (( i<=3 ))
+ dig +short +timeout=2 +tries=1 hub.docker.com @127.0.0.1
+ '[' 0 -ne 0 ']'
+ (( i++ ))
+ (( i<=3 ))
+ dig +short +timeout=2 +tries=1 hub.docker.com @127.0.0.1
+ '[' 0 -ne 0 ']'
+ (( i++ ))
+ (( i<=3 ))
+ log_to_file 'Healthcheck: DNS Resolver WORKING properly!'
++ date '+%Y-%m-%d %H:%M:%S'
+ echo '2024-05-19 17:45:42: Healthcheck: DNS Resolver WORKING properly!'
+ return 0
+ '[' 0 -ne 0 ']'
+ log_to_file 'Healthcheck: ALL CHECKS WERE SUCCESSFUL! Unbound is healthy!'
++ date '+%Y-%m-%d %H:%M:%S'
+ echo '2024-05-19 17:45:42: Healthcheck: ALL CHECKS WERE SUCCESSFUL! Unbound is healthy!'
+ exit 0

emericv

Ok in fact the check is performed by the unbound_checks form the mailcow/mailcow-dockerizedblob/master/data/Dockerfiles/watchdog/watchdog.sh#L286 file.

With debug mode, I observe this output:

++ dig com +dnssec
++ egrep flags:.+ad
+ DNSSEC=
+ [[ -z '' ]]
+ echo 'DNSSEC failure'
+ err_count=1

when I run dig com +dnssec I have:

; <<>> DiG 9.18.19 <<>> com +dnssec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55445
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;com.                           IN      A

;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11) (UDP)
;; WHEN: Sun May 19 18:28:57 CEST 2024
;; MSG SIZE  rcvd: 21

Hmmm, a dig @172.22.1.254 com +dnssec give:

;; AUTHORITY SECTION:
com.                    60      IN      SOA     a.gtld-servers.net. nstld.verisign-grs.com. 1716137409 1800 900 604800 86400
com.                    900     IN      RRSIG   SOA 13 1 900 20240526165009 20240519154009 956 com. y5cjojdcJgH3yaGhSRpvxhf2o4pBPilHpkaLhEMYRdIJpLN525Coo+y5 fW1I9TCh0fqy1EI6QbKjD4USQNzGeQ==
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86084 IN NSEC3 1 1 0 - CK0Q2D6NI4I7EQH8NA30NS61O48UL8G5 NS SOA RRSIG DNSKEY NSEC3PARAM
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86084 IN RRSIG NSEC3 13 2 86400 20240524042459 20240517031459 956 com. z1jR7JNLkGW/vGt4F/eJR0w0JigXU31VwLVexqIEy4SXZ0PQKOWsp8eG k9jjcJ3Ta1dcmyUb1qNcvVWuGcCzUA==

In the docker-compose.yml the dns of watchdog-mailcow is ${IPV4_NETWORK:-172.22.1}.254 (=172.22.1.254) and inside the container, 127.0.0.11 should forward to 172.22.1.254 but I have the impression that not the case…

Hmmm, a dig @172.22.1.254 com +dnssec give:

;; AUTHORITY SECTION:
com.                    60      IN      SOA     a.gtld-servers.net. nstld.verisign-grs.com. 1716137409 1800 900 604800 86400
com.                    900     IN      RRSIG   SOA 13 1 900 20240526165009 20240519154009 956 com. y5cjojdcJgH3yaGhSRpvxhf2o4pBPilHpkaLhEMYRdIJpLN525Coo+y5 fW1I9TCh0fqy1EI6QbKjD4USQNzGeQ==
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86084 IN NSEC3 1 1 0 - CK0Q2D6NI4I7EQH8NA30NS61O48UL8G5 NS SOA RRSIG DNSKEY NSEC3PARAM
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86084 IN RRSIG NSEC3 13 2 86400 20240524042459 20240517031459 956 com. z1jR7JNLkGW/vGt4F/eJR0w0JigXU31VwLVexqIEy4SXZ0PQKOWsp8eG k9jjcJ3Ta1dcmyUb1qNcvVWuGcCzUA==

Hmmm, a dig @172.22.1.254 com +dnssec give:

;; AUTHORITY SECTION:
com.                    60      IN      SOA     a.gtld-servers.net. nstld.verisign-grs.com. 1716137409 1800 900 604800 86400
com.                    900     IN      RRSIG   SOA 13 1 900 20240526165009 20240519154009 956 com. y5cjojdcJgH3yaGhSRpvxhf2o4pBPilHpkaLhEMYRdIJpLN525Coo+y5 fW1I9TCh0fqy1EI6QbKjD4USQNzGeQ==
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86084 IN NSEC3 1 1 0 - CK0Q2D6NI4I7EQH8NA30NS61O48UL8G5 NS SOA RRSIG DNSKEY NSEC3PARAM
CK0POJMG874LJREF7EFN8430QVIT8BSM.com. 86084 IN RRSIG NSEC3 13 2 86400 20240524042459 20240517031459 956 com. z1jR7JNLkGW/vGt4F/eJR0w0JigXU31VwLVexqIEy4SXZ0PQKOWsp8eG k9jjcJ3Ta1dcmyUb1qNcvVWuGcCzUA==

ooops… I have posted 3 time the same message, but there is no way to edit/remove it… :/