Waiting for database...

Heaps of CPU and RAM, resources are no issue.

As I say, it runs fine usually. Things work as per normal. As soon as I docker-compose down / up, or run the update.sh script (anything that stops, removes, recreates containers) they all get created and start up again but get “waiting on database” error.

If it’s a network issue, what steps do I need to run through to check why its waiting on the database? And why is it running fine now but as soon as containers are recreated it fails? (unless its coz containers are getting a new IP address, different to the current, then cant connect? where are the settings that tell each container where the database is or what it resolves too? Its been a while I cant recall what I may have changed originally to get this working!)

Ubuntu 22.04.3 LTS

root@mail:/var/mailcow-dockerized# docker -v
Docker version 25.0.3, build 4debf41

Pre-upgrade, test from acme… (although its not upgrade specific, its any process that remove and recreates containers that causes issues)

741e306f2c19:/# ping mailcowdockerized-mysql-mailcow-1
PING mailcowdockerized-mysql-mailcow-1 (172.22.1.10): 56 data bytes
64 bytes from 172.22.1.10: seq=0 ttl=64 time=0.365 ms
64 bytes from 172.22.1.10: seq=1 ttl=64 time=0.127 ms
64 bytes from 172.22.1.10: seq=2 ttl=64 time=0.145 ms
64 bytes from 172.22.1.10: seq=3 ttl=64 time=0.127 ms
nc64 bytes from 172.22.1.10: seq=4 ttl=64 time=0.132 ms
^C
--- mailcowdockerized-mysql-mailcow-1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.127/0.179/0.365 ms
741e306f2c19:/# 
741e306f2c19:/# 
741e306f2c19:/# nc mailcowdockerized-mysql-mailcow-1 3306
a
5.5.5-10.5.25-MariaDB-ubu2004
.^n1DB`>Sh0U>rGK"dTWmysql_native_password

^Cpunt!

Post upgrade (only change is different IP address)

f67a352b8136:/# ping mailcowdockerized-mysql-mailcow-1
PING mailcowdockerized-mysql-mailcow-1 (172.22.1.6): 56 data bytes
64 bytes from 172.22.1.6: seq=0 ttl=64 time=0.323 ms
64 bytes from 172.22.1.6: seq=1 ttl=64 time=0.162 ms
64 bytes from 172.22.1.6: seq=2 ttl=64 time=0.124 ms
64 bytes from 172.22.1.6: seq=3 ttl=64 time=0.129 ms
^C
--- mailcowdockerized-mysql-mailcow-1 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.124/0.184/0.323 ms
f67a352b8136:/# 
f67a352b8136:/# 
f67a352b8136:/# nc mailcowdockerized-mysql-mailcow-1 3306
a
5.5.5-10.5.26-MariaDB-ubu2004(t5HL#fZrQy1$a>PXJKwmysql_native_password

^Cpunt!

f67a352b8136:/# ^C

acme logs

Wed Sep 18 21:16:31 NZST 2024 - Waiting for Docker API...
Wed Sep 18 21:16:31 NZST 2024 - Docker API OK
Wed Sep 18 21:16:31 NZST 2024 - Waiting for Postfix...
Wed Sep 18 21:16:32 NZST 2024 - Postfix OK
Wed Sep 18 21:16:32 NZST 2024 - Waiting for Dovecot...
Wed Sep 18 21:16:32 NZST 2024 - Dovecot OK
Wed Sep 18 21:16:32 NZST 2024 - Waiting for database...

(stops here and doesnt proceed)

postfix logs

Waiting for database to come up...
Waiting for database to come up...
Waiting for database to come up...
Waiting for database to come up...
Waiting for database to come up...
Waiting for database to come up...

You know the irony… is I found that article when troubleshooting earlier today and actually must have stuffed up by pasting the wrong password in when I reset them… so thought id ruled that side out and ended up down a rabbit hole…

It was only after posting additional logs I google searched the “waiting for database to come up” and stumbled back on the same article. it was only when testing with these that I realised I’d fucked up

root@mail:/var/mailcow-dockerized# source mailcow.conf
root@mail:/var/mailcow-dockerized# docker compose exec mysql-mailcow mysql -u${DBUSER} -p${DBPASS} ${DBNAME}
ERROR 1045 (28000): Access denied for user 'mailcow'@'localhost' (using password: YES)

Cheers from NZ - enjoy your time when you’re down here!!! :-) PM me if you want any tips for places to go!