Waiting for database...

chimeranzl

Anytime I do anything vaguely different (or even restart mailcow) I get this sort of log entries (used to be waiting for sql prior to upgrading to 2024-08a patch)

Wed Sep 18 12:15:39 NZST 2024 - Waiting for Docker API...
Wed Sep 18 12:15:39 NZST 2024 - Docker API OK
Wed Sep 18 12:15:39 NZST 2024 - Waiting for Postfix...
Wed Sep 18 12:15:39 NZST 2024 - Postfix OK
Wed Sep 18 12:15:39 NZST 2024 - Waiting for Dovecot...
Wed Sep 18 12:15:39 NZST 2024 - Dovecot OK
Wed Sep 18 12:15:39 NZST 2024 - Waiting for database...

I have to revert my proxmox snapshot in order to connect to the GUI again

How do I resolve this? How do mailcow conntainers connect to SQL? I can ssh into containers (eg: acme) and ping the container ID of mariadb:10.5 and it resolves to the correct IP fine. I can also ping the container name “mailcowdockerized-mysql-mailcow-1” fine also.

The mariadb logs are as follows, this appears to be ok?

2024-09-18 12:09:13+12:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.5.26+maria~ubu2004 started.
2024-09-18 12:09:13+12:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2024-09-18 12:09:13+12:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.5.26+maria~ubu2004 started.
2024-09-18 12:09:14+12:00 [Note] [Entrypoint]: MariaDB upgrade not required
2024-09-18 12:09:14 0 [Note] Starting MariaDB 10.5.26-MariaDB-ubu2004 source revision 7a5b8bf0f5470a13094101f0a4bdfa9e1b9ded02 server_uid htq1MTURYLUrMuU4LgOH1bA5G2A= as process 1
2024-09-18 12:09:14 0 [Note] InnoDB: Uses event mutexes
2024-09-18 12:09:14 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2024-09-18 12:09:14 0 [Note] InnoDB: Number of pools: 1
2024-09-18 12:09:14 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2024-09-18 12:09:14 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
2024-09-18 12:09:14 0 [Note] InnoDB: Using Linux native AIO
2024-09-18 12:09:14 0 [Note] InnoDB: Initializing buffer pool, total size = 25165824, chunk size = 25165824
2024-09-18 12:09:14 0 [Note] InnoDB: Completed initialization of buffer pool
2024-09-18 12:09:14 0 [Note] InnoDB: 128 rollback segments are active.
2024-09-18 12:09:14 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2024-09-18 12:09:14 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2024-09-18 12:09:14 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2024-09-18 12:09:14 0 [Note] InnoDB: 10.5.26 started; log sequence number 122582461; transaction id 694813
2024-09-18 12:09:14 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2024-09-18 12:09:14 0 [Note] Server socket created on IP: '::'.
2024-09-18 12:09:14 0 [Note] mysqld: ready for connections.
Version: '10.5.26-MariaDB-ubu2004'  socket: '/run/mysqld/mysqld.sock'  port: 3306  mariadb.org binary distribution
2024-09-18 12:09:14 1 [Note] Event Scheduler: scheduler thread started with id 1
2024-09-18 12:09:14 0 [Note] InnoDB: Buffer pool(s) load completed at 240918 12:09:14

Help?!

esackbauer

Tell us more about your setup. How much RAM? Which OS? Using LXC or a VM?
Looks like you run into some docker network troubles. Are you on latest docker and docker compose versions?

chimeranzl

Heaps of CPU and RAM, resources are no issue.

As I say, it runs fine usually. Things work as per normal. As soon as I docker-compose down / up, or run the update.sh script (anything that stops, removes, recreates containers) they all get created and start up again but get “waiting on database” error.

If it’s a network issue, what steps do I need to run through to check why its waiting on the database? And why is it running fine now but as soon as containers are recreated it fails? (unless its coz containers are getting a new IP address, different to the current, then cant connect? where are the settings that tell each container where the database is or what it resolves too? Its been a while I cant recall what I may have changed originally to get this working!)

esackbauer

You have not answered all my questions.
Please make sure that you are at least on Docker 24.x, because that is the minimum supported version for 2024-08

Can you telnet or nc from the other containers to the port of mariadb?

chimeranzl

Ubuntu 22.04.3 LTS

root@mail:/var/mailcow-dockerized# docker -v
Docker version 25.0.3, build 4debf41

Pre-upgrade, test from acme… (although its not upgrade specific, its any process that remove and recreates containers that causes issues)

741e306f2c19:/# ping mailcowdockerized-mysql-mailcow-1
PING mailcowdockerized-mysql-mailcow-1 (172.22.1.10): 56 data bytes
64 bytes from 172.22.1.10: seq=0 ttl=64 time=0.365 ms
64 bytes from 172.22.1.10: seq=1 ttl=64 time=0.127 ms
64 bytes from 172.22.1.10: seq=2 ttl=64 time=0.145 ms
64 bytes from 172.22.1.10: seq=3 ttl=64 time=0.127 ms
nc64 bytes from 172.22.1.10: seq=4 ttl=64 time=0.132 ms
^C
--- mailcowdockerized-mysql-mailcow-1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.127/0.179/0.365 ms
741e306f2c19:/# 
741e306f2c19:/# 
741e306f2c19:/# nc mailcowdockerized-mysql-mailcow-1 3306
a
5.5.5-10.5.25-MariaDB-ubu2004
.^n1DB`>Sh0U>rGK"dTWmysql_native_password

^Cpunt!

Post upgrade (only change is different IP address)

f67a352b8136:/# ping mailcowdockerized-mysql-mailcow-1
PING mailcowdockerized-mysql-mailcow-1 (172.22.1.6): 56 data bytes
64 bytes from 172.22.1.6: seq=0 ttl=64 time=0.323 ms
64 bytes from 172.22.1.6: seq=1 ttl=64 time=0.162 ms
64 bytes from 172.22.1.6: seq=2 ttl=64 time=0.124 ms
64 bytes from 172.22.1.6: seq=3 ttl=64 time=0.129 ms
^C
--- mailcowdockerized-mysql-mailcow-1 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 0.124/0.184/0.323 ms
f67a352b8136:/# 
f67a352b8136:/# 
f67a352b8136:/# nc mailcowdockerized-mysql-mailcow-1 3306
a
5.5.5-10.5.26-MariaDB-ubu2004(t5HL#fZrQy1$a>PXJKwmysql_native_password

^Cpunt!

f67a352b8136:/# ^C

acme logs

Wed Sep 18 21:16:31 NZST 2024 - Waiting for Docker API...
Wed Sep 18 21:16:31 NZST 2024 - Docker API OK
Wed Sep 18 21:16:31 NZST 2024 - Waiting for Postfix...
Wed Sep 18 21:16:32 NZST 2024 - Postfix OK
Wed Sep 18 21:16:32 NZST 2024 - Waiting for Dovecot...
Wed Sep 18 21:16:32 NZST 2024 - Dovecot OK
Wed Sep 18 21:16:32 NZST 2024 - Waiting for database...

(stops here and doesnt proceed)

postfix logs

Waiting for database to come up...
Waiting for database to come up...
Waiting for database to come up...
Waiting for database to come up...
Waiting for database to come up...
Waiting for database to come up...

esackbauer

https://community.mailcow.email/d/126-waiting-for-database-to-come-up

chimeranzl

Thanks, found the same article about 30 mins ago.
Resetting SQL passwords sorted it, thanks!!

esackbauer

I cannot count how many times I reposted something that could be found by searching the forum or docs 😉
Greetings to NZ, looking forward to visit again in March next year.

chimeranzl

You know the irony… is I found that article when troubleshooting earlier today and actually must have stuffed up by pasting the wrong password in when I reset them… so thought id ruled that side out and ended up down a rabbit hole…

It was only after posting additional logs I google searched the “waiting for database to come up” and stumbled back on the same article. it was only when testing with these that I realised I’d fucked up

root@mail:/var/mailcow-dockerized# source mailcow.conf
root@mail:/var/mailcow-dockerized# docker compose exec mysql-mailcow mysql -u${DBUSER} -p${DBPASS} ${DBNAME}
ERROR 1045 (28000): Access denied for user 'mailcow'@'localhost' (using password: YES)

Cheers from NZ - enjoy your time when you’re down here!!! :-) PM me if you want any tips for places to go!

chimeranzl

fyi, its possible this also helped https://community.mailcow.email/d/3159-issues-after-updating/19