• THW1

      Moolevel 0
    • Post #1
    • Edited

    Hi all,

    I have upgraded to 2025-03 and are now facing the problem, that the mariadb container is not starting properly [1]. Subsequentially, th sogo container is also restarting frequently as it is “Waiting for schema update...”

    Maybe somebody has encountered a similar issue and has a suggestion on how to fix the DB?

    Cheers and thanks for all suggestions,
    Thomas

    [1]
    mysql-mailcow-1 | 2025-04-03 17:11:06 0 [ERROR] mariadbd: Got error ‘Could not get an exclusive lock; file is probably in use by another process’ when trying to use aria control file ‘/var/lib/mysql/aria_log_control’
    mysql-mailcow-1 | 2025-04-03 17:11:06 0 [ERROR] Plugin ‘Aria’ registration as a STORAGE ENGINE failed.
    mysql-mailcow-1 | 2025-04-03 17:11:06 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
    mysql-mailcow-1 | 2025-04-03 17:11:06 0 [Note] InnoDB: Number of transaction pools: 1
    mysql-mailcow-1 | 2025-04-03 17:11:06 0 [Note] InnoDB: Using ARMv8 crc32 + pmull instructions
    mysql-mailcow-1 | 2025-04-03 17:11:06 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
    mysql-mailcow-1 | 2025-04-03 17:11:06 0 [Note] InnoDB: Initializing buffer pool, total size = 24.000MiB, chunk size = 1.000MiB
    mysql-mailcow-1 | 2025-04-03 17:11:06 0 [Note] InnoDB: Completed initialization of buffer pool
    mysql-mailcow-1 | 2025-04-03 17:11:06 0 [Note] InnoDB: File system buffers for log disabled (block size=4096 bytes)
    mysql-mailcow-1 | 2025-04-03 17:11:06 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.5.27. You must start up and shut down MariaDB 10.7 or earlier.
    mysql-mailcow-1 | 2025-04-03 17:11:06 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
    mysql-mailcow-1 | 2025-04-03 17:11:06 0 [Note] InnoDB: Starting shutdown…
    mysql-mailcow-1 | 2025-04-03 17:11:06 0 [ERROR] Plugin ‘InnoDB’ registration as a STORAGE ENGINE failed.
    mysql-mailcow-1 | 2025-04-03 17:11:06 0 [ERROR] Could not open mysql.plugin table: “Unknown storage engine ‘Aria’”. Some plugins may be not loaded
    mysql-mailcow-1 | 2025-04-03 17:11:06 0 [ERROR] Failed to initialize plugins.
    mysql-mailcow-1 | 2025-04-03 17:11:06 0 [ERROR] Aborting
    mysql-mailcow-1 | 2025-04-03 17:11:08+02:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.11+mariaubu2204 started.
    mysql-mailcow-1 | 2025-04-03 17:11:08+02:00 [Warn] [Entrypoint]: /sys/fs/cgroup///memory.pressure not writable, functionality unavailable to MariaDB
    mysql-mailcow-1 | 2025-04-03 17:11:08+02:00 [Note] [Entrypoint]: Switching to dedicated user ‘mysql’
    mysql-mailcow-1 | 2025-04-03 17:11:08+02:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.11+mariaubu2204 started.
    mysql-mailcow-1 | 2025-04-03 17:11:09+02:00 [Note] [Entrypoint]: MariaDB upgrade (mariadb-upgrade or creating healthcheck users) required, but skipped due to $MARIADB_AUTO_UPGRADE setting
    mysql-mailcow-1 | 2025-04-03 17:11:09 0 [Note] Starting MariaDB 10.11.11-MariaDB-ubu2204 source revision e69f8cae1a15e15b9e4f5e0f8497e1f17bdc81a4 server_uid h7KYG0k9MEUPd9OA122fCL4wNdA= as process 1
    mysql-mailcow-1 | 2025-04-03 17:11:09 0 [ERROR] mariadbd: Can’t lock aria control file ‘/var/lib/mysql/aria_log_control’ for exclusive use, error: 11. Will retry for 30 seconds
    mysql-mailcow-1 | 2025-04-03 17:11:39 0 [ERROR] mariadbd: Got error ‘Could not get an exclusive lock; file is probably in use by another process’ when trying to use aria control file ‘/var/lib/mysql/aria_log_control’
    mysql-mailcow-1 | 2025-04-03 17:11:39 0 [ERROR] Plugin ‘Aria’ registration as a STORAGE ENGINE failed.
    mysql-mailcow-1 | 2025-04-03 17:11:39 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
    mysql-mailcow-1 | 2025-04-03 17:11:39 0 [Note] InnoDB: Number of transaction pools: 1
    mysql-mailcow-1 | 2025-04-03 17:11:39 0 [Note] InnoDB: Using ARMv8 crc32 + pmull instructions
    mysql-mailcow-1 | 2025-04-03 17:11:39 0 [Note] mariadbd: O_TMPFILE is not supported on /tmp (disabling future attempts)
    mysql-mailcow-1 | 2025-04-03 17:11:39 0 [Note] InnoDB: Initializing buffer pool, total size = 24.000MiB, chunk size = 1.000MiB
    mysql-mailcow-1 | 2025-04-03 17:11:39 0 [Note] InnoDB: Completed initialization of buffer pool
    mysql-mailcow-1 | 2025-04-03 17:11:39 0 [Note] InnoDB: File system buffers for log disabled (block size=4096 bytes)
    mysql-mailcow-1 | 2025-04-03 17:11:39 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.5.27. You must start up and shut down MariaDB 10.7 or earlier.
    mysql-mailcow-1 | 2025-04-03 17:11:39 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
    mysql-mailcow-1 | 2025-04-03 17:11:39 0 [Note] InnoDB: Starting shutdown…
    mysql-mailcow-1 | 2025-04-03 17:11:39 0 [ERROR] Plugin ‘InnoDB’ registration as a STORAGE ENGINE failed.
    mysql-mailcow-1 | 2025-04-03 17:11:39 0 [ERROR] Could not open mysql.plugin table: “Unknown storage engine ‘Aria’”. Some plugins may be not loaded
    mysql-mailcow-1 | 2025-04-03 17:11:39 0 [ERROR] Failed to initialize plugins.
    mysql-mailcow-1 | 2025-04-03 17:11:39 0 [ERROR] Aborting
    mysql-mailcow-1 | 2025-04-03 17:11:40+02:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.11+mariaubu2204 started.
    mysql-mailcow-1 | 2025-04-03 17:11:41+02:00 [Warn] [Entrypoint]: /sys/fs/cgroup///memory.pressure not writable, functionality unavailable to MariaDB
    mysql-mailcow-1 | 2025-04-03 17:11:41+02:00 [Note] [Entrypoint]: Switching to dedicated user ‘mysql’
    mysql-mailcow-1 | 2025-04-03 17:11:41+02:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.11+mariaubu2204 started.
    mysql-mailcow-1 | 2025-04-03 17:11:41+02:00 [Note] [Entrypoint]: MariaDB upgrade (mariadb-upgrade or creating healthcheck users) required, but skipped due to $MARIADB_AUTO_UPGRADE setting
    mysql-mailcow-1 | 2025-04-03 17:11:41 0 [Note] Starting MariaDB 10.11.11-MariaDB-ubu2204 source revision e69f8cae1a15e15b9e4f5e0f8497e1f17bdc81a4 server_uid GdLkEL5lwb8lB3sQXUkjGqckhq4= as process 1
    mysql-mailcow-1 | 2025-04-03 17:11:41 0 [ERROR] mariadbd: Can’t lock aria control file ‘/var/lib/mysql/aria_log_control’ for exclusive use, error: 11. Will retry for 30 seconds

    looks like the db version upgrade did not got applied 🤔

    # mysql_upgrade -p
Enter password: 
Major version upgrade detected from 10.5.27-MariaDB to 10.11.11-MariaDB. Check required!
Error: Server version (10.5.27-MariaDB-ubu2004)
does not match the version of the server (10.11.11-MariaDB)
with which this program was built/distributed. You can
use --skip-version-check to skip this check.
FATAL ERROR: Upgrade failed

    mariadb container is at mariadb:10.11 but it looks like my database got corrupted?!

    # mysql_upgrade -p --skip-version-check
Enter password: 
ERROR 1 (HY000) at line 14: Can't create/write to file '/tmp/#sql-temptable-1-384-3.MAI' (Errcode: 2 "No such file or directory")
ERROR 1 (HY000) at line 15: Can't create/write to file '/tmp/#sql-temptable-1-384-4.MAI' (Errcode: 2 "No such file or directory")
ERROR 1 (HY000) at line 124: Can't create/write to file '/tmp/#sql-temptable-1-384-7.MAI' (Errcode: 2 "No such file or directory")
ERROR 1193 (HY000) at line 184: Unknown system variable 'log_slow_query'
ERROR 1193 (HY000) at line 185: Unknown system variable 'log_slow_query'
ERROR 1193 (HY000) at line 203: Unknown system variable 'log_slow_query'
ERROR 1 (HY000) at line 568: Can't create/write to file '/tmp/#sql-temptable-1-384-14.MAI' (Errcode: 2 "No such file or directory")
ERROR 1 (HY000) at line 586: Can't create/write to file '/tmp/#sql-temptable-1-384-16.MAI' (Errcode: 2 "No such file or directory")
ERROR 1 (HY000) at line 587: Can't create/write to file '/tmp/#sql-temptable-1-384-17.MAI' (Errcode: 2 "No such file or directory")
ERROR 1 (HY000) at line 589: Can't create/write to file '/tmp/#sql-temptable-1-384-18.MAI' (Errcode: 2 "No such file or directory")
ERROR 1 (HY000) at line 648: Can't create/write to file '/tmp/#sql-temptable-1-384-19.MAI' (Errcode: 2 "No such file or directory")
ERROR 1 (HY000) at line 776: Can't create/write to file '/tmp/#sql-temptable-1-384-1b.MAI' (Errcode: 2 "No such file or directory")
ERROR 1 (HY000) at line 807: Can't create/write to file '/tmp/#sql-temptable-1-384-20.MAI' (Errcode: 2 "No such file or directory")
ERROR 1728 (HY000) at line 853: Cannot load from mysql.proc. The table is probably corrupted
ERROR 1728 (HY000) at line 855: Cannot load from mysql.proc. The table is probably corrupted
ERROR 1728 (HY000) at line 888: Cannot load from mysql.proc. The table is probably corrupted
ERROR 1728 (HY000) at line 890: Cannot load from mysql.proc. The table is probably corrupted
ERROR 1728 (HY000) at line 920: Cannot load from mysql.proc. The table is probably corrupted

    is there a good way to recover the DB - or would the way to go to scrap everything and retore it from a backup?

      THW1

      From your log:
      Upgrade after a crash is not supported. The redo log was created with MariaDB 10.5.27. You must start up and shut down MariaDB 10.7 or earlier.

      Seems there was no clean MariaDB shutdown before you installed the update.
      You could try to start only the old docker image of mariadb and trigger a clean shutdown.
      If this does not help would suggest you restore and check if mysql stops properly if the container is stopped.

      • THW1 replied to this.

          Have something to say?

          Join the community by quickly registering to participate in this discussion. We'd like to see you joining our great moo-community!

          • THW1

              Moolevel 0
            • Post #3
            • Edited

            esackbauer

            thanks - I checked for the db container image(s) but tbh it seems to much of a hassle and I am no restoring from the backup ~ fingers crossed 🤞

            shoot - I switched to the legacy branch and restored up to a week ago. But while the restore operations proceeded without an error, each restore attempt results in a vnilla mailcow setup, i.e., no mailboxes and admin on standard??

            shoot - I switched to the legacy branch and restored up to a week ago. But while the restore operations proceeded without an error, each restore attempt results in a vnilla mailcow setup, i.e., no mailboxes and admin on standard??

              • THW1

                  Moolevel 0
                • Post #6
                • Edited

                looks like something(??) broke 7-8 days and since the DB and dovecot including their backups are in a broken state. I.e., seeing an error like [1] in the restore outs…

                for good measures I have checked out the January release/commit 120366fec7b. Then I have restored to the last working backup I have found before the corruption seem to have taken place. Annoyingly, I had not noticed the issue/corruption, as apparently mail client operations had not been affected and I had been able to get my mails over the last days 🤷

                Maybe a sanity check for the DB before performing its particular backup might be a good idea. I have to check - maybe opening a feature request or merge request later, when I find some time.

                [1]

                Force a resync now? [y|N] y
Error: auth-master: userdb list: connect(/run/dovecot/auth-userdb) failed: Connection refused
Panic: file auth-master.c: line 436 (auth_master_unset_io): assertion failed: (conn->to == NULL)
7feb85488b1c
/rspamd/

                ok - I have autoupdates enabled for my base OS…

                …and looks like that on Mar.26 the docker packages got updated - which correlates quite with the time frame when things broke :-/

                thing to take from would be for me: do not allow autoupdate for docker packages!

                  THW1 thing to take from would be for me: do not allow autoupdate for docker packages!

                  I do never autoupdate with such critical things. Usually I take first a snapshot/backup before updateing mailcow. If that runs through I will update the OS right afterwards. Especially docker updates have sometimes negative consequences due to some changes in the network stack, so this has to be checked closely after OS/package update.

                    No one is typing