Database keeps crashing every

rose

for the context, i am using the same mysql container version (mariadb:10.11) as stated in the repo, and the my.cnf file is also exactly the same.

Today i did the following commands:

docker compose pull ; docker compose down ; docker compose up -d

however this resulted in the following mysql errors:

mysql-mailcow-1  | 2026-06-09 16:33:09 0x77cee6fff640  InnoDB: Assertion failure in file ./storage/innobase/trx/trx0purge.cc line 901
mysql-mailcow-1  | InnoDB: Failing assertion: tail.trx_no <= last_trx_no
mysql-mailcow-1  | InnoDB: We intentionally generate a memory trap.
mysql-mailcow-1  | InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
mysql-mailcow-1  | InnoDB: If you get repeated assertion failures or crashes, even
mysql-mailcow-1  | InnoDB: immediately after the mariadbd startup, there may be
mysql-mailcow-1  | InnoDB: corruption in the InnoDB tablespace. Please refer to
mysql-mailcow-1  | InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
mysql-mailcow-1  | InnoDB: about forcing recovery.
mysql-mailcow-1  | 260609 16:33:09 [ERROR] mariadbd got signal 6 ;

this error keeps happening and crashing the mysql container non-stop by default, and it keeps occuring every 15 or so minutes instead when i add the following line in the docker-compose.yaml file for the mysql-mailcow container:

      command: ["--innodb-force-recovery=3"]

after a bit of digging i found out that the mailcow.sasl_log table was corrupted:

MariaDB [mailcow]> CHECK TABLE mailcow.sasl_log EXTENDED;
+------------------+-------+----------+-----------------------------------------------+
| Table            | Op    | Msg_type | Msg_text                                      |
+------------------+-------+----------+-----------------------------------------------+
| mailcow.sasl_log | check | Warning  | InnoDB: Index datetime is marked as corrupted |
| mailcow.sasl_log | check | error    | Corrupt                                       |
+------------------+-------+----------+-----------------------------------------------+
2 rows in set (0.032 sec)

MariaDB [mailcow]> SELECT COUNT(*) FROM mailcow.sasl_log;
ERROR 1712 (HY000): Index sasl_log is corrupted

What is the correct way to fix this error ? recreating the table ? i tried truncating it but it exceeds the lock wait timeout:

MariaDB [mailcow]> TRUNCATE TABLE mailcow.sasl_log;
ERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction

rose

i ran a mysqlcheck against all the databases on the mysql-mailcow container and i found out that there are 3 tables that have their Index marked as corrupted:

root@mailcow:/mailcow# docker compose exec mysql-mailcow  mysqlcheck -c --all-databases -u root -p${DBROOT}
mailcow._sogo_static_view                          OK
mailcow.admin                                      OK
mailcow.alias                                      OK
mailcow.alias_domain                               OK
mailcow.api                                        OK
mailcow.app_passwd                                 OK
mailcow.bcc_maps                                   OK
mailcow.da_acl                                     OK
mailcow.da_sso                                     OK
mailcow.domain                                     OK
mailcow.domain_admins                              OK
mailcow.domain_wide_footer                         OK
mailcow.fido2                                      OK
mailcow.filterconf                                 OK
mailcow.forwarding_hosts                           OK
mailcow.imapsync                                   OK
mailcow.logs
Warning  : InnoDB: Index PRIMARY is marked as corrupted
error    : Corrupt
mailcow.mailbox
Warning  : InnoDB: Index domain is marked as corrupted
Warning  : InnoDB: Index kind is marked as corrupted
error    : Corrupt
mailcow.oauth_access_tokens                        OK
mailcow.oauth_authorization_codes                  OK
mailcow.oauth_clients                              OK
mailcow.oauth_refresh_tokens                       OK
mailcow.pushover                                   OK
mailcow.quarantine                                 OK
mailcow.quota2                                     OK
mailcow.quota2replica                              OK
mailcow.recipient_maps                             OK
mailcow.relayhosts                                 OK
mailcow.sasl_log
Warning  : InnoDB: Index datetime is marked as corrupted
error    : Corrupt
mailcow.sender_acl                                 OK
mailcow.settingsmap                                OK

...

i don’t think the mailcow.logs, and mailcow.sasl_log tables are that important i think so i suggest to try and drop the table s, but mailcow.mailbox does seem to be critical and it looks like it is corrupted, need some assistance on how to fix those 3 tables in particular.

ETNyx

Hello

first of all backup those data,… than you can try to restore those indexes, first check how it’s made

SHOW CREATE TABLE sasl_log

than recreate, make sure it is no at foreign key

ALTER TABLE sasl_log DROP INDEX `datetime`, ADD INDEX `datetime` (`datetime`);

Or this should rewrite whole table include indexes

ALTER TABLE sasl_log ENGINE=InnoDB;

Also check for OOM killers on your machine and do you have enough disk space?

rose

ETNyx

thanks, will try those today and keep you posted

the server has plenty of ram available and plenty of disk available too so i think i should focus on correcting the corruptions first

rose

looks like i found the fix, for whoever faces the same issue, need to run the following on the DB:

ALTER TABLE mailbox ENGINE=InnoDB;

(trying to do ALTER TABLE on the other 2 tables resulted in an error so i had to do the following instead) : 

SET GLOBAL innodb_lock_wait_timeout = 1;

CREATE TABLE mailcow.sasl_log_new LIKE mailcow.sasl_log;
ALTER TABLE mailcow.sasl_log RENAME TO mailcow.sasl_log_corrupted;
CREATE TABLE mailcow.sasl_log LIKE mailcow.sasl_log_new;
DROP TABLE mailcow.sasl_log_new;
DROP TABLE mailcow.sasl_log_corrupted;

CREATE TABLE mailcow.logs_new LIKE mailcow.logs;
ALTER TABLE mailcow.logs RENAME TO mailcow.logs_corrupted;
CREATE TABLE mailcow.logs LIKE mailcow.logs_new;
DROP TABLE mailcow.logs_new;
DROP TABLE mailcow.logs_corrupted;

then the mailcow databases are all OK afterwards

root@mailcow:/mailcow# docker compose exec mysql-mailcow  mysqlcheck -c --all-databases -u root -p${DBROOT}
mailcow._sogo_static_view                          OK
mailcow.admin                                      OK
mailcow.alias                                      OK
mailcow.alias_domain                               OK
mailcow.api                                        OK
mailcow.app_passwd                                 OK
mailcow.bcc_maps                                   OK
mailcow.da_acl                                     OK
mailcow.da_sso                                     OK
mailcow.domain                                     OK
mailcow.domain_admins                              OK
mailcow.domain_wide_footer                         OK
mailcow.fido2                                      OK
mailcow.filterconf                                 OK
mailcow.forwarding_hosts                           OK
mailcow.imapsync                                   OK
mailcow.logs                                       OK
mailcow.mailbox                                    OK
mailcow.oauth_access_tokens                        OK
mailcow.oauth_authorization_codes                  OK
mailcow.oauth_clients                              OK
mailcow.oauth_refresh_tokens                       OK
mailcow.pushover                                   OK
mailcow.quarantine                                 OK
mailcow.quota2                                     OK
mailcow.quota2replica                              OK
mailcow.recipient_maps                             OK
mailcow.relayhosts                                 OK
mailcow.sasl_log                                   OK
mailcow.sender_acl                                 OK
mailcow.settingsmap                                OK
mailcow.sieve_filters                              OK
mailcow.sogo_acl                                   OK
mailcow.sogo_alarms_folder                         OK
mailcow.sogo_cache_folder                          OK
mailcow.sogo_folder_info                           OK
mailcow.sogo_quick_appointment                     OK
mailcow.sogo_quick_contact                         OK
mailcow.sogo_sessions_folder                       OK
mailcow.sogo_store                                 OK
mailcow.sogo_user_profile                          OK
mailcow.spamalias                                  OK
mailcow.tags_domain                                OK
mailcow.tags_mailbox                               OK
mailcow.templates                                  OK
mailcow.tfa                                        OK
mailcow.tls_policy_override                        OK
mailcow.transports                                 OK
mailcow.user_acl                                   OK
mailcow.versions                                   OK
mysql.column_stats                                 OK
mysql.columns_priv                                 OK
mysql.db                                           OK
mysql.event                                        OK
mysql.func                                         OK
mysql.global_priv                                  OK
mysql.gtid_slave_pos                               OK
mysql.help_category                                OK
mysql.help_keyword                                 OK
mysql.help_relation                                OK
mysql.help_topic                                   OK
mysql.index_stats                                  OK
mysql.innodb_index_stats                           OK
mysql.innodb_table_stats                           OK
mysql.plugin                                       OK
mysql.proc                                         OK
mysql.procs_priv                                   OK
mysql.proxies_priv                                 OK
mysql.roles_mapping                                OK
mysql.servers                                      OK
mysql.table_stats                                  OK
mysql.tables_priv                                  OK
mysql.time_zone                                    OK
mysql.time_zone_leap_second                        OK
mysql.time_zone_name                               OK
mysql.time_zone_transition                         OK
mysql.time_zone_transition_type                    OK
mysql.transaction_registry                         OK
sys.sys_config                                     OK

then need to dump it all into a .sql file, create a new mysql container, create the ‘mailcow’@‘%’ user in the new mysql container, to be able to import the .sql file, and then copy the mysql volume of the new mysql container to replace the main production one, without the –innodb-force-recovery=2, and the database no longer crashes like before.

Still no idea why this occured in the first place however.