Catastrophic destruction of /var/lib/docker

fatalerrors

Hi Mailcow comunity !

I am actually in a very delicate situation. I had an attack on my Proxmox Backup Server, leading to the destruction of all my VM snapshot. To fix the problem, I decided to harden a bit anything connected to the outside world. I used a script that ran without any problem on all other machines.

I installed Mailcow long ago using Debian Docker implementation. After receiving warnings last year about that implementation, I upgraded Docker using the official Docker repository. It worked like a charm.

All that context to explain the catastrophe: my hardening script call dpkg to purge all old configuration and data files from no longer installed packages. As Docker and Debian implementation have different package name, it leads to complete destruction of /var/lib/docker and no backup anymore of the system.

Fortunately, all mails are stored somewhere else on a CEPH share as well as the git tree with my configuration.

Here is what I made to try to recover:

Make an other copy of mail tree (just in case)
Reinstall docker and run the upgrade.sh script to redeploy docker images
Database being erased, recreate domains and mailboxes
When I connect, only the newly received message are counted
In SOGO, the directory structure appears correctly, but all messages are grayed and unclickable

I heard about a key, are all message cypherred? Is it recoverable without it, obviously completely lost?

About DKIM, I guess the public keys in my DNS are unusable without the private keys stored in the docker images. I don’t think it’s too bad, I could use this to update to 4096 bits keys instead of 2048.

I’m running mail for a couple of non-profit foundation, it’s really important if I can do anything to recover.

Please note that all message are grayed in SOGO, even new ones, while with my mail client I completely have access to new messages, old messages appears but don’t load.

Thank a lot for any help !

Please note that all message are grayed in SOGO, even new ones, while with my mail client I completely have access to new messages, old messages appears but don’t load.

Thanks a lot for any help !

DocFraggle

fatalerrors I heard about a key, are all message cypherred? Is it recoverable without it, obviously completely lost?

Yes, the mail files are encrypted, the key is not stored on the vmail volume. Didn’t you have backups from the rest of mailcow? Without the key file you won’t be able to decrypt your old emails

fatalerrors

That’s catastrophic, as I said, I engage the action because of external destruction of my backup directory. The key is not part of configuration in git tree (I remember a create config step) ?

Where is that key? A so important file should be subject to warning on installation, like, for example, Nextcloud does. It’s not normal I had to do research for that particular problem today and learn that mails are encrypted. Especially that docker storage is not considered reliable.

fatalerrors

I found an old vzdump from before I was using PBS. I guess I can recover the key from that file. How do I convert the old mails to the new key?

DocFraggle

See here:

https://docs.mailcow.email/manual-guides/Dovecot/u_e-dovecot-mail-crypt/

You would need connect to your dovecot container, put your old key pair somewhere and decrypt the mails. Afterwards reencrypt them with your new key pair

Maybe even better: decrypt your new mails, replace the new key pair with the old one, and reencrypt the new mails. Should be faster

fatalerrors

Thank you, I manage to recover, but I really would like to store those key somewhere else (outside of containers) for more safety and control. How ?

DocFraggle

Just copy them to your backup location then! I don’t really understand the question.

Use the mailcow backup script to make real backups, they do contain the key as well