Hey there, I'm currently working on migrating my emails to a self-hosted mailcow instance running in the Hetzner Cloud. I tried to use Ansible as much as I can, which works really good (https://github.com/felixoi/mailcow-hetzner-ansible/). Now I wanted to test backup/restore before I start the production migration and got some problems with it. I'm using Borgmatic as described in the [docs](https://docs.mailcow.email/third_party/borgmatic/third_party-borgmatic/#manual-archiving-run-with-debugging-output). Backups are working good and running one an hour but when I restore I'm getting duplicated and unread messages in the testing account. Starting from these mails in the Inbox: [upl-image-preview url=https://community.mailcow.email/assets/files/2022-12-22/1671733265-346478-screenshot-from-2022-12-22-18-33-25.png] Restoring a backup from earlier this day: Removed readonly for the mounted borgmatic volumes. ``` docker-compose exec borgmatic-mailcow borgmatic restore --archive mailcow-2022-12-22T12:14:08.710457 docker-compose exec borgmatic-mailcow borgmatic extract --path mnt/source --archive mailcow-2022-12-22T09:32:48.442065 docker-compose down && docker-compose up -d ``` Results in something like this: [upl-image-preview url=https://community.mailcow.email/assets/files/2022-12-22/1671733260-941206-screenshot-from-2022-12-22-19-09-45.png] As you can see some emails are duplicated with a read and a new unread version. This is not what I expected. Someone has an idea what's going on maybe? Thanks, Felix

Restore results in duplicated and unread messages

felixoi

Hey there,

I’m currently working on migrating my emails to a self-hosted mailcow instance running in the Hetzner Cloud. I tried to use Ansible as much as I can, which works really good (https://github.com/felixoi/mailcow-hetzner-ansible/). Now I wanted to test backup/restore before I start the production migration and got some problems with it. I’m using Borgmatic as described in the docs. Backups are working good and running one an hour but when I restore I’m getting duplicated and unread messages in the testing account.
Starting from these mails in the Inbox:

Restoring a backup from earlier this day:
Removed readonly for the mounted borgmatic volumes.

docker-compose exec borgmatic-mailcow borgmatic restore --archive mailcow-2022-12-22T12:14:08.710457
docker-compose exec borgmatic-mailcow borgmatic extract --path mnt/source --archive mailcow-2022-12-22T09:32:48.442065
docker-compose down && docker-compose up -d

Results in something like this:

As you can see some emails are duplicated with a read and a new unread version.
This is not what I expected. Someone has an idea what’s going on maybe?

Thanks,
Felix

felixoi

Fixed it with docker-compose exec dovecot-mailcow doveadm -D deduplicate -m -u '*' mailbox INBOX but still would like to know why and when this happens.

felixoi

Ok, bumping this once again as I found the acutal underlaying issue.
Seems like docker-compose exec borgmatic-mailcow borgmatic restore --archive latest only merges the folder extracted the backup into the current folder state. This is why after a restore the maildirs end up in a weird situation. New mails in the maildir that were not existent at the time of the backup, will stay there. If you moved a mail then you are going to have duplicates as the restore will copy the old file into the old folder.

Is this intended behavior? How does backup_and_restore.sh work in this case? Is it totally resetting to the backup state? I can reproduce this for the Borgmatic setup described in the docs.