Mailcow lost all data after docker upgrade

stoyan-atanasoff

Hello guys, I have set up mailcow dockerized but didn’t upgrade it since this January. When I did it I lost everything. No users, no mail boxes, no domains. I can log only with admin:moohoo.
I hope the containers and images are somewhere saved on the disk space so I’ll abe able to recover my emails.

This is the log and basically everything I have as a trace back. Please I need your help.

root@mail:/opt/mailcow-dockerized# ./update.sh 
Checking internet connection... OK
Checking for newer update script...
remote: Enumerating objects: 3224, done.
remote: Counting objects: 100% (2238/2238), done.
remote: Compressing objects: 100% (405/405), done.
remote: Total 3224 (delta 1901), reused 2147 (delta 1830), pack-reused 986
Receiving objects: 100% (3224/3224), 4.86 MiB | 11.09 MiB/s, done.
Resolving deltas: 100% (2405/2405), completed with 201 local objects.
From https://github.com/mailcow/mailcow-dockerized
   b68eae16..cb058e91  master               -> origin/master
 * [new branch]        feature/bootstrap5   -> origin/feature/bootstrap5
 * [new branch]        feature/cors         -> origin/feature/cors
   f0e27312..eb4a33dc  feature/fts-xapian   -> origin/feature/fts-xapian
 * [new branch]        feature/mariadb-10.6 -> origin/feature/mariadb-10.6
 * [new branch]        feature/pigz-backup  -> origin/feature/pigz-backup
   e1db347d..a49305fd  hotfix               -> origin/hotfix
 * [new branch]        nightly              -> origin/nightly
 * [new branch]        postfix-myorigin-fix -> origin/postfix-myorigin-fix
   07ac195f..922603f9  staging              -> origin/staging
 * [new tag]           2022-09a             -> 2022-09a
 * [new tag]           2022-05d             -> 2022-05d
 * [new tag]           2022-06              -> 2022-06
 * [new tag]           2022-06a             -> 2022-06a
 * [new tag]           2022-06b             -> 2022-06b
 * [new tag]           2022-07              -> 2022-07
 * [new tag]           2022-07a             -> 2022-07a
 * [new tag]           2022-08              -> 2022-08
 * [new tag]           2022-08a             -> 2022-08a
 * [new tag]           2022-08b             -> 2022-08b
 * [new tag]           2022-09              -> 2022-09
update.sh changed, please run this script again, exiting.
root@mail:/opt/mailcow-dockerized# ./update.sh 
Cannot find Docker Compose with a Version Higher than 2.X.X.
Please update/install regarding to this doc site: https://mailcow.github.io/mailcow-dockerized-docs/i_u_m/i_u_m_install/
root@mail:/opt/mailcow-dockerized# apt update
Get:1 http://security.debian.org buster/updates InRelease [34.8 kB]
Hit:2 http://repo.mysql.com/apt/debian buster InRelease                                                                                                                                                                                                      
Hit:3 https://deb.nodesource.com/node_12.x buster InRelease                                                                                                                                                                                                  
Hit:4 http://debian.mirror.serveriai.lt/debian buster InRelease                                                                                                                                                                                              
Get:5 https://download.docker.com/linux/debian buster InRelease [54.0 kB]                 
Ign:6 http://repo.mongodb.org/apt/debian buster/mongodb-org/4.4 InRelease                                
Get:7 http://security.debian.org buster/updates/main amd64 Packages [379 kB]                             
Hit:9 http://repo.mongodb.org/apt/debian buster/mongodb-org/4.4 Release                                                             
Hit:11 https://packages.sury.org/php buster InRelease                                                                                            
Hit:8 https://packages.gitlab.com/gitlab/gitlab-ce/debian buster InRelease
Fetched 468 kB in 1s (483 kB/s)
Reading package lists... Done
Building dependency tree       
Reading state information... Done
7 packages can be upgraded. Run 'apt list --upgradable' to see them.
root@mail:/opt/mailcow-dockerized# apt install docker-compose-plugin
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libauthen-pam-perl libio-pty-perl libnet-ssleay-perl perl-openssl-defaults
Use 'apt autoremove' to remove them.
The following NEW packages will be installed:
  docker-compose-plugin
0 upgraded, 1 newly installed, 0 to remove and 7 not upgraded.
Need to get 9,562 kB of archives.
After this operation, 44.7 MB of additional disk space will be used.
Get:1 https://download.docker.com/linux/debian buster/stable amd64 docker-compose-plugin amd64 2.12.0~debian-buster [9,562 kB]
Fetched 9,562 kB in 1s (10.5 MB/s)                 
Selecting previously unselected package docker-compose-plugin.
(Reading database ... 232453 files and directories currently installed.)
Preparing to unpack .../docker-compose-plugin_2.12.0~debian-buster_amd64.deb ...
Unpacking docker-compose-plugin (2.12.0~debian-buster) ...
Setting up docker-compose-plugin (2.12.0~debian-buster) ...
root@mail:/opt/mailcow-dockerized# ./update.sh 
Found Docker Compose Plugin (native).
Setting the DOCKER_COMPOSE_VERSION Variable to native
Notice: You'll have to update this Compose Version via your Package Manager manually!
Adding new option "DOCKER_COMPOSE_VERSION" to mailcow.conf
Checking internet connection... OK
Detecting which build your mailcow runs on...
You are receiving stable updates (master).
To change that run the update.sh Script one time with the --nightly parameter to switch to nightly builds.
Checking for newer update script...
Are you sure you want to update mailcow: dockerized? All containers will be stopped. [y/N] y
Great! Native IPv6 NAT is active.
Validating docker-compose stack configuration...
Checking for conflicting bridges...
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Saving diff to update_diffs/diff_before_update_2022-10-24-13-12-05...
Prefetching images...
........

DerLinkman

Solved on Telegram.

pkernstock

To provide some more context here: It was NOT an issue with any mailcow-code or the update script.

After the update unfortunately the command docker-compose down -v was run. The parameter -v however DELETES ALL docker volumes with all its volumes in a unrecoverable way. That was manually accidentally run and caused the data to be deleted. It was NOT run by the update script.

sadasiva

pkernstock how to recover from that situation

aaronsmith

Thanks for sharing! Figured I’d add context on what -v means:

https://docs.docker.com/get-started/08_using_compose/

Removing Volumes

By default, named volumes in your compose file are NOT removed when running docker compose down. If you want to remove the volumes, you will need to add the –volumes flag.

The Docker Dashboard does not remove volumes when you delete the app stack.

pkernstock

sadasiva Restore your backup.

sadasiva

pkernstock
i didn’t take any manual backup

[unknown]
is it possible

to restore

[unknown]
is it possible

to restore

[unknown]
is it possible

to restore

[unknown]
is it possible

to restore

sadasiva

pkernstock

If you have used -v and docker has indeed deleted all your data volumes, the data is gone. You might potentially make a offline copy of your server’s disk and use some recovery utilities to restore some data, or use some specialized data recovery company to do that for you.

But if docker deleted the data, it is deleted. If you don’t have any backups or if you can’t recover them using different utilities, you’ll need to rebuild everything from scratch.

jabertech

Hi,
I accidently removed all volumes (was following another blog post) and this deleted all my data…fortunately I had taken a back-up so I have all the vmail etc…but it seems mySQL backup was corrupt and the db is empty - any possibility to re-construct the db again?