Problem after update

dtoenies

Hello all,
I did an update today. After the update, I then ran a ‘docker system prune’. Suddenly I could no longer access the server after it was still working. After restarting the Docker containers, I got the following error with unbound:

ERROR: for unbound-mailcow Cannot start service unbound-mailcow: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: rootfs_linux.go:59: mounting "/opt/mailcow-dockerized/data/conf/unbound/unbound.conf" to rootfs at "/var/lib/docker/overlay2/38eff2e62569ceff8d6db71e5795b92e93a81c895781b1c2956c7a3824f22ae9/merged/etc/unbound/unbound.conf" caused: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

So I deleted the unbound.conf and inserted the contents of this file: https://github.com/mailcow/mailcow-dockerized/blob/master/data/conf/unbound/unbound.conf

Now the server starts normally again, but I cannot access anything. I can’t retrieve or send mails or access SOGo at the moment.

Can anyone help me with this problem? Have I overlooked something?

Thank you very much in advance!

dtoenies

Okay, now it’s getting weird. All of a sudden, entire folder structures are no longer available. All the folders of my Nextcloud installed via mailcow are completely gone. I will have to completely rebuild the system and hope that the backup can be restored.

pkernstock

Have you run docker system prune while some containers were down? Have you ensure the containers were up and running?

dtoenies

I checked and they were running.
Just before everything went bad I updated and got this enormous diff_file: https://gofile.io/d/SXBGtU

diekuh

Something went wrong purging system data… never seen it happening, but this may help:

systemctl stop docker
rm -rf /var/lib/docker/aufs
rm -rf /var/lib/docker/overlay2
rm -rf /var/lib/docker/image/aufs
rm -f /var/lib/docker/linkgraph.db
systemctl start docker

Run docker-compose pull from within the mailcow directory afterwards.

The commands above will purge all containers and images!
User data (volumes) will not be removed.

dtoenies

Did this and now getting the following message if I try to start the docker images:
ERROR: stat /var/lib/docker/overlay2/bee3a585a05cfb96d5fc5a1db62865fee671e3cb5dc08968247c437312c89c14: no such file or directory

An other big problem is thr folder structure. For example this is all the folder ‘data/web’ contains:

/opt/mailcow-dockerized# ls -lisah data/web/
insgesamt 24K
4194393 4,0K drwxr-xr-x 6 root root 4,0K Jan 12 21:26 .
4194332 4,0K drwxr-xr-x 7 root root 4,0K Jan 12 18:11 ..
4194428 4,0K drwxr-xr-x 3 root root 4,0K Jan 12 18:11 inc
4194490 4,0K drwxr-xr-x 3 root root 4,0K Jan 12 21:26 js
4194431 4,0K drwxr-xr-x 2 root root 4,0K Jan 12 18:11 lang
4194459 4,0K drwxr-xr-x 3 root root 4,0K Jan 12 18:11 .well-known

I think I will completely reinstall the server. The biggest problem is that the Nextcloud data is also gone.

diekuh

Why would it be deleted? docker system prune does never ever touch these files. Nor do we delete them with git. That will never happen.

Something went wrong, but there is no way the update script would delete these files. You seem to also have a broken /var/lib/docker directory. It looks like someone deleted the wrong files or issued a malformed rm command. :/