Issue with last 2x Updates Failing

skeelkat

Hi Everyone.

I have strange issue, that started with first attempt to updrade to 2026-01.

Running on Debian 12, If I update using update.sh script, everything runs perfect, new images pulled and containers restart. But mailcow remains stuck at the “What is happening?” screen.

When I check the logs, the only real error I see is “waiting for schema update” and “php on port 9001”, nothing else of value that I can determine.

The kicker is, when I clone the entire VM, and run the update on the cloned machine, update completes, no issues, but it is counter intuitive to clone a VM every time I want to upgrade Mailcow

I have minimal extra configs, pretty much only in postfix via extra.conf

I also have docker changes to allow access from Uptime Kuma via IP to monitor container statuses, and this is running via tailscale (no DNS, no routes)

I’m really at the end of my rope here and cannot for the life of me figure out why this is happening. Any insights to a solution would be appreciated

ETNyx

Hi, can you post whoami and umask ?

skeelkat

umask returns 022
woami returns root

ETNyx

skeelkat seems ok, those messages you wrote are repeating it self? Believe if it’s stuck on those they should appear every 5 seconds or so,… If not it does not seems to be the issue.

hans-holgersson

Hi.
I updated my mailcow instance yesterday to 2026-03a.
After update the Sogo does not work at all:

or if i access sogo from the Sogo button on the mailboxes admin page, it displays only a word on white background: email.

Any clue about this “feature”?

Thank you.

DocFraggle

hans-holgersson I suggest you run updates.sh again and restart the stack

skeelkat

ETNyx

Yes. Both of the errors are repeating. Rspamd and Sogod container

But nothing I do seems to bring it out of this state

hans-holgersson

DocFraggle i did it. Those was the first actions i took.
The issue remained.

DocFraggle

Did you try incognito mode / clearing the cache? Another browser or computer?

hans-holgersson

DocFraggle
I did.
Cleared the cahces, tried egde and FF (on windows 11) in incognito or not the issue is the same.
I tried on another account on the windows machine and on phone on chrome, the issue is present.

DocFraggle

hans-holgersson Last idea: check via

git status

in your mailcow directory if there are modified files causing the issue

git diff origin/master

hans-holgersson

DocFraggle

git status gives me that my local branch is ahead of origin master:

[root@smtp server]# git status
On branch master
Your branch is ahead of ‘origin/master’ by 52 commits.
(use “git push” to publish your local commits)

nothing to commit, working tree clean
[root@smtp server]#

git diff origin/master - show me a very long list of differences … most of them are file i never touched.

but if i do git fetc ; git pull it says me that “Already up to date.”

in the meantime i checked logs from sogo, nginx and phpfm containers, and found nginx is crying for a lot of files it does not found when i try to access https://<server-url>/SOGo:

nginx-mailcow-1 | 127.0.0.1 - - [15/Mar/2026:18:51:57 +0200] "GET /sogo-auth HTTP/1.0" 200 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0" php-fpm-mailcow-1 | 10.2.1.10 - 15/Mar/2026:18:51:57 +0200 "GET /sogo-auth.php" 200 sogo-mailcow-1 | Mar 15 18:51:57 0f0e9d9e58bc sogod [61]: <0x0x561347536f10[NGHttpRequest]> got 2 values for cookie 'MicrosoftApplicationsTelemetryDeviceId', using first only: ffd2291f-9c24-48fd-849d-dced7b36f1c4,ffd2291f-9c24-48fd-849d-dced7b36f1c4 sogo-mailcow-1 | Mar 15 18:51:57 0f0e9d9e58bc sogod [61]: <0x0x561347536f10[NGHttpRequest]> got 2 values for cookie 'MicrosoftApplicationsTelemetryFirstLaunchTime', using first only: 2026-03-15T16:42:15.185Z,2026-03-15T16:42:15.185Z nginx-mailcow-1 | <client_ip_address> - - [15/Mar/2026:18:51:57 +0200] "GET /SOGo/ HTTP/2.0" 200 9825 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0" sogo-mailcow-1 | Mar 15 18:51:57 0f0e9d9e58bc sogod [61]: <client_ip_address> "GET /SOGo/ HTTP/1.0" 200 9825/0 0.024 40061 75% 0 - 12 nginx-mailcow-1 | 2026/03/15 18:51:57 [error] 20#20: *54013 open() "/usr/local/lib/GNUstep/SOGo/WebServerResources/css/theme-default.css" failed (2: No such file or directory), client: <client_ip_address>, server: <server_url>, request: "GET /SOGo.woa/WebServerResources/css/theme-default.css?lm=1773145093 HTTP/2.0", host: "<server_url>", referrer: "https://<server_url>/" nginx-mailcow-1 | <client_ip_address> - - [15/Mar/2026:18:51:57 +0200] "GET /SOGo.woa/WebServerResources/css/theme-default.css?lm=1773145093 HTTP/2.0" 404 167 "https://<server_url>/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0" nginx-mailcow-1 | 2026/03/15 18:51:57 [error] 20#20: *54013 open() "/usr/local/lib/GNUstep/SOGo/WebServerResources/css/styles.css" failed (2: No such file or directory), client: <client_ip_address>, server: <server_url>, request: "GET /SOGo.woa/WebServerResources/css/styles.css?lm=1773145093 HTTP/2.0", host: "<server_url>", referrer: "https://<server_url>/" nginx-mailcow-1 | <client_ip_address> - - [15/Mar/2026:18:51:57 +0200] "GET /SOGo.woa/WebServerResources/css/styles.css?lm=1773145093 HTTP/2.0" 404 167 "https://<server_url>/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0" nginx-mailcow-1 | 2026/03/15 18:51:57 [error] 20#20: *54013 open() "/usr/local/lib/GNUstep/SOGo/WebServerResources/img/sogo-full.svg" failed (2: No such file or directory), client: <client_ip_address>, server: <server_url>, request: "GET /SOGo.woa/WebServerResources/img/sogo-full.svg?lm=1742925855 HTTP/2.0", host: "<server_url>", referrer: "https://<server_url>/" nginx-mailcow-1 | <client_ip_address> - - [15/Mar/2026:18:51:57 +0200] "GET /SOGo.woa/WebServerResources/img/sogo-full.svg?lm=1742925855 HTTP/2.0" 404 167 "https://<server_url>/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0" nginx-mailcow-1 | 2026/03/15 18:51:57 [error] 20#20: *54013 open() "/usr/local/lib/GNUstep/SOGo/WebServerResources/js/vendor/lodash.min.js" failed (2: No such file or directory), client: <client_ip_address>, server: <server_url>, request: "GET /SOGo.woa/WebServerResources/js/vendor/lodash.min.js?lm=1773145093 HTTP/2.0", host: "<server_url>", referrer: "https://<server_url>/" nginx-mailcow-1 | <client_ip_address> - - [15/Mar/2026:18:51:57 +0200] "GET /SOGo.woa/WebServerResources/js/vendor/lodash.min.js?lm=1773145093 HTTP/2.0" 404 167 "https://<server_url>/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0" nginx-mailcow-1 | 2026/03/15 18:51:57 [error] 20#20: *54013 open() "/usr/local/lib/GNUstep/SOGo/WebServerResources/js/vendor/angular.min.js" failed (2: No such file or directory), client: <client_ip_address>, server: <server_url>, request: "GET /SOGo.woa/WebServerResources/js/vendor/angular.min.js?lm=1773145093 HTTP/2.0", host: "<server_url>", referrer: "https://<server_url>/" nginx-mailcow-1 | <client_ip_address> - - [15/Mar/2026:18:51:57 +0200] "GET /SOGo.woa/WebServerResources/js/vendor/angular.min.js?lm=1773145093 HTTP/2.0" 404 167 "https://<server_url>/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0" nginx-mailcow-1 | 2026/03/15 18:51:57 [error] 20#20: *54013 open() "/usr/local/lib/GNUstep/SOGo/WebServerResources/js/vendor/angular-animate.min.js" failed (2: No such file or directory), client: <client_ip_address>, server: <server_url>, request: "GET /SOGo.woa/WebServerResources/js/vendor/angular-animate.min.js?lm=1773145093 HTTP/2.0", host: "<server_url>", referrer: "https://<server_url>/" nginx-mailcow-1 | <client_ip_address> - - [15/Mar/2026:18:51:57 +0200] "GET /SOGo.woa/WebServerResources/js/vendor/angular-animate.min.js?lm=1773145093 HTTP/2.0" 404 167 "https://<server_url>/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0" nginx-mailcow-1 | 2026/03/15 18:51:57 [error] 20#20: *54013 open() "/usr/local/lib/GNUstep/SOGo/WebServerResources/js/vendor/angular-sanitize.min.js" failed (2: No such file or directory), client: <client_ip_address>, server: <server_url>, request: "GET /SOGo.woa/WebServerResources/js/vendor/angular-sanitize.min.js?lm=1773145093 HTTP/2.0", host: "<server_url>", referrer: "https://<server_url>/" nginx-mailcow-1 | <client_ip_address> - - [15/Mar/2026:18:51:57 +0200] "GET /SOGo.woa/WebServerResources/js/vendor/angular-sanitize.min.js?lm=1773145093 HTTP/2.0" 404 167 "https://<server_url>/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0" nginx-mailcow-1 | 2026/03/15 18:51:57 [error] 20#20: *54013 open() "/usr/local/lib/GNUstep/SOGo/WebServerResources/js/vendor/angular-aria.min.js" failed (2: No such file or directory), client: <client_ip_address>, server: <server_url>, request: "GET /SOGo.woa/WebServerResources/js/vendor/angular-aria.min.js?lm=1773145093 HTTP/2.0", host: "<server_url>", referrer: "https://<server_url>/" nginx-mailcow-1 | <client_ip_address> - - [15/Mar/2026:18:51:57 +0200] "GET /SOGo.woa/WebServerResources/js/vendor/angular-aria.min.js?lm=1773145093 HTTP/2.0" 404 167 "https://<server_url>/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0" nginx-mailcow-1 | 2026/03/15 18:51:57 [error] 20#20: *54013 open() "/usr/local/lib/GNUstep/SOGo/WebServerResources/js/vendor/angular-cookies.min.js" failed (2: No such file or directory), client: <client_ip_address>, server: <server_url>, request: "GET /SOGo.woa/WebServerResources/js/vendor/angular-cookies.min.js?lm=1773145093 HTTP/2.0", host: "<server_url>", referrer: "https://<server_url>/" nginx-mailcow-1 | <client_ip_address> - - [15/Mar/2026:18:51:57 +0200] "GET /SOGo.woa/WebServerResources/js/vendor/angular-cookies.min.js?lm=1773145093 HTTP/2.0" 404 167 "https://<server_url>/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0" nginx-mailcow-1 | 2026/03/15 18:51:57 [error] 20#20: *54013 open() "/usr/local/lib/GNUstep/SOGo/WebServerResources/js/vendor/angular-messages.min.js" failed (2: No such file or directory), client: <client_ip_address>, server: <server_url>, request: "GET /SOGo.woa/WebServerResources/js/vendor/angular-messages.min.js?lm=1773145093 HTTP/2.0", host: "<server_url>", referrer: "https://<server_url>/" nginx-mailcow-1 | <client_ip_address> - - [15/Mar/2026:18:51:57 +0200] "GET /SOGo.woa/WebServerResources/js/vendor/angular-messages.min.js?lm=1773145093 HTTP/2.0" 404 167 "https://<server_url>/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0" nginx-mailcow-1 | 2026/03/15 18:51:57 [error] 20#20: *54013 open() "/usr/local/lib/GNUstep/SOGo/WebServerResources/js/vendor/angular-material.min.js" failed (2: No such file or directory), client: <client_ip_address>, server: <server_url>, request: "GET /SOGo.woa/WebServerResources/js/vendor/angular-material.min.js?lm=1773145093 HTTP/2.0", host: "<server_url>", referrer: "https://<server_url>/" nginx-mailcow-1 | <client_ip_address> - - [15/Mar/2026:18:51:57 +0200] "GET /SOGo.woa/WebServerResources/js/vendor/angular-material.min.js?lm=1773145093 HTTP/2.0" 404 167 "https://<server_url>/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0" nginx-mailcow-1 | <client_ip_address> - - [15/Mar/2026:18:51:57 +0200] "GET /SOGo.woa/WebServerResources/js/vendor/angular-ui-router.min.js?lm=1773145093 HTTP/2.0" 404 167 "https://<server_url>/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0" nginx-mailcow-1 | 2026/03/15 18:51:57 [error] 20#20: *54013 open() "/usr/local/lib/GNUstep/SOGo/WebServerResources/js/vendor/angular-ui-router.min.js" failed (2: No such file or directory), client: <client_ip_address>, server: <server_url>, request: "GET /SOGo.woa/WebServerResources/js/vendor/angular-ui-router.min.js?lm=1773145093 HTTP/2.0", host: "<server_url>", referrer: "https://<server_url>/" nginx-mailcow-1 | 2026/03/15 18:51:57 [error] 20#20: *54013 open() "/usr/local/lib/GNUstep/SOGo/WebServerResources/js/Main.js" failed (2: No such file or directory), client: <client_ip_address>, server: <server_url>, request: "GET /SOGo.woa/WebServerResources/js/Main.js?lm=1773145093 HTTP/2.0", host: "<server_url>", referrer: "https://<server_url>/" nginx-mailcow-1 | <client_ip_address> - - [15/Mar/2026:18:51:57 +0200] "GET /SOGo.woa/WebServerResources/js/Main.js?lm=1773145093 HTTP/2.0" 404 167 "https://<server_url>/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0"

DocFraggle

i want to mention, i never have made changes to the SOGo directoies manually …

DocFraggle

i want to mention, i never have made changes to the SOGo directoies manually …

ETNyx

Than you need to trace further,.. Expect you are not able to reproduce it right now,… Check where it comes from, check depends_on in docker compose and try to find the problem.

PHP message comes from rspamd-mailcow is pointing php-fpm-mailcow that depend on redis-mailcow that depend on netfilter-mailcow so lot of surface to your search,..

skeelkat

ETNyx

Not very helpful 🤣. Been tracing and going over the logs for the past week now… But the only way I guess.

Thank you for the cross checking of my sanity though that is very much appreciated. I will soldier on for the next week or two.

BUT

I think I will create a new VM, in Isolated Environment and run updates and migrate all live data at a time of least disruption

DocFraggle

hans-holgersson git diff origin/master - show me a very long list of differences … most of them are file i never touched.

OK, I guess here’s your problem. You should reset files where you can be sure it’s not something you locally modified by yourself to the files from the origin/master from GitHub.

If you want paste the output here so we can have a look

hans-holgersson

DocFraggle

diff.diff

23kB

here is the diff file for git diff origin/master

i got in the nginx container, and found something:
the directory /usr/local/lib/GNUstep/SOGo/ is completely empty, so this epxlains why nginx claims that file in that directory can not be found …

have something changed in how the containers external directories are used/mounted? i am asking this, because at the beginnings, when i installed mailcow i moved all important directories from /var/lib/docker/volumes to a location which is not so deeply burried in the filesystem.

so … the fact that in logs saw errors regarding files not found, i found the issue:
as i said i moved directories defined in volumes section of the docker services, in order to have an eye on them.
with the latest update, some volumes definitions got changed, in my case the problem was this line, in the nginx-mailcow container definition:

sogo-web-vol-1:/usr/local/lib/GNUstep/SOGo/

because my volumes were moved, in that location the docker engine founde nothing, so nginx cryed for nonexistent files.
with this i found a few more volumes that was changed in docker-compose.yml.

thanks for your time!

DocFraggle

hans-holgersson i am asking this, because at the beginnings, when i installed mailcow i moved all important directories from /var/lib/docker/volumes to a location which is not so deeply burried in the filesystem.

Even if I don’t really get why you did this, it would have been great if you told us about this from the beginning… if you change the defaults it’s mostly just a matter of time before things break