Problems with big Mailbox (> 100k Mails in Inbox)

toasty

Hello Together,
we migrated over the weekend a company to mailcow (from an old exchange).

Did some PST Exports on the Exchange Server and import again with Outlook (IMAP Profiles).
After successfully import the mails and sync with mailcow, we switch the local Outlook Profiles to Active Sync Profiles for the Users.
This was working fine on nearly all clients but without the CEO s (Murphys law …)

He has over 120.000 mails in the Inbox folder and when we start connecting his Iphone / Laptop then it will sync ~ 500 mails from the Inbox Folder and stop working. Outlook says “Everything is Up to Date” but in SoGo and in the IMAP Profile we see that there is a lot missing …

I saw a tipp to setup the Dovecout value “imap_max_line_length” to be able to handle bigger folders but we still ran in the same error.

(created an extra.conf with the following content:
root@mail:/opt/mailcow-dockerized/data/conf/dovecot# cat extra.conf
protocol imap {
mail_plugins = </etc/dovecot/mail_plugins_imap
imap_metadata = yes
imap_max_line_length = 2097152
}

Is there any Idea from one colleague to rescue my Monday ? 🙂

Thank you and best regards,
Thorsten

Additional Information: All Other Mailboxes with less mails in a single folder are working like expected… seams to be a problem of this huge inbox.

Are there some Best Practice for such big mailboxes / folders for the Dovecot / Sogo configuration?

(I know he has to cleanup but we need to get this running for the moment)

DocFraggle

Have a look at this thread, may be the same issue and/or an issue with not enough RAM for SOLR

https://community.mailcow.email/d/3491-errors-on-large-mailbox-logs-relate-to-solr/8

toasty

Hello @DocFraggle ,
thank you for the fast reply.

I tried this and could free about 6-8 GB of HDD Space but this could not solve the problem.

Here are some Logs from the SoGo Portal - maybe this is helpful?

XX.YY.ZZ.116 is the User Client
XX.YY.ZZ.19 is a Terminalserver from which i can test currently the mailbox

From my understanding there are a lot of “Http 501” Errors. I compared this with some different installations but i can only find them here …

=======================================================
06.05.2024, 08:42:12 notice [71]: xxx.yyy.zzz.19 “POST /SOGo/Microsoft-Server-ActiveSync?Cmd=Ping&User=firstname@domain.de&DeviceId=24BC2246547A4EF48837B736092C38F0&DeviceType=WindowsOutlook15 HTTP/1.0” 200 79/1486 0.289 - - 0 - 25
06.05.2024, 08:42:12 notice [71]: <0×0×5597f4ecd490[SOGoActiveSyncDispatcher]> Change detected using Ping, we let the EAS client know to send a Sync.
06.05.2024, 08:42:12 notice [10040]: <0×0×5597f9322880[NGImap4Client]> Note: no key found for sorting, using ‘DATE’: (null)
06.05.2024, 08:42:12 notice [10040]: xxx.yyy.zzz.19 “POST /SOGo/Microsoft-Server-ActiveSync?Cmd=Sync&User=firstname@domain.de&DeviceId=24BC2246547A4EF48837B736092C38F0&DeviceType=WindowsOutlook15 HTTP/1.0” 501 0/2726 1.461 - - -20K - 33
06.05.2024, 08:42:12 notice 08:42:12.534 sogod[10:10040] EXCEPTION: <NSException: 0×5597f95ece60> NAME:NSInvalidArgumentException REASON:[NSString+stringWithUTF8String:]: NULL cString INFO🙁null)
06.05.2024, 08:42:12 notice [71]: xxx.yyy.zzz.116 “POST /SOGo/Microsoft-Server-ActiveSync?Cmd=Ping&User=firstname@domain.de&DeviceId=D3F5ADC638D74669B859CFCFBA8E3AD3&DeviceType=WindowsOutlook15 HTTP/1.0” 200 144/1486 0.470 - - 0 - 25
06.05.2024, 08:42:12 notice [71]: <0×0×5597f05690f0[SOGoActiveSyncDispatcher]> Change detected using Ping, we let the EAS client know to send a Sync.
06.05.2024, 08:42:12 notice [73]: <0×0×5597f1093a30[NGImap4Client]> Note: no key found for sorting, using ‘DATE’: (null)
06.05.2024, 08:42:11 notice [75]: <0×0×5597efae4cd0[SOGoActiveSyncDispatcher]> Change detected during Sync, we push the content.
06.05.2024, 08:42:11 notice [71]: xxx.yyy.zzz.116 “POST /SOGo/Microsoft-Server-ActiveSync?Cmd=Sync&User=firstname@domain.de&DeviceId=D3F5ADC638D74669B859CFCFBA8E3AD3&DeviceType=WindowsOutlook15 HTTP/1.0” 501 0/2656 1.543 - - -20K - 25
06.05.2024, 08:42:11 notice 08:42:11.572 sogod[71:71] EXCEPTION: <NSException: 0×5597f24daca0> NAME:NSInvalidArgumentException REASON:[NSString+stringWithUTF8String:]: NULL cString INFO🙁null)
06.05.2024, 08:42:11 notice [73]: xxx.yyy.zzz.19 “POST /SOGo/Microsoft-Server-ActiveSync?Cmd=Ping&User=firstname@domain.de&DeviceId=24BC2246547A4EF48837B736092C38F0&DeviceType=WindowsOutlook15 HTTP/1.0” 200 79/1486 0.323 - - 0 - 27
06.05.2024, 08:42:11 notice [73]: <0×0×5597ef3b0550[SOGoActiveSyncDispatcher]> Change detected using Ping, we let the EAS client know to send a Sync.
06.05.2024, 08:42:11 notice [10040]: <0×0×5597f82fa4e0[NGImap4Client]> Note: no key found for sorting, using ‘DATE’: (null)
06.05.2024, 08:42:11 notice [75]: <0×0×5597f0e3ef20[NGImap4Client]> Note: no key found for sorting, using ‘DATE’: (null)

esackbauer

Is there a reverse proxy or WAF in front? Could also be a source of the problem, especially with WAF IPS rules.
Which client are you using? Microsoft Outlook? Mobile or Desktop? Try something else like EM Client.
SOGo is not 100% ActiveSync compliant, as it is reverse engineered. It adheres to the ActiveSync standard of 2016, so newer clients of Microsoft might give problems.

toasty

There is a Sophos in front but the Clients connects locally to this Server (over VPN or LAN / WIfi) - no Access without Access to the Network.

Clients are Mainly Outlook (2013 / 2016 / 2021) and Iphone Mail

esackbauer

And you do have enough memory and CPU as well as SOGo worker threads? You need 1 worker for each ActiveSync device, and then some more for SOGo itself, backend IMAP connections.
You could also try another email client like Bluemail app

[unknown] imap_max_line_length = 2097152

Also doubles this value, as these 2 MB were only tested with _100k mails. If your boss has more, you may need more.

toasty

I was running this Server (virtual) with 8 CPU and 24GB RAM. (20 Sogo Workers)
Now changed to 12 CPU and 36GB RAM + 40 Sogo Workers in sogo.conf + double the imap_max_length

Currently there are 22 Mailboxes

Will test it now - thank you for your help with this!

esackbauer

toasty I was running this Server (virtual) with 8 CPU and 24GB RAM. (20 Sogo Workers)

That should be fine even for 40 workers, as one has up to 350MB RAM.

toasty

Sooooo - thank you for your time to help me with this.

Could not find a suitable way to get this running so we split the Mailbox and reduce the content.
(Start with a clean Mailbox, creare more folders, less mails per folder, remove old things) and everything is fine.