Hello,
i did get into similar problem in my previous stack (not MailCow), so what I wrote is based on those experiences and may not apply in your case.
Searches and sorting across huge folders can be expensive to resources so 504 (timeout) can be fairly common in those situations. We solved it by re-indexing mailboxes at that time doveadm index -u user@example.com '*' not entirely sure, if MC for those search/sorts use FST try re-index or repair affected mailbox see the docs (adding more resource to FTS may help too,…)
For some mailboxes, even bigger than your case, we had to force user to use automatic script for filtering messages to folder structure, that helps keep index reasonably small (per folder). We had 2 strategies filtering by date and by domain (for messages older than 1 year).
You can try to bump timeouts, but i guess that’s not a solution you want, since users will wait too long,..