Search slow and can not search over all folders?

Lars2

Hey everyone,
I have imported that past 20 years of my mails. However, the serverside search does not return any or way too few results.
I have an archive-folder with yearly subfolders, so archive/2023/, archive/2022/, archive/2021/, etc.
Solr is running according to docker-compose and the admin menu, and shows roughly the amount of mails that I also have in my thunderbird.
Using the SoGO mail interface search, when I search for a title, a sender, something, I can see the request from dovecot-fts been made to the solr logs.
When I specifically search in one of the archived subfolders (e.g. by selecting archive/2022 which contains roughly 7.000 mails), I can get a result after around a minute or so. Which is unusably slow.
Also, I did not find a possibility to search over my complete folder-structure, which is quite an annoyance.
The VM this is running in has 6gb of RAM and uses around 3.3gb.

Is there any way to tweak solr to return a search result faster?
Can I search over all folders somehow?

While I’m happy to not use the webui, all IMAP connected devices with server-side search available suffer the same problems.

Best,
Lars

esackbauer

you give no indication how much memory the host is using, and how much for solr. Also we don’t know nothing about your CPU, how many, if they are guaranteed or shared, and you have not written how much it is used.

Search all folders in SOGo:
in the left pane, click the 3 dots next to your email address, click search.
Then check the cogwheel in the middle pane, if searching subfolder is activated.
For IMAP I don’t know, seems to depend on the client.

Lars2

There are two dedicated cpus, the VMs’ load is at 0.5 on average. If I search heavily, the load may hit 3.5 - yes, that is a lot.

Thanks for the tipp of the other search interface - I was using the quicksearch(?) before:

Here’s the memory usage of the docker containers while idling:

And here it is while searching for a sender mailadress from the web interface - it seems that solr is not used at all, but dovecot? I see the same behavior when searching for a specific content.

esackbauer

Stupid question: Is solr activated in mailcow.conf? The container will run even if it is set to disable. The low memory consumption lets me assume that.

Lars2

esackbauer

Here’s the relevant settings from the mailcow.conf - did I miss or misunderstood a setting?
SOLR_PORT=127.0.0.1:18983 SKIP_SOLR=n SOLR_HEAP=1024

tbrummell

I think @esackbauer may be on the right track. I run with Solr on SSD Nodes with 6 cores and 24 gigs of RAM. The system uses 3 gigs of RAM (and no swap) when idle, load average: 0.19, 0.52, 0.61.
When I do a search with SOGo, CPU rises (sogod) but that’s it. I get results from 2009 almost instantly. Although I don’t have nested folders. My wife has 1000’s of emails in nested folders, I’ll try her account….

When using her account with nested folders, if I search for something 1 level below INBOX, it returns data instantly (almost). If I search for something within a nested folder, I do get an error popup in SOGo.

esackbauer

@Lars2 That looks ok.
Maybe you should check this here:
https://docs.mailcow.email/manual-guides/Dovecot/u_e-dovecot-fts/

Lars2

Interestingly, a reindex did not do anything and finished after 2sec. Same for a rescan. But repeating the reindex now starts working as expected. After it finished, the search is a lot faster. Thanks for the help!

mayk

I have the same issue. Discussed this in the telegram group a few times and if you search you can find posts of me in the forum about this also i think. There is a solr replacement in the making that hopefully will be available “soon” .
What kind of works for me, disable solr. The search in my mailbox and my partners mailbox when she searches takes “long” but you get the results. The faster the server the faster the results i presume.