Flatcurve indexing does not seem to work

jacotec

Hi,

after updating to 2025-01 which changes from Solr to Flatcurve all runs well … except that I have the feeling that the indexing does not work properly.

I’ve of course changed to “SKIP_FTS=n” and gave FTS 3 threads, then restarted Mailcow.

However, giving the command to reindex my mailbox just gives a short CPU peak for around 5-10 seconds. Ways to fast.

Doing a search for a word through the email bodies in SoGo starts 3 “imap” threads (which is what I have defined for FTS) and takes around a minute to come back with results. Doing the same search a few minutes later again starts 3 imap threads and again takes around a minute for results.

This is not what I expect … with Solr I saw FTS results very fast.

For me it looks like the indexing is not indexing … any ideas?

DocFraggle

Hmm, worked great for me. Did you run

docker compose exec dovecot-mailcow doveadm index -A '*'

I accidentally did a rescan first which finished 2 seconds later, which did nothing as there wasn’t an index in the first place 😃

The indexing took around half an hour for me

jacotec

DocFraggle
Yap …
docker-compose exec dovecot-mailcow doveadm index -A '*'

Returns nearly immediately, processes stay idle.

DocFraggle

Well, you can have a look if the index was created:

find /var/lib/docker/volumes/mailcowdockerized_vmail-index-vol-1/_data/ -type d -name fts-flatcurve

jacotec

… and there is no “fts-flatcurve” directory in my mailbox folders:

[unknown]
Negative. I’ve looked through a couple of boxes and also ran your find command … not a single fts-flatcurve index.

DocFraggle

You restarted via

docker compose down
docker compose up -d

? I had no problems.. this is my FTS config file inside the dovecot container

d26c173e5696:/# cat /etc/dovecot/conf.d/fts.conf
# mailcow FTS Flatcurve Settings, change them as you like.
plugin {
    fts_autoindex = yes
    fts_autoindex_exclude = \Junk
    fts_autoindex_exclude2 = \Trash
    # Tweak this setting if you only want to ensure big and frequent folders are indexed, not all.
    fts_autoindex_max_recent_msgs = 20
    fts = flatcurve

    # Maximum term length can be set via the 'maxlen' argument (maxlen is
    # specified in bytes, not number of UTF-8 characters)
    fts_tokenizer_email_address = maxlen=100
    fts_tokenizer_generic = algorithm=simple maxlen=30

    # These are not flatcurve settings, but required for Dovecot FTS. See
    # Dovecot FTS Configuration link above for further information.
    fts_languages = en es de
    fts_tokenizers = generic email-address

    # OPTIONAL: Recommended default FTS core configuration
    fts_filters = normalizer-icu snowball stopwords
    fts_filters_en = lowercase snowball english-possessive stopwords

    fts_index_timeout = 300s
}

### THIS PART WILL BE CHANGED BY MODIFYING mailcow.conf AUTOMATICALLY DURING RUNTIME! ###

service indexer-worker {
  # Max amount of simultaniously running indexer jobs.
  process_limit=1

  # Max amount of RAM used by EACH indexer process.
  vsz_limit=128 MB
}

jacotec

DocFraggle You restarted via

docker compose down
docker compose up -d

I did a full reboot because I also did a dist-upgrade which updated the linux kernel after the update.

But I guess the reboot is not the same as the docker-compose down and up again … no clue. I reran the update script (which did not pull any container updates) and now the fts files are created. FTS now returns results within seconds.

Somehow after running the update script, enabling FTS and then rebooting the host missed to add the fts to the mail plugin config files in the /conf/dovecot directory.

I’ve rechecked on my second Mailcow server and really … you need to run

docker-compose down
docker-compose up -d

to have the fts config added to the mail_plugin files. A complete reboot (which also restarts the containers) does not help. Interesting. Thanks for your help!

jacotec

This is interesting …

DocFraggle

For me it looks like you didn’t run “docker compose up -d” after editing the SKIP_FTS parameter. The docker-entrypoint.sh script would enable fts otherwise:

mailcow/mailcow-dockerizedblob/ba0349a9113575cea07fddc29d6060bc6795afed/data/Dockerfiles/dovecot/docker-entrypoint.sh#L120

DocFraggle

Yes, a reboot changes nothing regarding the docker compose config 🙂

Glad to have helped!