I had an issue with mailcow’s default inline synchronous ham/spam learning when a user moved more than a few mails in/out of their Junk mailbox. The delays induced would cause timeouts on the client, and the client would then re-copy the messages, resulting in infinite mail duplications on my clients. I eventually worked out the cause, and implemented this solution, which I have pushed to a github repository.
TLDR;
This makes it so Ham/Spam learning is super quick when you move messages in/out of your Junk folder.
I’m not an expert on Docker, Mailcow, Dovecot or Bash, so if anyone has any suggestions for improvements, I’d appreciate them.
https://github.com/mrstux/imapsieve_copy
—
imapsieve_copy for Mailcow
Asynchronous ham/spam learning for mailcow-dovecot
In a normal mailcow installation, spam/ham learning as a result of user actions
is done in-line. This slows down moving mails to and from the Junk folders. This
causes issues with timeouts on clients, which can then cause mails to be
re-copied which results in possible infinite mail duplications, and a poor user
experience.
The imapsieve_copy tool copies these emails to a separate directory in your
vmail spool for later processing by the scan_reported_mail.sh script
The imapsieve_copy is inserted into the sieve processing by modififying the
normal report-spam and report-ham sieves.
The scan_reported_mails.sh then calls the normal rspamd-pipe-ham and
rspamd-pipe-spam tools, respectively, when run. This script is written as a
singleton, ie if a second invocation is made when the first is still in process,
then it will immediately exit so as to not conflict with the first.
When scan_reported_mails.sh is run it will move all currently reported mails
into its processing directory, process them, and then cleanup. The next time it
is run, by default every minute, then it will process all new messages which
have been reported in the meantime.
This version of imapsieve_copy was developed by [Stuart Espey](mailto:stux+imapsieve_copy%20at%20mactrix.com),
based on the original version by Zhang Huangbin,
Original version: IRedMail Imapsieve
Installation
Clone the imapsieve_copy repository into your mailcow’s data/conf/dovecot
directory,
cd data/conf/dovecot
git clone https://github.com/mrstux/imapsieve_copy.git
then ensure that the permissions are correct:
:/opt/mailcow-dockerized# ls -al data/conf/dovecot/imapsieve_copy/
total 28
drwxr-xr-x 2 root root 4096 Aug 12 13:05 .
drwxr-xr-x 6 root root 4096 Aug 12 12:31 ..
-rwxr-xr-x 1 root root 895 Aug 12 12:44 imapsieve_copy
-rw-r--r-- 1 root root 1024 Aug 12 12:58 README.md
-rw-r--r-- 1 root root 573 Aug 12 12:15 report-ham.sieve
-rw-r--r-- 1 root root 471 Aug 12 12:16 report-spam.sieve
-rwxr-xr-x 1 root root 3137 Aug 12 12:06 scan_reported_mails.sh
Then add the following to your docker-compose.override.yml:
dovecot-mailcow:
volumes:
# install imapsieve_copy
- ./data/conf/dovecot/imapsieve_copy/report-ham.sieve:/usr/lib/dovecot/sieve/report-ham.sieve
- ./data/conf/dovecot/imapsieve_copy/report-spam.sieve:/usr/lib/dovecot/sieve/report-spam.sieve
- ./data/conf/dovecot/imapsieve_copy/imapsieve_copy:/usr/lib/dovecot/sieve/imapsieve_copy
- ./data/conf/dovecot/imapsieve_copy/scan_reported_mails.sh:/usr/lib/dovecot/scan_reported_mails.sh
labels:
# and schedule it
- "ofelia.job-exec.dovecot_scan_reported_emails.schedule=@every 1m"
- "ofelia.job-exec.dovecot_scan_reported_emails.command=/usr/lib/dovecot/scan_reported_mails.sh"
- "ofelia.job-exec.dovecot_scan_reported_emails.tty=false"
- "ofelia.job-exec.dovecot_scan_reported_emails.no-overlap=true"
This will mount the imapsieve_copy components over the top of the original imap
sieve components in mailcow.
You will need to restart dovecot and ofelia. Dovecot needs a full restart to
re-compile the sieves. Ofelia needs a restart to install the job scheduler which
will scan the mails.
docker-compose up -d ; docker-compose restart ofelia-mailcow
Dovecot.conf changes
No changes are actually needed, but the original author suggests adding APPEND
to the imapsieve_mailbox1_causes property, as this should catch Outlook moves.
Auto learn spam/ham with Dovecot and Outlook
# From elsewhere to Spam folder
imapsieve_mailbox1_name = Junk
imapsieve_mailbox1_causes = COPY APPEND
imapsieve_mailbox1_before = file:/usr/lib/dovecot/sieve/report-spam.sieve
# END
Testing
If you want, you can execute the scan_reported_mails.sh anyway you want,
including manually while testing.
I found it useful to modify scan_reported_mails.sh to utilize different spam/ham
incoming directories, and to use cp -at instead of mv -t while testing, as
this prevented removal of incoming spam/ham.
Logs
scan_reported_mails.sh outputs basic information to the ofelia log, and more
detailed information including rspamd status to the dovecot log imapsieve_copy
logs to the dovecot log when an email is copied.
Directories
By default imapsieve_copy creates an imapsieve_copy directory in the root of
your vmail spool with ham and spam subdirectories. scan_reported_mails.sh
creates a temporary processing directory in this imapsieve_copy directory,
with spam and ham subdirectories. These directories are not on the /tmp fs
to prevent losing messages in the event of a restart.