Hi, Since a few days (not sure if it is caused by an update) the rspamd is using 100% cpu and no new messages are coming in.

Last log lines:

rspamd-mailcow_1 | 2022-01-10 03:30:20 #1515(controller) <6d314c>; main; rspamd_worker_set_limits: use system max file descriptors limit: 1024KiB cur and 1024KiB max
rspamd-mailcow_1 | 2022-01-10 03:30:20 #1515(controller) <6d314c>; main; rspamd_worker_set_limits: use system max core size limit: -1B cur and -1B max
rspamd-mailcow_1 | 2022-01-10 03:30:20 #1515(controller) <m3mgp9>; controller; rspamd_controller_password_sane: your normal password is not encrypted, we strongly recommend to replace it with the encrypted one
rspamd-mailcow_1 | 2022-01-10 03:30:20 #1515(controller) <m3mgp9>; controller; rspamd_controller_password_sane: enable password is not set, so you should filter controller availability by using of firewall orsecure_ipoption
rspamd-mailcow_1 | 2022-01-10 03:30:21 #1515(controller) rspamd_srv_request_handler: cannot read from server pipe: Resource temporarily unavailable
rspamd-mailcow_1 | 2022-01-10 03:31:06 #1515(controller) <m3mgp9>; lua; neural.lua:544: cannot train ANN SHORT:default : Unexpected EOF
rspamd-mailcow_1 | 2022-01-10 03:32:12 #1515(controller) <m3mgp9>; lua; neural.lua:544: cannot train ANN SHORT:default : Worker has died without reply
rspamd-mailcow_1 | 2022-01-10 03:32:23 #1(main) <6d314c>; main; rspamd_check_termination_clause: controller process 1515 terminated abnormally with exit code 0 by signal: Killed but NOT created core file (throttled=no); core file limits: -1 current, -1 max
rspamd-mailcow_1 | 2022-01-10 03:32:23 #1(main) <6d314c>; main; rspamd_cld_handler: respawn process controller in lieu of terminated process with pid 1515
rspamd-mailcow_1 | 2022-01-10 03:32:25 #1(main) <6d314c>; main; rspamd_fork_worker: prepare to fork process controller (0); listen on: /var/lib/rspamd/rspamd.sock mode=0666 owner=nobody
rspamd-mailcow_1 | 2022-01-10 03:32:25 #1520(controller) <6d314c>; main; rspamd_worker_set_limits: use system max file descriptors limit: 1024KiB cur and 1024KiB max
rspamd-mailcow_1 | 2022-01-10 03:32:25 #1520(controller) <6d314c>; main; rspamd_worker_set_limits: use system max core size limit: -1B cur and -1B max
rspamd-mailcow_1 | 2022-01-10 03:32:25 #1520(controller) <m3mgp9>; controller; rspamd_controller_password_sane: your normal password is not encrypted, we strongly recommend to replace it with the encrypted one
rspamd-mailcow_1 | 2022-01-10 03:32:25 #1520(controller) <m3mgp9>; controller; rspamd_controller_password_sane: enable password is not set, so you should filter controller availability by using of firewall orsecure_ipoption
rspamd-mailcow_1 | 2022-01-10 03:32:26 #1520(controller) rspamd_srv_request_handler: cannot read from server pipe: Resource temporarily unavailable
rspamd-mailcow_1 | 2022-01-10 03:33:11 #1520(controller) <m3mgp9>; lua; neural.lua:544: cannot train ANN SHORT:default : Worker has died without reply
rspamd-mailcow_1 | 2022-01-10 03:34:10 #1520(controller) <m3mgp9>; lua; neural.lua:544: cannot train ANN SHORT:default : Unexpected EOF
rspamd-mailcow_1 | 2022-01-10 03:34:28 #1520(controller) <m3mgp9>; lua; neural.lua:544: cannot train ANN SHORT:default : Unexpected EOF
rspamd-mailcow_1 | 2022-01-10 03:35:45 #1(main) <6d314c>; main; rspamd_main_heartbeat_cb: lost 1 heartbeat from worker type controller with pid 1520, last beat on: 2022-01-10 03:35:15.53806
rspamd-mailcow_1 | 2022-01-10 03:35:50 #1(main) <6d314c>; main; rspamd_main_heartbeat_cb: lost heartbeat from worker type normal with pid 42, last beat on: 2022-01-10 03:35:29.61557 (1 beats received previously)

How can I debug and fix this?

7 days later

Hey, i’m having same issue, have you found any solution ?

Have something to say?

Join the community by quickly registering to participate in this discussion. We'd like to see you joining our great moo-community!

I did a apt-get update of rspamd inside the container. Not sure of this is the right solution but its running fine for a few days now.

a month later

The problem still exists. Any hints on how to debug?

15 days later

Looks like you are out of memory 😉

10 days later

I get a similar problem a few times a year.
Usually I just reboot mailcow stack and I’m good for a few months.

But it got more frequent recently. Maybe because memory requirements indeed changed lately.
That is I have 4 MB of RAM and requirements were lower than that not that long ago.

I noticed that today around 13h25 (1:25 PM) my host load jumped to the roof.
Total CPU usage and rspam container memory usage then went up and down hill non-stop.
When I checked command top around 19h tonight Rspamd was using 98-100% CPU.


Here are logs of the rspamd container from 13h00 until around 19h30 when I noticed the issue.
https://pastebin.com/bAY5Yx8c

pastebin.com
https://pastebin.com/bAY5Yx8c
Retrieving...

I so rebooted Mailcow stack and everything then settled calm.

N.B. Disregard the random-numbered rspamd container that can be seen in screenshot, I had made a mistake while trying to reboot it.

Any idea of what is happening?

Thank you.

No one is typing