Hi, Since a few days (not sure if it is caused by an update) the rspamd is using 100% cpu and no new messages are coming in. [upl-image-preview url=https://community.mailcow.email/assets/files/2022-01-10/1641846175-987585-image.png] Last log lines: `rspamd-mailcow_1 | 2022-01-10 03:30:20 #1515(controller) ; main; rspamd_worker_set_limits: use system max file descriptors limit: 1024KiB cur and 1024KiB max rspamd-mailcow_1 | 2022-01-10 03:30:20 #1515(controller) ; main; rspamd_worker_set_limits: use system max core size limit: -1B cur and -1B max rspamd-mailcow_1 | 2022-01-10 03:30:20 #1515(controller) ; controller; rspamd_controller_password_sane: your normal password is not encrypted, we strongly recommend to replace it with the encrypted one rspamd-mailcow_1 | 2022-01-10 03:30:20 #1515(controller) ; controller; rspamd_controller_password_sane: enable password is not set, so you should filter controller availability by using of firewall or `secure_ip` option rspamd-mailcow_1 | 2022-01-10 03:30:21 #1515(controller) rspamd_srv_request_handler: cannot read from server pipe: Resource temporarily unavailable rspamd-mailcow_1 | 2022-01-10 03:31:06 #1515(controller) ; lua; neural.lua:544: cannot train ANN SHORT:default : Unexpected EOF rspamd-mailcow_1 | 2022-01-10 03:32:12 #1515(controller) ; lua; neural.lua:544: cannot train ANN SHORT:default : Worker has died without reply rspamd-mailcow_1 | 2022-01-10 03:32:23 #1(main) ; main; rspamd_check_termination_clause: controller process 1515 terminated abnormally with exit code 0 by signal: Killed but NOT created core file (throttled=no); core file limits: -1 current, -1 max rspamd-mailcow_1 | 2022-01-10 03:32:23 #1(main) ; main; rspamd_cld_handler: respawn process controller in lieu of terminated process with pid 1515 rspamd-mailcow_1 | 2022-01-10 03:32:25 #1(main) ; main; rspamd_fork_worker: prepare to fork process controller (0); listen on: /var/lib/rspamd/rspamd.sock mode=0666 owner=nobody rspamd-mailcow_1 | 2022-01-10 03:32:25 #1520(controller) ; main; rspamd_worker_set_limits: use system max file descriptors limit: 1024KiB cur and 1024KiB max rspamd-mailcow_1 | 2022-01-10 03:32:25 #1520(controller) ; main; rspamd_worker_set_limits: use system max core size limit: -1B cur and -1B max rspamd-mailcow_1 | 2022-01-10 03:32:25 #1520(controller) ; controller; rspamd_controller_password_sane: your normal password is not encrypted, we strongly recommend to replace it with the encrypted one rspamd-mailcow_1 | 2022-01-10 03:32:25 #1520(controller) ; controller; rspamd_controller_password_sane: enable password is not set, so you should filter controller availability by using of firewall or `secure_ip` option rspamd-mailcow_1 | 2022-01-10 03:32:26 #1520(controller) rspamd_srv_request_handler: cannot read from server pipe: Resource temporarily unavailable rspamd-mailcow_1 | 2022-01-10 03:33:11 #1520(controller) ; lua; neural.lua:544: cannot train ANN SHORT:default : Worker has died without reply rspamd-mailcow_1 | 2022-01-10 03:34:10 #1520(controller) ; lua; neural.lua:544: cannot train ANN SHORT:default : Unexpected EOF rspamd-mailcow_1 | 2022-01-10 03:34:28 #1520(controller) ; lua; neural.lua:544: cannot train ANN SHORT:default : Unexpected EOF rspamd-mailcow_1 | 2022-01-10 03:35:45 #1(main) ; main; rspamd_main_heartbeat_cb: lost 1 heartbeat from worker type controller with pid 1520, last beat on: 2022-01-10 03:35:15.53806 rspamd-mailcow_1 | 2022-01-10 03:35:50 #1(main) ; main; rspamd_main_heartbeat_cb: lost heartbeat from worker type normal with pid 42, last beat on: 2022-01-10 03:35:29.61557 (1 beats received previously) ` How can I debug and fix this?

I get a similar problem a few times a year. Usually I just reboot mailcow stack and I'm good for a few months. But it got more frequent recently. Maybe because memory requirements indeed changed lately. That is I have 4 MB of RAM and requirements were lower than that not that long ago. I noticed that today around 13h25 (1:25 PM) my host load jumped to the roof. Total CPU usage and rspam container memory usage then went up and down hill non-stop. When I checked command top around 19h tonight Rspamd was using 98-100% CPU. [upl-image-preview url=https://community.mailcow.email/assets/files/2022-03-15/1647306142-396007-capture-decran-de-2022-03-14-20-08-55.png] [upl-image-preview url=https://community.mailcow.email/assets/files/2022-03-15/1647306143-778329-capture-decran-de-2022-03-14-19-58-40.png] Here are logs of the rspamd container from 13h00 until around 19h30 when I noticed the issue. https://pastebin.com/bAY5Yx8c I so rebooted Mailcow stack and everything then settled calm. [upl-image-preview url=https://community.mailcow.email/assets/files/2022-03-15/1647306141-657556-capture-decran-de-2022-03-14-20-29-19.png] N.B. Disregard the random-numbered rspamd container that can be seen in screenshot, I had made a mistake while trying to reboot it. Any idea of what is happening? Thank you.

Rspamd 100% CPU no messages comming in

MartinTilma

Hi, Since a few days (not sure if it is caused by an update) the rspamd is using 100% cpu and no new messages are coming in.

Last log lines:

rspamd-mailcow_1 | 2022-01-10 03:30:20 #1515(controller) <6d314c>; main; rspamd_worker_set_limits: use system max file descriptors limit: 1024KiB cur and 1024KiB max rspamd-mailcow_1 | 2022-01-10 03:30:20 #1515(controller) <6d314c>; main; rspamd_worker_set_limits: use system max core size limit: -1B cur and -1B max rspamd-mailcow_1 | 2022-01-10 03:30:20 #1515(controller) <m3mgp9>; controller; rspamd_controller_password_sane: your normal password is not encrypted, we strongly recommend to replace it with the encrypted one rspamd-mailcow_1 | 2022-01-10 03:30:20 #1515(controller) <m3mgp9>; controller; rspamd_controller_password_sane: enable password is not set, so you should filter controller availability by using of firewall orsecure_ipoption rspamd-mailcow_1 | 2022-01-10 03:30:21 #1515(controller) rspamd_srv_request_handler: cannot read from server pipe: Resource temporarily unavailable rspamd-mailcow_1 | 2022-01-10 03:31:06 #1515(controller) <m3mgp9>; lua; neural.lua:544: cannot train ANN SHORT:default : Unexpected EOF rspamd-mailcow_1 | 2022-01-10 03:32:12 #1515(controller) <m3mgp9>; lua; neural.lua:544: cannot train ANN SHORT:default : Worker has died without reply rspamd-mailcow_1 | 2022-01-10 03:32:23 #1(main) <6d314c>; main; rspamd_check_termination_clause: controller process 1515 terminated abnormally with exit code 0 by signal: Killed but NOT created core file (throttled=no); core file limits: -1 current, -1 max rspamd-mailcow_1 | 2022-01-10 03:32:23 #1(main) <6d314c>; main; rspamd_cld_handler: respawn process controller in lieu of terminated process with pid 1515 rspamd-mailcow_1 | 2022-01-10 03:32:25 #1(main) <6d314c>; main; rspamd_fork_worker: prepare to fork process controller (0); listen on: /var/lib/rspamd/rspamd.sock mode=0666 owner=nobody rspamd-mailcow_1 | 2022-01-10 03:32:25 #1520(controller) <6d314c>; main; rspamd_worker_set_limits: use system max file descriptors limit: 1024KiB cur and 1024KiB max rspamd-mailcow_1 | 2022-01-10 03:32:25 #1520(controller) <6d314c>; main; rspamd_worker_set_limits: use system max core size limit: -1B cur and -1B max rspamd-mailcow_1 | 2022-01-10 03:32:25 #1520(controller) <m3mgp9>; controller; rspamd_controller_password_sane: your normal password is not encrypted, we strongly recommend to replace it with the encrypted one rspamd-mailcow_1 | 2022-01-10 03:32:25 #1520(controller) <m3mgp9>; controller; rspamd_controller_password_sane: enable password is not set, so you should filter controller availability by using of firewall orsecure_ipoption rspamd-mailcow_1 | 2022-01-10 03:32:26 #1520(controller) rspamd_srv_request_handler: cannot read from server pipe: Resource temporarily unavailable rspamd-mailcow_1 | 2022-01-10 03:33:11 #1520(controller) <m3mgp9>; lua; neural.lua:544: cannot train ANN SHORT:default : Worker has died without reply rspamd-mailcow_1 | 2022-01-10 03:34:10 #1520(controller) <m3mgp9>; lua; neural.lua:544: cannot train ANN SHORT:default : Unexpected EOF rspamd-mailcow_1 | 2022-01-10 03:34:28 #1520(controller) <m3mgp9>; lua; neural.lua:544: cannot train ANN SHORT:default : Unexpected EOF rspamd-mailcow_1 | 2022-01-10 03:35:45 #1(main) <6d314c>; main; rspamd_main_heartbeat_cb: lost 1 heartbeat from worker type controller with pid 1520, last beat on: 2022-01-10 03:35:15.53806 rspamd-mailcow_1 | 2022-01-10 03:35:50 #1(main) <6d314c>; main; rspamd_main_heartbeat_cb: lost heartbeat from worker type normal with pid 42, last beat on: 2022-01-10 03:35:29.61557 (1 beats received previously)

How can I debug and fix this?

Rib0ul0u

Hey, i’m having same issue, have you found any solution ?

MartinTilma

I did a apt-get update of rspamd inside the container. Not sure of this is the right solution but its running fine for a few days now.

MartinTilma

The problem still exists. Any hints on how to debug?

heavygale

Looks like you are out of memory 😉

D4niel

Your server only has 2 GB of RAM? The requirements are at least 6 GB for mailcow: https://mailcow.github.io/mailcow-dockerized-docs/prerequisite/prerequisite-system/

Of course the containers can’t work properly if they don’t even have enough memory.

Bishnubee

I get a similar problem a few times a year.
Usually I just reboot mailcow stack and I’m good for a few months.

But it got more frequent recently. Maybe because memory requirements indeed changed lately.
That is I have 4 MB of RAM and requirements were lower than that not that long ago.

I noticed that today around 13h25 (1:25 PM) my host load jumped to the roof.
Total CPU usage and rspam container memory usage then went up and down hill non-stop.
When I checked command top around 19h tonight Rspamd was using 98-100% CPU.

Here are logs of the rspamd container from 13h00 until around 19h30 when I noticed the issue.
https://pastebin.com/bAY5Yx8c

I so rebooted Mailcow stack and everything then settled calm.

N.B. Disregard the random-numbered rspamd container that can be seen in screenshot, I had made a mistake while trying to reboot it.

Any idea of what is happening?

Thank you.