Mailcow on 8GB RAM + swap = OOM killed

adriano

Hi, i am experiencing regular service interuptions due to the kernels OOM killer on my server. My mailcow is running on an 8GB + swap enabled machine and I allready disabled the solr-container - I am serving a only handful of (SOGo) users. the machine is running on Ubuntu 18.04 LTS and the base os as well as the mailcow is updated (and rebooted) on a regular base (./update.sh)

After the OOME only a reboot of the machine helps to get the services up and running again. In the meantime I extended the RAM from 4 to 8 GB and disabled the solr-container which did not solve the problem but might have extended the timespan between failures.

As I have read CLAMD might be the culprit (as solr is allready out of game), is there a “moo”-ish way / recommendation of restrincting the clam or others containers RAM ressources thus preventing the host kernel from producing the OOMEs? For example by using something like a memory restricition in the docker(-compose) file in combination with “oom-kill-disable” ?Docker ?

As far as I understand there are currently no restricitons?

CONTAINER ID        NAME                                    CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
39e3707b495c        mailcowdockerized_ipv6nat-mailcow_1     0.11%               7.953MiB / 7.747GiB   0.10%               0B / 0B             10.8MB / 0B         12
8c4b63591409        mailcowdockerized_rspamd-mailcow_1      0.15%               275.1MiB / 7.747GiB   3.47%               158MB / 3.99MB      30.3MB / 4.5MB      5
66e89a27532b        mailcowdockerized_netfilter-mailcow_1   0.09%               13.75MiB / 7.747GiB   0.17%               0B / 0B             11.9MB / 0B         6
50baa8e5a505        mailcowdockerized_acme-mailcow_1        0.00%               4.457MiB / 7.747GiB   0.06%               105kB / 44.2kB      12.9MB / 0B         3
b8ce6af56f47        mailcowdockerized_nginx-mailcow_1       0.00%               7.68MiB / 7.747GiB    0.10%               158MB / 148MB       6.66MB / 28.7kB     6
6f10a8eb5493        mailcowdockerized_php-fpm-mailcow_1     0.00%               44.83MiB / 7.747GiB   0.57%               8.63MB / 12.8MB     52.9MB / 16.4kB     10
07964cda6f1a        mailcowdockerized_dovecot-mailcow_1     0.97%               93.08MiB / 7.747GiB   1.17%               15.9MB / 177MB      202MB / 8.04MB      43
05cf0a439928        mailcowdockerized_postfix-mailcow_1     0.38%               41.51MiB / 7.747GiB   0.52%               2.84MB / 4.8MB      34MB / 815kB        21
922429d38930        mailcowdockerized_mysql-mailcow_1       0.05%               177.4MiB / 7.747GiB   2.24%               16.9kB / 1.37kB     282MB / 299MB       50
2f23d8a46c56        mailcowdockerized_solr-mailcow_1        0.00%               2.477MiB / 7.747GiB   0.03%               20.6kB / 1.37kB     930kB / 0B          2
9b117429861c        mailcowdockerized_clamd-mailcow_1       0.00%               969.8MiB / 7.747GiB   12.23%              17.9MB / 79kB       212MB / 20.5kB      9
90f605d27ff8        mailcowdockerized_watchdog-mailcow_1    0.00%               2.473MiB / 7.747GiB   0.03%               26.1kB / 1.37kB     2.91MB / 0B         2
e8b8b07b5e0b        mailcowdockerized_memcached-mailcow_1   0.03%               11.02MiB / 7.747GiB   0.14%               1.59MB / 4.11MB     1MB / 0B            10
78de8dfbaee3        mailcowdockerized_unbound-mailcow_1     0.02%               10.98MiB / 7.747GiB   0.14%               2.83MB / 1.88MB     6MB / 28.7kB        1
653e01691873        mailcowdockerized_olefy-mailcow_1       0.00%               9.938MiB / 7.747GiB   0.13%               25.6kB / 1.37kB     6.46MB / 0B         1
0c8dc6b5613d        mailcowdockerized_redis-mailcow_1       0.42%               23.12MiB / 7.747GiB   0.29%               15.2MB / 168MB      6.6MB / 148MB       4
3826f009b88b        mailcowdockerized_sogo-mailcow_1        0.06%               1.064GiB / 7.747GiB   13.73%              182MB / 154MB       70.6MB / 65.5kB     88
db4eb7d286f2        mailcowdockerized_dockerapi-mailcow_1   0.02%               27.43MiB / 7.747GiB   0.35%               112kB / 945kB       12.3MB / 8.19kB     2

Thank you for any ideas or help with this!
Regards
Adrian

diekuh

Hi,

The log lines (all of them) containing the oom killing would be useful (journal/syslog).

You should also monitor your memory usage with an external tool or simply use SNMP with whatever backend. You can track the RAM usage and find the exact moment it starts to eat your RAM. Then go see which process is responsible.

Check what this service/process does, check its RAM and CPU usage, check what other containers might be doing in that moment.

That’s all I can do to help. Just the usual monitoring stuff. 🙂

adriano

Hi @diekuh, thanks for your answer! Sure, will do this.

adriano

Following up this topic, I now try to limit docker container ressources but fail.
I think I have to use “memory: 1g” (and: “memory-swap: 2g”) in docker-compose.yml but when adding it it says:

root@mail0:/opt/mailcow-dockerized# docker-compose down && docker-compose up -d ERROR: The Compose file './docker-compose.yml' is invalid because: Unsupported config option for services.clamd-mailcow: 'memory'

I also followed https://mailcow.github.io/mailcow-dockerized-docs/firststeps-logging/#logging-drivers example given of using docker-compose.override.yml but the error stays.

Any help in pointing a docker noob into the right direction in implementing ressource limits?

Thanks a ton!
Best regards

adriano

Sorry, my fault. Consider this closed.
I just did not get that the compose file changed between 2 and 3.

The resources section replaces the older resource constraint options in Compose files prior to version 3 (cpu_shares, cpu_quota, cpuset, mem_limit, memswap_limit, mem_swappiness). Refer to Upgrading version 2.x to 3.x to learn about differences between version 2 and 3 of the compose-file format.

docker

Seems to work now.