netfilter logs not showing authentication events

mikeyjoel

Hi Mailcow Community!
I’ve been reading, doing some homework and looking through some netfilter discussions here on the forum before deciding to post on the discussions, especially this one: https://community.mailcow.email/d/3230-redis-error-since-update-to-2024-01d/34

I’m running compose V2 with no alias, so I don’t have any issues that initially prompted that discussion.

The issue I’m running into is the following, which was noticed this morning after updating to the following version: mailcow/mailcow-dockerizedreleases/tag/2024-06

Note that I just noticed that I have not been actively monitoring or setting up an SIEM for this.
The host has been actively patched at the OS level and mailcow application layer without issues.

The main goal is to ensure authentication failure events are logged in by netfilter for auditing.
Emails receiving and sending are working with no issues at all, same for calendar, contacts, sogo, etc.

!! Please let me know if you need any more information !! More than happy to provide.

System information (I plan to migrate to SLES in the future. Understand this is not LTS or officially supported.)

OS: Fedora 
Kernel: x86_64 Linux 6.9.5-200.fc40.x86_64
Uptime: 5m
Packages: 1346
Shell: fish 3.7.0
Disk: 7.8T / 12T (68%)
CPU: Intel Core i5-4278U @ 4x 3.1GHz [77.0°C]
RAM: 4984MiB / 27810MiB

fail2ban daemon running (checked status to directly with the fail2ban-client binary)

# systemctl status fail2ban.service
● fail2ban.service - Fail2Ban Service
     Loaded: loaded (/usr/lib/systemd/system/fail2ban.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (running) since Thu 2024-06-27 09:27:50 EDT; 22s ago
       Docs: man:fail2ban(1)
    Process: 1248 ExecStartPre=/bin/mkdir -p /run/fail2ban (code=exited, status=0/SUCCESS)
   Main PID: 1256 (fail2ban-server)
      Tasks: 3 (limit: 9327)
     Memory: 13.9M (peak: 14.4M)
        CPU: 159ms
     CGroup: /system.slice/fail2ban.service
             └─1256 /usr/bin/python3 -sP /usr/bin/fail2ban-server -xf start

# fail2ban-client status
Status
|- Number of jail:      0
`- Jail list:

last reboots

# last reboot
reboot   system boot  6.9.5-200.fc40.x Thu Jun 27 09:19   still running
reboot   system boot  6.9.5-200.fc40.x Wed Jun 26 11:15 - 09:18  (22:03)
reboot   system boot  6.8.11-300.fc40. Sun Jun 23 22:54 - 11:15 (2+12:21)

Mailcow UI fail2ban parameters (I believe these are all still set to default)

netfilter logs (tried from different IP’s to fail login and no events are showing up)

netfilter-mailcow-1  | Clearing all bans
netfilter-mailcow-1  | Error reading log line from pubsub: Connection closed by server.
netfilter-mailcow-1  | Failed logging to redis: Error while reading from 172.22.1.249:6379 : (104, 'Connection reset by peer')
netfilter-mailcow-1  | Failed logging to redis: Error 111 connecting to 172.22.1.249:6379. Connection refused.
netfilter-mailcow-1  | Clear completed: ip
netfilter-mailcow-1  | Failed logging to redis: Error 111 connecting to 172.22.1.249:6379. Connection refused.
netfilter-mailcow-1  | Clear completed: ip6
netfilter-mailcow-1  | Failed logging to redis: Error 111 connecting to 172.22.1.249:6379. Connection refused.
netfilter-mailcow-1  | Error clearing redis keys F2B_ACTIVE_BANS and F2B_PERM_BANS: Error 111 connecting to 172.22.1.249:6379. Connection refused.
netfilter-mailcow-1  | Failed logging to redis: Error 111 connecting to 172.22.1.249:6379. Connection refused.
netfilter-mailcow-1  | # Warning: table ip nat is managed by iptables-nft, do not touch!
netfilter-mailcow-1  | # Warning: table ip filter is managed by iptables-nft, do not touch!
netfilter-mailcow-1  | # Warning: table ip6 nat is managed by iptables-nft, do not touch!
netfilter-mailcow-1  | # Warning: table ip6 filter is managed by iptables-nft, do not touch!
netfilter-mailcow-1  | Using NFTables backend
netfilter-mailcow-1  | Clearing all bans
netfilter-mailcow-1  | Initializing mailcow netfilter chain
netfilter-mailcow-1  | MAILCOW ip chain created successfully.
netfilter-mailcow-1  | MAILCOW ip6 chain created successfully.
netfilter-mailcow-1  | Setting MAILCOW isolation
netfilter-mailcow-1  | Error 111 connecting to 172.22.1.249:6379. Connection refused. - trying again in 3 seconds
netfilter-mailcow-1  | Watching Redis channel F2B_CHANNEL

redis (I’m not so familiar with redis at the moment but followed some basic instructions from the official doc and this is the output)

# docker compose exec redis-mailcow redis-cli
127.0.0.1:6379> monitor
OK
1719495643.033016 [0 172.22.1.5:35368] "LPUSH" "WATCHDOG_LOG" "{\"time\":\"1719495643\",\"service\":\"Unbound\",\"lvl\":\"100\",\"hpnow\":\"5\",\"hptotal\":\"5\",\"hpdiff\":\"0\"}"
1719495644.342624 [0 172.22.1.3:50270] "QUIT"
1719495645.057052 [0 172.22.1.3:44438] "QUIT"
1719495646.673328 [0 172.22.1.3:44454] "QUIT"
1719495646.730512 [0 172.22.1.3:44434] "QUIT"
1719495647.059463 [0 172.22.1.1:36698] "GET" "F2B_OPTIONS"
1719495647.060265 [0 172.22.1.1:36698] "GET" "F2B_OPTIONS"
1719495647.061546 [0 172.22.1.1:36698] "SET" "F2B_OPTIONS" "{\"ban_time\": 1800, \"ban_time_increment\": true, \"max_ban_time\": 10000, \"netban_ipv4\": 32, \"netban_ipv6\": 128, \"max_attempts\": 10, \"retry_window\": 600, \"banlist_id\": \"efec9052-d339-4342-bbbf-40738e9ab138\", \"manage_external\": 0}"
1719495647.062598 [0 172.22.1.1:36698] "HGETALL" "F2B_QUEUE_UNBAN"
1719495647.193943 [0 172.22.1.5:35370] "LPUSH" "WATCHDOG_LOG" "{\"time\":\"1719495647\",\"service\":\"Olefy\",\"lvl\":\"100\",\"hpnow\":\"5\",\"hptotal\":\"5\",\"hpdiff\":\"0\"}"
1719495647.237506 [0 172.22.1.248:40018] "LPUSH" "SOGO_LOG" "{\"time\":\"1719495647\",\"program\":\"sogod\",\"priority\":\"notice\",\"message\":\"[66]: mailcowdockerized-watchdog-mailcow-1.mailcowdockerized_mailcow-network \\\"GET /SOGo.index/ HTTP/1.1\\\" 200 2581/0 0.003 - - 0 - 13\"}\n"
1719495647.237851 [0 172.22.1.248:40020] "PUBLISH" "F2B_CHANNEL" "[66]: mailcowdockerized-watchdog-mailcow-1.mailcowdockerized_mailcow-network \"GET _SOGo.index_ HTTP_1.1\" 200 2581_0 0.003 - - 0 - 13"
1719495647.238375 [0 172.22.1.1:36698] "GET" "F2B_REGEX"
1719495647.238606 [0 172.22.1.1:36698] "GET" "F2B_REGEX"
1719495647.241763 [0 172.22.1.5:35382] "LPUSH" "WATCHDOG_LOG" "{\"time\":\"1719495647\",\"service\":\"SOGo\",\"lvl\":\"100\",\"hpnow\":\"3\",\"hptotal\":\"3\",\"hpdiff\":\"0\"}"
1719495647.924823 [0 172.22.1.3:50276] "QUIT"
1719495647.986447 [0 172.22.1.3:50272] "QUIT"
1719495648.383925 [0 172.22.1.5:35394] "LPUSH" "WATCHDOG_LOG" "{\"time\":\"1719495648\",\"service\":\"Mail queue\",\"lvl\":\"100\",\"hpnow\":\"20\",\"hptotal\":\"20\",\"hpdiff\":\"0\"}"
1719495649.144331 [0 172.22.1.3:53924] "SSCAN" "BAYES_SPAM_keys" "0" "COUNT" "1000"
1719495649.144818 [0 172.22.1.3:53924] "HGET" "RS" "learns_spam"
1719495649.299995 [0 172.22.1.1:52634] "PING"
1719495649.300544 [0 172.22.1.1:52634] "INFO"
1719495649.301415 [0 172.22.1.1:52634] "PING"
1719495652.076427 [0 172.22.1.5:41830] "LPUSH" "WATCHDOG_LOG" "{\"time\":\"1719495652\",\"service\":\"MySQL/MariaDB\",\"lvl\":\"100\",\"hpnow\":\"5\",\"hptotal\":\"5\",\"hpdiff\":\"0\"}"
1719495654.200649 [0 172.22.1.5:41832] "HKEYS" "F2B_ACTIVE_BANS"
1719495654.209365 [0 172.22.1.5:41844] "LPUSH" "WATCHDOG_LOG" "{\"time\":\"1719495654\",\"service\":\"Fail2ban\",\"lvl\":\"100\",\"hpnow\":\"1\",\"hptotal\":\"1\",\"hpdiff\":\"0\"}"
1719495654.478720 [0 172.22.1.12:43446] "HGETALL" "WHITELISTED_FWD_HOST"
1719495656.143274 [0 172.22.1.248:40020] "PUBLISH" "F2B_CHANNEL" "[8]: [WARN] <0x0x55c3181df790[WOWatchDogChild]> pid 79 has been hanging in the same request for 1 minutes"
1719495656.143643 [0 172.22.1.248:40018] "LPUSH" "SOGO_LOG" "{\"time\":\"1719495656\",\"program\":\"sogod\",\"priority\":\"notice\",\"message\":\"[8]: [WARN] <0x0x55c3181df790[WOWatchDogChild]> pid 79 has been hanging in the same request for 1 minutes\"}\n"
1719495656.144169 [0 172.22.1.1:36698] "GET" "F2B_REGEX"
1719495656.144683 [0 172.22.1.1:36698] "GET" "F2B_REGEX"
1719495657.063781 [0 172.22.1.1:36698] "GET" "F2B_OPTIONS"
1719495657.064617 [0 172.22.1.1:36698] "GET" "F2B_OPTIONS"
1719495657.065750 [0 172.22.1.1:36698] "SET" "F2B_OPTIONS" "{\"ban_time\": 1800, \"ban_time_increment\": true, \"max_ban_time\": 10000, \"netban_ipv4\": 32, \"netban_ipv6\": 128, \"max_attempts\": 10, \"retry_window\": 600, \"banlist_id\": \"efec9052-d339-4342-bbbf-40738e9ab138\", \"manage_external\": 0}"
1719495657.066586 [0 172.22.1.1:36698] "HGETALL" "F2B_QUEUE_UNBAN"
1719495657.315861 [0 172.22.1.5:41850] "PING"
1719495657.320468 [0 172.22.1.5:41856] "LPUSH" "WATCHDOG_LOG" "{\"time\":\"1719495657\",\"service\":\"Redis\",\"lvl\":\"100\",\"hpnow\":\"5\",\"hptotal\":\"5\",\"hpdiff\":\"0\"}"
1719495659.482101 [0 172.22.1.12:43462] "HGETALL" "WHITELISTED_FWD_HOST"
1719495660.686511 [0 172.22.1.5:44604] "LPUSH" "WATCHDOG_LOG" "{\"time\":\"1719495660\",\"service\":\"Clamd\",\"lvl\":\"100\",\"hpnow\":\"15\",\"hptotal\":\"15\",\"hpdiff\":\"0\"}"
1719495664.300896 [0 172.22.1.1:52634] "PING"
1719495664.302123 [0 172.22.1.1:52634] "INFO"
1719495664.304329 [0 172.22.1.1:52634] "PING"
1719495666.901718 [0 172.22.1.1:36698] "HGETALL" "F2B_BLACKLIST"
1719495666.908974 [0 172.22.1.1:36698] "HGETALL" "F2B_WHITELIST"
1719495667.067561 [0 172.22.1.1:36698] "GET" "F2B_OPTIONS"
1719495667.068119 [0 172.22.1.1:36698] "GET" "F2B_OPTIONS"
1719495667.068963 [0 172.22.1.1:36698] "SET" "F2B_OPTIONS" "{\"ban_time\": 1800, \"ban_time_increment\": true, \"max_ban_time\": 10000, \"netban_ipv4\": 32, \"netban_ipv6\": 128, \"max_attempts\": 10, \"retry_window\": 600, \"banlist_id\": \"efec9052-d339-4342-bbbf-40738e9ab138\", \"manage_external\": 0}"
1719495667.069751 [0 172.22.1.1:36698] "HGETALL" "F2B_QUEUE_UNBAN"
^C(27.08s)

127.0.0.1:6379>  PING
PONG

Notice how Netfilter is trying to communicate to Redis via the 249 octet, but on Redis, it seems like it’s using 248. Could this be the cause, and how can it be fixed? I really appreciate any help you can provide.

If it helps, here is a list of the stacks with all of it’s IP configurations and Ports in use currently.

If it helps, here a list of the stack from portainer with ipv4:ports

Note* mailcow was not provisioned using portainer. Official mailcow scripts and updated using the update.sh script from the opt path.

mikeyjoel

Update: A bug was submitted for the same issue I’m experiencing. Will follow that instead.

mailcow/mailcow-dockerized5879

mikeyjoel

I guess for now, how are most of you other than via netfilter - logs for authentication failures today? Sogo, Dovecot, others and/or combined? Want to get some logs ingested and have those events into SOAR for automation for the firewall to block failed logins from outside actors/attackers.

ETNyx

Hello,

Netfilter is logging only events, that are triggered by defined regexp patterns. For purpose of external log and security analysis you will need all logs, most likely,.. For this reason you can modify docker logging configuration and send all logs to external server. Check this docs https://docs.mailcow.email/post_installation/firststeps-logging/ and https://docs.docker.com/config/containers/logging/configure/

Currently I’m using fluentd driver to get logs from docker to fluent-bit https://fluentbit.io/ and from there to external server.

mikeyjoel

@ETNyx Thank you for your feedback. I will look into getting this implemented as soon as possible and log in my efforts on this discussion. Truly appreciated.