redis error since update to 2024-01d

DocFraggle

maybl8 that’s correct, the “old” docker-compose version is incompatible with the latest docker compose values.
You have to use

sudo docker compose logs -f --tail=200 netfilter-mailcow

As I said WITHOUT the ‘-’ 🙂

EDIT: no, wait, what exactly did you do before the restart? What’s the value of your DOCKER_COMPOSE_VERSION variable?

maybl8

DOCKER_COMPOSE_VERSION=native

Well I thought I knew what was triggering the issue but now I am not sure.

What my process to monitor my machine is.
Start the server.
Open the web page for mailcow.
Open another desktop
Open monitor windows for postfix , dovecot, netfilter and now redis but that is new.
To see netfilter I have to run a script to reset netfilter first or I get the screen with all the code in it.
which does this:

[demo@mail mailcow-dockerized]$ sudo ./netfilter.sh
[sudo] password for demo:
insmod /lib/modules/6.6.16-1-lts/kernel/net/ipv6/netfilter/ip6_tables.ko.zst
insmod /lib/modules/6.6.16-1-lts/kernel/net/ipv6/netfilter/ip6table_filter.ko.zst
[+] Restarting 1/1
 ✔ Container mailcowdockerized-netfilter-mailcow-1  Started                                                                                                  0.8s

This is what is in the script:

#!/bin/bash

sudo modprobe -v ip6table_filter
sudo modprobe -v iptable_filter
sudo docker-compose restart netfilter-mailcow

Running that now causes the issue.
The web page errors out with the redis error.
Postfix seems to still be working at the moment but I expect it to fail at some point with a timeout error.
Sorry for the runaround on this . I am trying to systematically test it.
I now have to restart to get the web page back.
I hope this helps.

postfix just crashed

postfix-mailcow-1  | Feb  6 12:08:31 2bb078433cad postfix/master[378]: warning: process /usr/lib/postfix/sbin/postscreen pid 974 exit status 1
postfix-mailcow-1  | Feb  6 12:08:31 2bb078433cad postfix/master[378]: warning: /usr/lib/postfix/sbin/postscreen: bad command startup -- throttling
postfix-mailcow-1  | Feb  6 12:05:21 2bb078433cad syslog-ng[9]: REDIS server error, suspending; driver='d_redis_f2b_channel#0', command='PUBLISH "F2B_CHANNEL" "warning: hostname apzg-0720c-035.stretchoid.com does not resolve to address 192.241.208.64"', error='Connection reset by peer', time_reopen='60'
postfix-mailcow-1  | Feb  6 12:05:21 2bb078433cad syslog-ng[9]: Error occurred while trying to send a message, trying again; driver='d_redis_f2b_channel#0', location='/etc/syslog-ng/syslog-ng.conf:27:3', worker_index='0', retries='1', time_reopen='60', batch_size='1'
postfix-mailcow-1  | Feb  6 12:05:21 2bb078433cad syslog-ng[9]: REDIS server error, suspending; driver='d_redis_ui_log#0', command='LPUSH "POSTFIX_MAILLOG" "{\"time\":\"1707239121\",\"program\":\"postfix/submission/smtpd\",\"priority\":\"warning\",\"message\":\"warning: hostname apzg-0720c-035.stretchoid.com does not resolve to address 192.241.208.64\"}\n"', error='Connection reset by peer', time_reopen='60'
postfix-mailcow-1  | Feb  6 12:05:21 2bb078433cad syslog-ng[9]: Error occurred while trying to send a message, trying again; driver='d_redis_ui_log#0', location='/etc/syslog-ng/syslog-ng.conf:19:3', worker_index='0', retries='1', time_reopen='60', batch_size='1'
postfix-mailcow-1  | Feb  6 12:09:25 2bb078433cad postfix/submission/smtpd[982]: connect from unknown[91.215.85.17]
postfix-mailcow-1  | Feb  6 12:09:26 2bb078433cad postfix/submission/smtpd[982]: discarding EHLO keywords: CHUNKING
postfix-mailcow-1  | Feb  6 12:09:26 2bb078433cad postfix/submission/smtpd[982]: Anonymous TLS connection established from unknown[91.215.85.17]: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256
postfix-mailcow-1  | Feb  6 12:09:26 2bb078433cad postfix/submission/smtpd[982]: discarding EHLO keywords: CHUNKING
postfix-mailcow-1  | Feb  6 12:09:29 2bb078433cad postfix/submission/smtpd[982]: warning: unknown[91.215.85.17]: SASL PLAIN authentication failed: (reason unavailable), sasl_username=administracion@dccathome.com
postfix-mailcow-1  | Feb  6 12:09:29 2bb078433cad postfix/submission/smtpd[982]: disconnect from unknown[91.215.85.17] ehlo=2 starttls=1 auth=0/1 quit=1 unknown=0/1 commands=4/6
postfix-mailcow-1  | Feb  6 12:09:51 2bb078433cad syslog-ng[9]: REDIS server error during connection; driver='d_redis_ui_log#0', error='Connection timed out', time_reopen='60'
postfix-mailcow-1  | Feb  6 12:09:51 2bb078433cad syslog-ng[9]: REDIS server error during connection; driver='d_redis_f2b_channel#0', error='Connection timed out', time_reopen='60'
postfix-mailcow-1  | Feb  6 12:10:21 2bb078433cad whitelist_forwardinghosts: Look up 193.222.96.141 on whitelist, result <html>
postfix-mailcow-1  | Feb  6 12:10:21 2bb078433cad syslog-ng[9]: Error processing log message: <head><title>504 Gateway Time-out</title></head>
postfix-mailcow-1  | <body>
postfix-mailcow-1  | <center><h1>504 Gateway Time-out</h1></center>
postfix-mailcow-1  | <hr><center>nginx/1.25.3</center>
postfix-mailcow-1  | </body>
postfix-mailcow-1  | </html>
postfix-mailcow-1  | submission/smtpd[959]: improper command pipelining after CONNECT from unknown[192.241.208.64]: EHLO apzg-0720c-035\r\n>@<
postfix-mailcow-1  | <head><title>504 Gateway Time-out</title></head>
postfix-mailcow-1  | Feb  6 12:10:21 2bb078433cad syslog-ng[9]: Error processing log message: <body>
postfix-mailcow-1  | <center><h1>504 Gateway Time-out</h1></center>
postfix-mailcow-1  | <hr><center>nginx/1.25.3</center>
postfix-mailcow-1  | </body>
postfix-mailcow-1  | </html>
postfix-mailcow-1  | submission/smtpd[959]: improper command pipelining after CONNECT from unknown[192.241.208.64]: EHLO apzg-0720c-035\r\n>@<
postfix-mailcow-1  | <body>
postfix-mailcow-1  | Feb  6 12:10:21 2bb078433cad syslog-ng[9]: Error processing log message: <center><h1>504 Gateway Time-out</h1></center>
postfix-mailcow-1  | <hr><center>nginx/1.25.3</center>
postfix-mailcow-1  | </body>
postfix-mailcow-1  | </html>
postfix-mailcow-1  | submission/smtpd[959]: improper command pipelining after CONNECT from unknown[192.241.208.64]: EHLO apzg-0720c-035\r\n>@<
postfix-mailcow-1  | <center><h1>504 Gateway Time-out</h1></center>
postfix-mailcow-1  | Feb  6 12:10:21 2bb078433cad syslog-ng[9]: Error processing log message: <hr><center>nginx/1.25.3</center>
postfix-mailcow-1  | </body>
postfix-mailcow-1  | </html>
postfix-mailcow-1  | submission/smtpd[959]: improper command pipelining after CONNECT from unknown[192.241.208.64]: EHLO apzg-0720c-035\r\n>@<
postfix-mailcow-1  | <hr><center>nginx/1.25.3</center>
postfix-mailcow-1  | Feb  6 12:10:21 2bb078433cad syslog-ng[9]: Error processing log message: </body>
postfix-mailcow-1  | </html>
postfix-mailcow-1  | submission/smtpd[959]: improper command pipelining after CONNECT from unknown[192.241.208.64]: EHLO apzg-0720c-035\r\n>@<
postfix-mailcow-1  | </body>
postfix-mailcow-1  | Feb  6 12:10:21 2bb078433cad syslog-ng[9]: Error processing log message: </html>
postfix-mailcow-1  | submission/smtpd[959]: improper command pipelining after CONNECT from unknown[192.241.208.64]: EHLO apzg-0720c-035\r\n>@<
postfix-mailcow-1  | </html>

DocFraggle

You are still using “docker-compose” instead of “docker compose”!
If I wasn’t clear enough before:

“docker-compose” is the old standalone version which doesn’t work with the latest changes in some of the containers.

“docker compose” uses the native version. So change your script to:

#!/bin/bash

sudo modprobe -v ip6table_filter
sudo modprobe -v iptable_filter
sudo docker compose restart netfilter-mailcow

and give it a try (even if I still don’t understand why that is because in your case…)

maybl8

I changed it and still the same issue.

DocFraggle

Did you try restarting everything?

docker compose down
docker compose up -d

maybl8

I ran the ./netfilter script.
Which crashed the webpage and postfix.
Ran the docker compose down and up
This is what shows on the web page and postfix

What is happening?
Please do not stop the stack while we are initializing the database or do other preparations.

What is happening? - Nginx cannot connect to an upstream server or other services are not ready yet.
This is fine if mailcow was just installed or updated and can take a few minutes to complete.
Please check the logs or contact support if the error persists.

Quick debugging
Check Nginx and PHP logs:

docker compose logs --tail=200 php-fpm-mailcow nginx-mailcow
Make sure your SQL credentials in mailcow.conf (a link to .env) do fit your initialized SQL volume. If you see an access denied, you might have the wrong mailcow.conf:

source mailcow.conf ; docker compose exec mysql-mailcow mysql -u${DBUSER} -p${DBPASS} ${DBNAME}
In case of a previous failed installation, create a backup of your existing data, followed by removing all volumes and starting over (NEVER do this with a production system, it will remove ALL data):

BACKUP_LOCATION=/tmp/ ./helper-scripts/backup_and_restore.sh backup all
docker compose down --volumes ; docker compose up -d
Make sure your timezone is correct. Use "America/New_York" for example, do not use spaces. Check here for a list.


Click to learn more about getting support.

[demo@mail mailcow-dockerized]$ sudo docker compose logs -f --tail=200 postfix-mailcow
postfix-mailcow-1  | 2024-02-06 13:06:58,718 INFO Set uid to user 0 succeeded
postfix-mailcow-1  | 2024-02-06 13:06:58,719 INFO supervisord started with pid 1
postfix-mailcow-1  | 2024-02-06 13:06:59,724 INFO spawned: 'processes' with pid 8
postfix-mailcow-1  | 2024-02-06 13:06:59,728 INFO spawned: 'postfix' with pid 9
postfix-mailcow-1  | 2024-02-06 13:06:59,732 INFO spawned: 'syslog-ng' with pid 10
postfix-mailcow-1  | Uptime: 37  Threads: 7  Questions: 246  Slow queries: 0  Opens: 40  Open tables: 31  Queries per second avg: 6.648
postfix-mailcow-1  | Feb  6 13:06:59 e4b420c1ecc5 syslog-ng[10]: syslog-ng starting up; version='3.28.1'
postfix-mailcow-1  | Checking if ASN for your IP is listed for Spamhaus Bad ASN List...
postfix-mailcow-1  | 2024-02-06 13:07:01,049 INFO success: processes entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
postfix-mailcow-1  | 2024-02-06 13:07:01,049 INFO success: postfix entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
postfix-mailcow-1  | 2024-02-06 13:07:01,049 INFO success: syslog-ng entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
postfix-mailcow-1  | The AS of your IP is NOT listed as a banned AS from Spamhaus!
postfix-mailcow-1  | Using the open Spamhaus blocklists.
postfix-mailcow-1  | chown: cannot access '/usr/share/man/man1/mailq.1.gz': No such file or directory
postfix-mailcow-1  | Feb  6 13:07:04 e4b420c1ecc5 postfix/postfix-script[376]: starting the Postfix mail system
postfix-mailcow-1  | Feb  6 13:07:04 e4b420c1ecc5 postfix/master[378]: daemon started -- version 3.5.23, configuration /opt/postfix/conf
postfix-mailcow-1  | Feb  6 13:06:59 e4b420c1ecc5 syslog-ng[10]: REDIS server error during connection; driver='d_redis_f2b_channel#0', error='Connection timed out', time_reopen='60'
postfix-mailcow-1  | Feb  6 13:06:59 e4b420c1ecc5 syslog-ng[10]: REDIS server error during connection; driver='d_redis_ui_log#0', error='Connection timed out', time_reopen='60'
postfix-mailcow-1  | Feb  6 13:08:09 e4b420c1ecc5 syslog-ng[10]: REDIS server error during connection; driver='d_redis_f2b_channel#0', error='Connection timed out', time_reopen='60'
postfix-mailcow-1  | Feb  6 13:08:09 e4b420c1ecc5 syslog-ng[10]: REDIS server error during connection; driver='d_redis_ui_log#0', error='Connection timed out', time_reopen='60'
postfix-mailcow-1  | Feb  6 13:09:19 e4b420c1ecc5 syslog-ng[10]: REDIS server error during connection; driver='d_redis_ui_log#0', error='Connection timed out', time_reopen='60'
postfix-mailcow-1  | Feb  6 13:09:19 e4b420c1ecc5 syslog-ng[10]: REDIS server error during connection; driver='d_redis_f2b_channel#0', error='Connection timed out', time_reopen='60'
[code]



Thanks for sticking with me on this.
I really appreciate it.

DocFraggle

Did you wait until all containers were completely shut down before starting them up again?

maybl8

yes
I’ll try it again and post results

[demo@mail mailcow-dockerized]$ sudo docker compose down
[sudo] password for demo:
[+] Running 18/18
[+] Running 18/19cowdockerized-rspamd-mailcow-1     Removed                                                                                                     3.5s
[+] Running 18/19cowdockerized-rspamd-mailcow-1     Removed                                                                                                     3.5s
[+] Running 19/19cowdockerized-rspamd-mailcow-1     Removed                                                                                                     3.5s
 ✔ Container mailcowdockerized-rspamd-mailcow-1     Removed                                                                                                     3.5s
 ✔ Container mailcowdockerized-solr-mailcow-1       Removed                                                                                                     2.7s
 ✔ Container mailcowdockerized-watchdog-mailcow-1   Removed                                                                                                     2.2s
 ✔ Container mailcowdockerized-olefy-mailcow-1      Removed                                                                                                    11.2s
 ✔ Container mailcowdockerized-dockerapi-mailcow-1  Removed                                                                                                     4.1s
 ✔ Container mailcowdockerized-clamd-mailcow-1      Removed                                                                                                     1.4s
 ✔ Container mailcowdockerized-memcached-mailcow-1  Removed                                                                                                     3.2s
 ✔ Container mailcowdockerized-ofelia-mailcow-1     Removed                                                                                                    11.5s
 ✔ Container mailcowdockerized-postfix-mailcow-1    Removed                                                                                                     2.9s
 ✔ Container mailcowdockerized-acme-mailcow-1       Removed                                                                                                     1.9s
 ✔ Container mailcowdockerized-nginx-mailcow-1      Removed                                                                                                    10.9s
 ✔ Container mailcowdockerized-dovecot-mailcow-1    Removed                                                                                                     7.8s
 ✔ Container mailcowdockerized-php-fpm-mailcow-1    Removed                                                                                                     0.7s
 ✔ Container mailcowdockerized-sogo-mailcow-1       Removed                                                                                                    10.9s
 ✔ Container mailcowdockerized-redis-mailcow-1      Removed                                                                                                     1.3s
 ✔ Container mailcowdockerized-mysql-mailcow-1      Removed                                                                                                     1.2s
 ✔ Container mailcowdockerized-netfilter-mailcow-1  Removed                                                                                                     0.8s
 ✔ Container mailcowdockerized-unbound-mailcow-1    Removed                                                                                                     1.1s
 ✔ Network mailcowdockerized_mailcow-network        Removed                                                                                                     0.2s

I can’t get the up results pasted in here.
It just spins and nothing shows up.

Still have the issue after doing this.

DocFraggle

You can try to run the update.sh script again… if that still doesn’t help stop all containers, reboot the server and try again.

Very strange…

maybl8

Ran update.sh again.
Didn’t reboot yet.

Got this when I ran the netfilter log

[demo@mail mailcow-dockerized]$ sudo docker compose logs --tail=200  netfilter-mailcow
[sudo] password for demo:
netfilter-mailcow-1  | Using IPTables backend
netfilter-mailcow-1  | Clearing all bans
netfilter-mailcow-1  | Initializing mailcow netfilter chain
netfilter-mailcow-1  | Setting MAILCOW isolation
netfilter-mailcow-1  | Error 113 connecting to 172.22.1.249:6379. Host is unreachable. - trying again in 3 seconds
netfilter-mailcow-1  | Watching Redis channel F2B_CHANNEL

Different error but seems to be running.

maybl8

Everything is working fine right now.
I am going to mark this solved.
I am going to install this update 2024-01e.

« Previous Page