Hi, I'm trying to figure out why, once again, outbound mail is not getting signed. We have again been blackholed by outlook.com and indeed, I noticed that DKIM signatures were missing. In my case, I'm sending from a whitelisted IP (`forwarding host`) and the `spam filter` setting for that IP is [active this time](https://github.com/mailcow/mailcow-dockerized/issues/4493). However, somehow rspamd seems to confuse inbound and outbound mail. Here, `sending-domain.tld` is the sender domain and `hotmail.com` would be the recipient domain. ``` rspamd-mailcow-1 | 2024-05-13 17:05:58 #42(normal) ; symcache; rspamd_symcache_item_async_dec_full: decrease async events counter for DKIM_CHECK(220) = 1 - 1; subsystem rspamd dkim plugin (./src/plugins/dkim_check.c:1285) rspamd-mailcow-1 | 2024-05-13 17:05:58 #42(normal) ; dkim_signing; lua_dkim_tools.lua:195: mail is ineligible for signing rspamd-mailcow-1 | 2024-05-13 17:05:59 #42(normal) ; dmarc; dmarc.lua:287: got DMARC record: v=DMARC1; p=quarantine; rua=mailto:dmarc@mydomain.tld; ruf=mailto:dmarc@mydomain.tld; adkim=r; aspf=r, tld_flag=false, processed={[p] = quarantine, [rua] = mailto:dmarc@mydomain.tld, [ruf] = mailto:dmarc@mydomain.tld, [aspf] = r, [adkim] = r} rspamd-mailcow-1 | 2024-05-13 17:05:59 #42(normal) ; dmarc; dmarc.lua:376: validate DMARC policy (final=true): {[raw_elts] = {[p] = quarantine, [aspf] = r, [ruf] = mailto:dmarc@mydomain.tld, [rua] = mailto:dmarc@mydomain.tld, [adkim] = r}, [domain] = sending-domain.tld, [rua] = mailto:dmarc@mydomain.tld, [dmarc_policy] = quarantine} rspamd-mailcow-1 | 2024-05-13 17:05:59 #42(normal) ; dmarc; dmarc.lua:182: validated dmarc policy for sending-domain.tld: quarantine; dkim_ok=false, dkim_tempfail=false, spf_ok=false, spf_tempfail=false rspamd-mailcow-1 | 2024-05-13 17:05:59 #42(normal) ; arc; lua_dkim_tools.lua:193: mail was sent to us rspamd-mailcow-1 | 2024-05-13 17:05:59 #42(normal) ; arc; lua_dkim_tools.lua:378: inbound: use domain(recipient) for signature: hotmail.com rspamd-mailcow-1 | 2024-05-13 17:05:59 #42(normal) ; arc; lua_dkim_tools.lua:427: final DKIM domain: hotmail.com rspamd-mailcow-1 | 2024-05-13 17:05:59 #42(normal) ; arc; lua_dkim_tools.lua:48: add selector "dkim" using default selector rspamd-mailcow-1 | 2024-05-13 17:05:59 #42(normal) ; arc; lua_dkim_tools.lua:53: set domain to "hotmail.com" using dkim_domain rspamd-mailcow-1 | 2024-05-13 17:05:59 #42(normal) ; lua; lua_dkim_tools.lua:590: using selector prefix 'DKIM_SELECTORS' for domain 'hotmail.com' rspamd-mailcow-1 | dkim=none" ``` The code in `lua_dkim_tools.lua` that logs these two lines is this: ```lua if settings.sign_authenticated and auser then lua_util.debugm(N, task, 'user is authenticated') is_authed = true elseif (settings.sign_networks and settings.sign_networks:get_key(ip)) then is_sign_networks = true lua_util.debugm(N, task, 'mail is from address in sign_networks') elseif settings.sign_local and is_local then lua_util.debugm(N, task, 'mail is from local address') elseif settings.sign_inbound and not is_local and not auser then lua_util.debugm(N, task, 'mail was sent to us') else lua_util.debugm(N, task, 'mail is ineligible for signing') return false, {} end ``` And thus I'm also confused why both `mail is ineligible for signing` and `mail was sent to us` are logged, particularly in that specific order. Are we passing through `rspamd` twice? I have no specific settings at all, nothing that would be relevant to `rspamd` as far as I can tell: ``` root@rspamd:/# curl http://nginx:8081/settings.php settings { watchdog { priority = 10; rcpt_mime = "/null@localhost/i"; from_mime = "/watchdog@localhost/i"; apply "default" { symbols_disabled = ["HISTORY_SAVE", "ARC", "ARC_SIGNED", "DKIM", "DKIM_SIGNED", "CLAM_VIRUS"]; want_spam = yes; actions { reject = 9999.0; greylist = 9998.0; "add header" = 9997.0; } } } ham_trap { rcpt = "/^ham[+].*@sending-domain.tld$/i"; rcpt = "ham@sending-domain.tld"; priority = 9; apply "default" { symbols_enabled = ["HISTORY_SAVE"]; } symbols [ "HAM_TRAP" ] } spam_trap { rcpt = "/^spam[+].*@sending-domain.tld$/i"; rcpt = "spam@sending-domain.tld"; priority = 9; apply "default" { symbols_enabled = ["HISTORY_SAVE"]; } symbols [ "SPAM_TRAP" ] } additional_settings_1 { task:set_milter_reply{ add_headers = [ From = me@anotherdomain.tld", ], remove_headers = [ From = 0, ], change_from = "helpdesk@sending-domain.tld", } } } ``` Since I'm not authenticating to send this mail, my guess is that the `settings.sign_networks and settings.sign_networks:get_key(ip)` condition will have to match instead for that mail to be signed. I can't say I understand why that is now necessary, I seriously think this used to work just fine before. So, this is [where `sign_networks` is defined](https://github.com/mailcow/mailcow-dockerized/blob/master/data/conf/rspamd/local.d/dkim_signing.conf): ``` ./data/conf/rspamd/local.d/dkim_signing.conf:sign_networks = "/etc/rspamd/custom/dovecot_trusted.map"; ``` I couldn't find any documentation on how to customize this… so I wanted to modify `dovecot_trusted.map`… ``` root@rspamd:/# cat /etc/rspamd/custom/dovecot_trusted.map 172.22.1.250/32 fd4d:6169:6c63:6f77::c/128 ``` but that file gets overwritten, at least whenever the `rspamd` container is restarted.

I also tried adding an `rspamd settings map`, but either I don't understand how to properly format it (frankly, the [rspamd docs](https://rspamd.com/doc/configuration/settings.html#settings-structure) aren't helping) or something else is still wrong: ``` priority = high; from = "@sending-domain.tld"; apply { actions { sign = true; } } ``` It's unclear to me whether I need to use `symbols` or not

I can confirm this, and added my finding to this open issue: https://github.com/mailcow/mailcow-dockerized/issues/5814#issuecomment-2108126179

@"kwisatz"#p15755 No, I just tried it as you mentioned it.

I have manually edited `dovecot_trusted.map` and added the `/24` that corresponds to our `forwarded hosts` entries. I can confirm that this gets my mails signed: ``` rspamd-mailcow-1 | 2024-05-15 18:59:03 #42(normal) ; dkim_signing; lua_dkim_tools.lua:189: mail is from address in sign_networks rspamd-mailcow-1 | 2024-05-15 18:59:03 #42(normal) ; dkim_signing; lua_dkim_tools.lua:369: sign_networks: use domain(header) for signature: sending-domain.tld rspamd-mailcow-1 | 2024-05-15 18:59:03 #42(normal) ; dkim_signing; lua_dkim_tools.lua:427: final DKIM domain: sending-domain.tld rspamd-mailcow-1 | 2024-05-15 18:59:03 #42(normal) ; dkim_signing; lua_dkim_tools.lua:48: add selector "dkim" using default selector rspamd-mailcow-1 | 2024-05-15 18:59:03 #42(normal) ; dkim_signing; lua_dkim_tools.lua:53: set domain to "sending-domain.tld" using dkim_domain ``` However, this will currently only persist until the next restart of the `rspamd` container, given that this file is created by its `entrypoint` script: ``` (new)root@hoth:/opt/mailcow-dockerized# grep -R 'dovecot_trusted' . ./data/Dockerfiles/rspamd/docker-entrypoint.sh:echo ${DOVECOT_V4}/32 > /etc/rspamd/custom/dovecot_trusted.map ./data/Dockerfiles/rspamd/docker-entrypoint.sh: echo ${DOVECOT_V6}/128 >> /etc/rspamd/custom/dovecot_trusted.map ```

dkim_signing; lua_dkim_tools.lua:195: mail is ineligible for signing

kwisatz

Hi,

I’m trying to figure out why, once again, outbound mail is not getting signed. We have again been blackholed by outlook.com and indeed, I noticed that DKIM signatures were missing.

In my case, I’m sending from a whitelisted IP (forwarding host) and the spam filter setting for that IP is active this time.

However, somehow rspamd seems to confuse inbound and outbound mail. Here, sending-domain.tld is the sender domain and hotmail.com would be the recipient domain.

rspamd-mailcow-1  | 2024-05-13 17:05:58 #42(normal) <264f80>; symcache; rspamd_symcache_item_async_dec_full: decrease async events counter for DKIM_CHECK(220) = 1 - 1; subsystem rspamd dkim plugin (./src/plugins/dkim_check.c:1285)
rspamd-mailcow-1  | 2024-05-13 17:05:58 #42(normal) <264f80>; dkim_signing; lua_dkim_tools.lua:195: mail is ineligible for signing
rspamd-mailcow-1  | 2024-05-13 17:05:59 #42(normal) <264f80>; dmarc; dmarc.lua:287: got DMARC record: v=DMARC1; p=quarantine; rua=mailto:dmarc@mydomain.tld; ruf=mailto:dmarc@mydomain.tld; adkim=r; aspf=r, tld_flag=false, processed={[p] = quarantine, [rua] = mailto:dmarc@mydomain.tld, [ruf] = mailto:dmarc@mydomain.tld, [aspf] = r, [adkim] = r}
rspamd-mailcow-1  | 2024-05-13 17:05:59 #42(normal) <264f80>; dmarc; dmarc.lua:376: validate DMARC policy (final=true): {[raw_elts] = {[p] = quarantine, [aspf] = r, [ruf] = mailto:dmarc@mydomain.tld, [rua] = mailto:dmarc@mydomain.tld, [adkim] = r}, [domain] = sending-domain.tld, [rua] = mailto:dmarc@mydomain.tld, [dmarc_policy] = quarantine}
rspamd-mailcow-1  | 2024-05-13 17:05:59 #42(normal) <264f80>; dmarc; dmarc.lua:182: validated dmarc policy for sending-domain.tld: quarantine; dkim_ok=false, dkim_tempfail=false, spf_ok=false, spf_tempfail=false
rspamd-mailcow-1  | 2024-05-13 17:05:59 #42(normal) <264f80>; arc; lua_dkim_tools.lua:193: mail was sent to us
rspamd-mailcow-1  | 2024-05-13 17:05:59 #42(normal) <264f80>; arc; lua_dkim_tools.lua:378: inbound: use domain(recipient) for signature: hotmail.com
rspamd-mailcow-1  | 2024-05-13 17:05:59 #42(normal) <264f80>; arc; lua_dkim_tools.lua:427: final DKIM domain: hotmail.com
rspamd-mailcow-1  | 2024-05-13 17:05:59 #42(normal) <264f80>; arc; lua_dkim_tools.lua:48: add selector "dkim" using default selector
rspamd-mailcow-1  | 2024-05-13 17:05:59 #42(normal) <264f80>; arc; lua_dkim_tools.lua:53: set domain to "hotmail.com" using dkim_domain
rspamd-mailcow-1  | 2024-05-13 17:05:59 #42(normal) <264f80>; lua; lua_dkim_tools.lua:590: using selector prefix 'DKIM_SELECTORS' for domain 'hotmail.com'
rspamd-mailcow-1  | 	dkim=none"

The code in lua_dkim_tools.lua that logs these two lines is this:

  if settings.sign_authenticated and auser then
    lua_util.debugm(N, task, 'user is authenticated')
    is_authed = true
  elseif (settings.sign_networks and settings.sign_networks:get_key(ip)) then
    is_sign_networks = true
    lua_util.debugm(N, task, 'mail is from address in sign_networks')
  elseif settings.sign_local and is_local then
    lua_util.debugm(N, task, 'mail is from local address')
  elseif settings.sign_inbound and not is_local and not auser then
    lua_util.debugm(N, task, 'mail was sent to us')
  else
    lua_util.debugm(N, task, 'mail is ineligible for signing')
    return false, {}
  end

And thus I’m also confused why both mail is ineligible for signing and mail was sent to us are logged, particularly in that specific order. Are we passing through rspamd twice?

I have no specific settings at all, nothing that would be relevant to rspamd as far as I can tell:

root@rspamd:/# curl http://nginx:8081/settings.php
settings {
  watchdog {
    priority = 10;
    rcpt_mime = "/null@localhost/i";
    from_mime = "/watchdog@localhost/i";
    apply "default" {
      symbols_disabled = ["HISTORY_SAVE", "ARC", "ARC_SIGNED", "DKIM", "DKIM_SIGNED", "CLAM_VIRUS"];
      want_spam = yes;
      actions {
        reject = 9999.0;
        greylist = 9998.0;
        "add header" = 9997.0;
      }

    }
  }
  ham_trap {
    rcpt = "/^ham[+].*@sending-domain.tld$/i";
    rcpt = "ham@sending-domain.tld";
    priority = 9;
    apply "default" {
      symbols_enabled = ["HISTORY_SAVE"];
    }
    symbols [
      "HAM_TRAP"
    ]
  }

  spam_trap {
    rcpt = "/^spam[+].*@sending-domain.tld$/i";
    rcpt = "spam@sending-domain.tld";
    priority = 9;
    apply "default" {
      symbols_enabled = ["HISTORY_SAVE"];
    }
    symbols [
      "SPAM_TRAP"
    ]
  }
  additional_settings_1 {
    task:set_milter_reply{
      add_headers = [
         From = me@anotherdomain.tld",
      ],
      remove_headers = [
        From = 0,
      ],
      change_from = "helpdesk@sending-domain.tld",
    }
  }
}

Since I’m not authenticating to send this mail, my guess is that the settings.sign_networks and settings.sign_networks:get_key(ip) condition will have to match instead for that mail to be signed.

I can’t say I understand why that is now necessary, I seriously think this used to work just fine before.

So, this is where sign_networks is defined:

./data/conf/rspamd/local.d/dkim_signing.conf:sign_networks = "/etc/rspamd/custom/dovecot_trusted.map";

I couldn’t find any documentation on how to customize this… so I wanted to modify dovecot_trusted.map…

root@rspamd:/# cat /etc/rspamd/custom/dovecot_trusted.map
172.22.1.250/32
fd4d:6169:6c63:6f77::c/128

but that file gets overwritten, at least whenever the rspamd container is restarted.

kwisatz

I also tried adding an rspamd settings map, but either I don’t understand how to properly format it (frankly, the rspamd docs aren’t helping) or something else is still wrong:

priority = high;
from = "@sending-domain.tld";
apply {
  actions {
    sign = true;
  }
}

It’s unclear to me whether I need to use symbols or not

esackbauer

I can confirm this, and added my finding to this open issue:
mailcow/mailcow-dockerized5814

kwisatz

esackbauer Do you happen to know if this was changed/broken by a recent commit and if so, which one?

esackbauer

kwisatz No, I just tried it as you mentioned it.

kwisatz

I have manually edited dovecot_trusted.map and added the /24 that corresponds to our forwarded hosts entries.

I can confirm that this gets my mails signed:

rspamd-mailcow-1  | 2024-05-15 18:59:03 #42(normal) <d5cac2>; dkim_signing; lua_dkim_tools.lua:189: mail is from address in sign_networks
rspamd-mailcow-1  | 2024-05-15 18:59:03 #42(normal) <d5cac2>; dkim_signing; lua_dkim_tools.lua:369: sign_networks: use domain(header) for signature: sending-domain.tld
rspamd-mailcow-1  | 2024-05-15 18:59:03 #42(normal) <d5cac2>; dkim_signing; lua_dkim_tools.lua:427: final DKIM domain: sending-domain.tld
rspamd-mailcow-1  | 2024-05-15 18:59:03 #42(normal) <d5cac2>; dkim_signing; lua_dkim_tools.lua:48: add selector "dkim" using default selector
rspamd-mailcow-1  | 2024-05-15 18:59:03 #42(normal) <d5cac2>; dkim_signing; lua_dkim_tools.lua:53: set domain to "sending-domain.tld" using dkim_domain

However, this will currently only persist until the next restart of the rspamd container, given that this file is created by its entrypoint script:

(new)root@hoth:/opt/mailcow-dockerized# grep -R 'dovecot_trusted' .
./data/Dockerfiles/rspamd/docker-entrypoint.sh:echo ${DOVECOT_V4}/32 > /etc/rspamd/custom/dovecot_trusted.map
./data/Dockerfiles/rspamd/docker-entrypoint.sh:  echo ${DOVECOT_V6}/128 >> /etc/rspamd/custom/dovecot_trusted.map

esackbauer

kwisatz However, this will currently only persist until the next restart of the rspamd container, given that this file is created by its entrypoint script:

That is strange, because in mailcow directory under data/conf/rspamd/custom you will find a dovecot_trusted.map file which should survive restart.

kwisatz

esackbauer That’s the same file which is mounted into the rspamd container and overwritten by the docker-entrypoint.sh on each start as can be seen in my comment above.

        "Mounts": [
[…]
            {
                "Type": "bind",
                "Source": "/opt/mailcow-dockerized/data/conf/rspamd/custom",
                "Destination": "/etc/rspamd/custom",
                "Mode": "rw,z",
                "RW": true,
                "Propagation": "rprivate"
            },
[…]

I could perhaps change the mode to read-only to prevent it from being overwritten. Not sure how the entrypoint script will react to that but may be worth a try.

But in the end, I would love to get someone from the Infrastructure Company involved to hear/read what their thoughts are on signing mails sent by forwarding hosts and whether they know of any changes in the 2024-04 release that might have caused a change of behavior.

As much as I look at the commits, I can’t really identify one that would be responsible. And as I mentioned in the github ticket you linked to, for us, the behavior has changed somewhere between April 10th and April 23rd.

kwisatz

I might open another ticket on github to get more traction onto this, unless someone has a better idea.

kwisatz

Before opening a new ticket, I have quickly searched github to see if there were new tickets. This one doesn’t seem new, but related.
Only, I do not understand what seems to be the solution. The thread mentions to modify the configuration is one is using the unauthenticated relaying configuration, and also mentions setting forwarding hosts as a fix. So, all in all, the ticket is confusing me even more, but it seems that the PR that caused all this is #5812.

In the github thread linked to above, Patrick Schult (FreddieSpl0it) mentions updating the documentation if this works, but I have not seen this mentioned in the documentation yet and unfortunately, the github thread has been locked for additional comments.

In any case, if one of the maintainers reads this, the solution does seem to work and is more like the solution I was looking for, being persistent and more or less maintainable.

Mail is now signed, not for being part of sign_networks but for being part of local_addrs:

rspamd-mailcow-1  | 2024-05-26 16:33:29 #42(normal) <9fdbd0>; dkim_signing; lua_dkim_tools.lua:191: mail is from local address
rspamd-mailcow-1  | 2024-05-26 16:33:29 #42(normal) <9fdbd0>; dkim_signing; lua_dkim_tools.lua:407: use domain(envelope) for signature: sending-domain.tld
rspamd-mailcow-1  | 2024-05-26 16:33:29 #42(normal) <9fdbd0>; dkim_signing; lua_dkim_tools.lua:427: final DKIM domain: sending-domain.tld
rspamd-mailcow-1  | 2024-05-26 16:33:29 #42(normal) <9fdbd0>; dkim_signing; lua_dkim_tools.lua:48: add selector "dkim" using default selector
rspamd-mailcow-1  | 2024-05-26 16:33:29 #42(normal) <9fdbd0>; dkim_signing; lua_dkim_tools.lua:53: set domain to "sending-domain.tld" using dkim_domain
rspamd-mailcow-1  | 2024-05-26 16:33:29 #42(normal) <9fdbd0>; dkim_signing; lua_dkim_tools.lua:569: found and parsed key for sending-domain.tld:dkim in Redis
rspamd-mailcow-1  | 2024-05-26 16:33:29 #42(normal) <>; dkim; rspamd_dkim_sign_key_load: got public key with length 1703 and type 4

In a nutshell:

If you have modfied mynetworks in data/conf/postfix/extra.cf, then set the same networks as local_addrs in data/conf/rspamd/local.d/options.inc. Only watch out, the syntax is not the same for postfix and rspamd:

(new)root@hoth:/opt/mailcow-dockerized# cat data/conf/rspamd/local.d/options.inc
dns {
  enable_dnssec = true;
}
map_watch_interval = 30s;
disable_monitoring = true;
# In case a task times out (like DNS lookup), soft reject the message
# instead of silently accepting the message without further processing.
soft_reject_on_timeout = true;
#local_addrs = /etc/rspamd/custom/mailcow_networks.map;
# See https://github.com/mailcow/mailcow-dockerized/issues/5826#issuecomment-2041952576
local_addrs = [127.0.0.0/8, ::ffff:127.0.0.0/104, ::1/128, fe80::/10, 172.22.1.0/24, fd4d:6169:6c63:6f77::/64, 10.24.0.0/24, 192.168.122.0/24];
(new)root@hoth:/opt/mailcow-dockerized# cat data/conf/postfix/extra.cf
myhostname = mail.sending-domain.tld
# Note: 172.22.1.0/24 and [fd4d:6169:6c63:6f77::]/64 is the mailcow bridge interface and MUST be included!
# See https://docs.mailcow.email/manual-guides/Postfix/u_e-postfix-unauthenticated-relaying/
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128 [fe80::]/10 172.22.1.0/24 [fd4d:6169:6c63:6f77::]/64 10.24.0.0/16 192.168.122.0/24

esackbauer

@kwisatz I just tried this with modifying options.inc but it still does not work for me.
I did docker compose up -d and also restarted rspamd, to no avail. Mails sent out are still with dkim=none (message not signed)

OMG, I forgot to go to “Forwarding Hosts” and enable spamfilter for the hosts… Since I did this, it works! Yay, finally!

spanguel

esackbauer

Thank you very much! its getting signed now:

tbh i think this should be referenced in the docs here: https://docs.mailcow.email/manual-guides/Postfix/u_e-postfix-unauthenticated-relaying/
or at least mentioned that dkim wont work on those if not setup in rspamd aswell