Failed to get renew ACME certificates due to NGINX server configuration issue

Faster_Than_Light

Hi all,

I’ve been running a mailcow server for about 6 years now, and apart from a few blips along the way, it’s generally been a pretty smooth ride.

I mention that as I may have changed some of these files along the road and forgotten about it…

Anyway, I was notified that my SSL certificates had stopped renewing earlier today. I checked it out, and it had indeed failed.

The issue being reported by the acme container was:

Traceback (most recent call last):
  File "/usr/bin/acme-tiny", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/usr/lib/python3.12/site-packages/acme_tiny.py", line 195, in main
    signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca, disable_check=args.disable_check, directory_url=args.directory_url, contact=args.contact, check_port=args.check_port)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/acme_tiny.py", line 153, in get_crt
    raise ValueError("Challenge did not pass for {0}: {1}".format(domain, authorization))
ValueError: Challenge did not pass for autoconfig.domain-01.com: {
  'identifier': {
    'type': 'dns',
    'value': 'autoconfig.domain-01.com'
  },
  'status': 'invalid',
  'expires': '2024-07-13T10:18:02Z',
  'challenges': [
    {
      'type': 'http-01',
      'url': 'https://acme-v02.api.letsencrypt.org/acme/chall-v3/<SNIP>/<SNIP>',
      'status': 'invalid',
      'validated': '2024-07-06T10:24:14Z',
      'error': {
        'type': 'urn:ietf:params:acme:error:unauthorized',
        'detail': '<IP ADDRESS>: Invalid response from http://autoconfig.domain-01.com/.well-known/acme-challenge/<TOKEN>: "<?xml version=\\"1.0\\"?><clientConfig version=\\"1.1\\">\\n    <emailProvider id=\\"mail.domain-00.com\\">\\n      <domain>%EMAILDOMAIN%</d"',
        'status': 403
      },
      'token': '<TOKEN>',
      'validationRecord': [
        {
          'url': 'http://autoconfig.domain-01.com/.well-known/acme-challenge/<TOKEN>',
          'hostname': 'autoconfig.domain-01.com',
          'port': '80',
          'addressesResolved': ['<IP ADDRESS>'],
          'addressUsed': '<IP ADDRESS>'
        }
      ]
    }
  ]
}

Sat Jul  6 10:24:17 UTC 2024 - Failed to obtain certificate /var/lib/acme/mail.domain-00.com/cert.pem for domains 'mail.domain-00.com autoconfig.domain-01.com autoconfig.domain-02.com autoconfig.domain-00.com autodiscover.domain-01.com autodiscover.domain-02.com autodiscover.domain-00.com mail.domain-01.com mail.domain-02.com'

Which led me to look at the NGINX logs, which showed:

2024/07/06 10:17:13 [warn] 17#17: conflicting server name "autoconfig.*" on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "autoconfig.*" on 0.0.0.0:80, ignored
2024/07/06 10:17:13 [warn] 17#17: conflicting server name "mail.domain-00.com" on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "mail.domain-00.com" on 0.0.0.0:80, ignored
2024/07/06 10:17:13 [warn] 17#17: conflicting server name "autodiscover.*" on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "autodiscover.*" on 0.0.0.0:80, ignored
2024/07/06 10:17:13 [warn] 17#17: conflicting server name "autoconfig.*" on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "autoconfig.*" on 0.0.0.0:80, ignored
2024/07/06 10:17:13 [warn] 17#17: conflicting server name "autoconfig.*" on [::]:80, ignored
nginx: [warn] conflicting server name "autoconfig.*" on [::]:80, ignored
2024/07/06 10:17:13 [warn] 17#17: conflicting server name "mail.domain-00.com" on [::]:80, ignored
nginx: [warn] conflicting server name "mail.domain-00.com" on [::]:80, ignored
2024/07/06 10:17:13 [warn] 17#17: conflicting server name "autodiscover.*" on [::]:80, ignored
nginx: [warn] conflicting server name "autodiscover.*" on [::]:80, ignored
2024/07/06 10:17:13 [warn] 17#17: conflicting server name "autoconfig.*" on [::]:80, ignored
nginx: [warn] conflicting server name "autoconfig.*" on [::]:80, ignored
2024/07/06 10:17:14 [warn] 1#1: conflicting server name "autoconfig.*" on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "autoconfig.*" on 0.0.0.0:80, ignored
2024/07/06 10:17:14 [warn] 1#1: conflicting server name "mail.domain-00.com" on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "mail.domain-00.com" on 0.0.0.0:80, ignored
2024/07/06 10:17:14 [warn] 1#1: conflicting server name "autodiscover.*" on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "autodiscover.*" on 0.0.0.0:80, ignored
2024/07/06 10:17:14 [warn] 1#1: conflicting server name "autoconfig.*" on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "autoconfig.*" on 0.0.0.0:80, ignored
2024/07/06 10:17:14 [warn] 1#1: conflicting server name "autoconfig.*" on [::]:80, ignored
nginx: [warn] conflicting server name "autoconfig.*" on [::]:80, ignored
2024/07/06 10:17:14 [warn] 1#1: conflicting server name "mail.domain-00.com" on [::]:80, ignored
nginx: [warn] conflicting server name "mail.domain-00.com" on [::]:80, ignored
2024/07/06 10:17:14 [warn] 1#1: conflicting server name "autodiscover.*" on [::]:80, ignored
nginx: [warn] conflicting server name "autodiscover.*" on [::]:80, ignored
2024/07/06 10:17:14 [warn] 1#1: conflicting server name "autoconfig.*" on [::]:80, ignored
nginx: [warn] conflicting server name "autoconfig.*" on [::]:80, ignored

Which finally resulted in me modifying:

/opt/mailcow-dockerized/data/conf/nginx/80-redirect-443.conf

To comment out the first server block, so that my file looks like:

#server {
#  listen 80;
#  listen [::]:80;
#  server_name autoconfig.*;
#  root /web;
#  location / {
#    fastcgi_split_path_info ^(.+\.php)(/.+)$;
#    fastcgi_pass phpfpm:9002;
#    include /etc/nginx/fastcgi_params;
#    fastcgi_param SCRIPT_FILENAME $document_root/autoconfig.php;
#    try_files /autoconfig.php =404;
#  }
#}
server {
  listen 80 default_server;
  listen [::]:80 default_server;
  include /etc/nginx/conf.d/server_name.active;
  return 301 https://$host$uri$is_args$args;
}

This resolved the conflict, and allowed ACME and NGINX to work together, and everything is back up and running now.

A couple of questions though…

1) What are the consequences of what I’ve done?
2) Will these changes be reverted upon updating mailcow?
3) Was there a better way to fix my issue?

Thanks all. 🙂

D4niel

Faster_Than_Light 3) Was there a better way to fix my issue?

If your reverse proxy is already issuing certificates you can just pass those certificates to mailcow. No need to use the acme container.

Faster_Than_Light

D4niel I don’t think it is… I don’t remember ever setting anything up to do that, I always just relied on the mailcow docker containers to do their thing.

I’m a little confused about why this is an issue to start with. Is that 80-redirect-443.conf a non-standard file?

esackbauer

Faster_Than_Light I’m a little confused about why this is an issue to start with. Is that 80-redirect-443.conf a non-standard file?

Yes, it seems a non-standard file. You can search the github repo and you won’t find it.

D4niel

Faster_Than_Light I don’t think it is… I don’t remember ever setting anything up to do that, I always just relied on the mailcow docker containers to do their thing.

If you can access the mailcow interface on port 443 and it is behind your reverse proxy, the proxy has to have some kind of certificate.

Faster_Than_Light

esackbauer Gotcha. I %^$@ed myself then it seems. Makes sense.

Do I need it anymore? I imagine I wanted to redirect 80 links to 443 for things like webmail… Is that taken care of elsewhere in the standard NGINX setup?

esackbauer

Faster_Than_Light Is that taken care of elsewhere in the standard NGINX setup?

Yes, maybe this was changed during the last 6 years 😉
My installation is ₂ years and it does auto redirect 80 to 443

Faster_Than_Light

esackbauer In that case, just to be sure, I’ll change the filename to .conf.old at the end, so it shouldn’t be picked up by nginx automatically (if my thinking is correct).

If I then docker compose down && docker compose up -d, I’m assuming that nginx will be in a more stock configuration, and I can make sure that everything works as intended?

D4niel Yep, but mailcow should handle that for me. I can see in the stock configuration that it makes reference to the .pem and .key files in its config.