Complete crash when requesting a second wild-card cert from GoDaddy with DNS

[JohnGalt1717]

Checklist

• Have you pulled and found the error with jc21/nginx-proxy-manager:latest docker image?
  Yes
• Are you sure you're not using someone else's docker image?
  Yes
• Have you searched for similar issues (both open and closed)?
  Yes - Sort of the same exists but this gives explicit steps and the last one was closed with no repro

Describe the bug
If you try and add a second wild card cert from the SSL tab using go-daddy (not sure if it does this with others) you'll get an internal error about an npm folder in /letsencrypt/live not existing. Anything else you try and do in the session will error although the existing proxies will continue to function. If you restart the container, it will crash on boot. The only way to work around is to copy one of the other npm folders into the one it's looking for in the log and then it will start.

Nginx Proxy Manager Version
2.9.7

To Reproduce
Steps to reproduce the behavior:

1. Go to SSL
2. Click on Add SSL Cert
3. Add wildcard (*.example.com) and choose go daddy and fill in secret and key click create.
4. Add a second with the same information for a different domain (i.e. *.example2.com), click create => error as described.

Expected behavior
Should add the second certificate without error and not bork nginx manager entirely.

Operating System
Debian Linux

Additional context
❯ /data/nginx/redirection_host/1.conf
nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/npm-8/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/npm-8/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)

[chaptergy]

Could you please provide the letsencrypt logs (see #1271 (comment))

[JohnGalt1717]

I don't think it's in there (because I "fixed" it by copying the directory in) but the one that failed to create the directory etc was npm-8:

2021-12-03 09:39:48,732:DEBUG:certbot._internal.main:certbot version: 1.17.0
2021-12-03 09:39:48,735:DEBUG:certbot._internal.main:Location of certbot entry point: /opt/certbot/bin/certbot
2021-12-03 09:39:48,735:DEBUG:certbot._internal.main:Arguments: ['--non-interactive', '--quiet', '--config', '/etc/letsencrypt.ini', '--preferred-challenges', 'dns,http', '--disable-hook-validation']
2021-12-03 09:39:48,735:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#certbot-dns-godaddy:dns-godaddy,PluginEntryPoint#dns-godaddy,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2021-12-03 09:39:48,884:DEBUG:certbot._internal.log:Root logging level set at 40
2021-12-03 09:39:48,887:DEBUG:certbot.display.util:Notifying user: Processing /etc/letsencrypt/renewal/npm-11.conf
2021-12-03 09:39:49,014:DEBUG:certbot._internal.plugins.selection:Requested authenticator <certbot._internal.cli.cli_utils._Default object at 0x7fb4eeaac8> and installer <certbot._internal.cli.cli_utils._Default object at 0x7fb4eeaac8>
2021-12-03 09:39:49,014:DEBUG:certbot._internal.cli:Var pref_challs=dns,http (set by user).
2021-12-03 09:39:49,014:DEBUG:certbot._internal.cli:Var key_type=ecdsa (set by user).
2021-12-03 09:39:49,015:DEBUG:certbot._internal.cli:Var elliptic_curve=secp384r1 (set by user).
2021-12-03 09:39:49,015:DEBUG:certbot._internal.cli:Var webroot_path=/data/letsencrypt-acme-challenge (set by user).
2021-12-03 09:39:49,015:DEBUG:certbot._internal.cli:Var webroot_map={'webroot_path'} (set by user).
2021-12-03 09:39:49,015:DEBUG:certbot._internal.cli:Var webroot_path=/data/letsencrypt-acme-challenge (set by user).
2021-12-03 09:39:49,120:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): r3.o.lencr.org:80
2021-12-03 09:39:49,316:DEBUG:urllib3.connectionpool:http://r3.o.lencr.org:80 "POST / HTTP/1.1" 200 503
2021-12-03 09:39:49,318:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/archive/npm-11/cert5.pem is signed by the certificate's issuer.
2021-12-03 09:39:49,326:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/archive/npm-11/cert5.pem is: OCSPCertStatus.GOOD
2021-12-03 09:39:49,341:DEBUG:certbot.display.util:Notifying user: Certificate not yet due for renewal
2021-12-03 09:39:49,344:DEBUG:certbot._internal.plugins.selection:Requested authenticator webroot and installer None
2021-12-03 09:39:49,344:DEBUG:certbot.display.util:Notifying user: Processing /etc/letsencrypt/renewal/npm-2.conf
2021-12-03 09:39:49,350:DEBUG:certbot._internal.cli:Var pref_challs=dns,http (set by user).
2021-12-03 09:39:49,351:DEBUG:certbot._internal.cli:Var key_type=ecdsa (set by user).
2021-12-03 09:39:49,351:DEBUG:certbot._internal.cli:Var elliptic_curve=secp384r1 (set by user).
2021-12-03 09:39:49,351:DEBUG:certbot._internal.cli:Var webroot_path=/data/letsencrypt-acme-challenge (set by user).
2021-12-03 09:39:49,352:DEBUG:certbot._internal.cli:Var webroot_map={'webroot_path'} (set by user).
2021-12-03 09:39:49,352:DEBUG:certbot._internal.cli:Var webroot_path=/data/letsencrypt-acme-challenge (set by user).
2021-12-03 09:39:49,478:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): r3.o.lencr.org:80
2021-12-03 09:39:49,630:DEBUG:urllib3.connectionpool:http://r3.o.lencr.org:80 "POST / HTTP/1.1" 200 503
2021-12-03 09:39:49,633:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/archive/npm-2/cert5.pem is signed by the certificate's issuer.
2021-12-03 09:39:49,635:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/archive/npm-2/cert5.pem is: OCSPCertStatus.GOOD
2021-12-03 09:39:49,638:DEBUG:certbot.display.util:Notifying user: Certificate not yet due for renewal
2021-12-03 09:39:49,640:DEBUG:certbot._internal.plugins.selection:Requested authenticator webroot and installer None
2021-12-03 09:39:49,641:DEBUG:certbot.display.util:Notifying user: Processing /etc/letsencrypt/renewal/npm-20.conf
2021-12-03 09:39:49,769:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): r3.o.lencr.org:80
2021-12-03 09:39:49,865:DEBUG:urllib3.connectionpool:http://r3.o.lencr.org:80 "POST / HTTP/1.1" 200 503
2021-12-03 09:39:49,868:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/archive/npm-20/cert1.pem is signed by the certificate's issuer.
2021-12-03 09:39:49,869:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/archive/npm-20/cert1.pem is: OCSPCertStatus.GOOD
2021-12-03 09:39:49,871:DEBUG:certbot.display.util:Notifying user: Certificate not yet due for renewal
2021-12-03 09:39:49,873:DEBUG:certbot._internal.plugins.selection:Requested authenticator dns-godaddy and installer None
2021-12-03 09:39:49,873:DEBUG:certbot.display.util:Notifying user: Processing /etc/letsencrypt/renewal/npm-3.conf
2021-12-03 09:39:49,877:DEBUG:certbot._internal.cli:Var pref_challs=dns,http (set by user).
2021-12-03 09:39:49,878:DEBUG:certbot._internal.cli:Var key_type=ecdsa (set by user).
2021-12-03 09:39:49,878:DEBUG:certbot._internal.cli:Var elliptic_curve=secp384r1 (set by user).
2021-12-03 09:39:49,878:DEBUG:certbot._internal.cli:Var webroot_path=/data/letsencrypt-acme-challenge (set by user).
2021-12-03 09:39:49,878:DEBUG:certbot._internal.cli:Var webroot_map={'webroot_path'} (set by user).
2021-12-03 09:39:49,878:DEBUG:certbot._internal.cli:Var webroot_path=/data/letsencrypt-acme-challenge (set by user).
2021-12-03 09:39:49,997:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): r3.o.lencr.org:80
2021-12-03 09:39:50,092:DEBUG:urllib3.connectionpool:http://r3.o.lencr.org:80 "POST / HTTP/1.1" 200 503
2021-12-03 09:39:50,094:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/archive/npm-3/cert5.pem is signed by the certificate's issuer.
2021-12-03 09:39:50,096:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/archive/npm-3/cert5.pem is: OCSPCertStatus.GOOD
2021-12-03 09:39:50,098:DEBUG:certbot.display.util:Notifying user: Certificate not yet due for renewal
2021-12-03 09:39:50,101:DEBUG:certbot._internal.plugins.selection:Requested authenticator webroot and installer None
2021-12-03 09:39:50,102:DEBUG:certbot.display.util:Notifying user: Processing /etc/letsencrypt/renewal/npm-4.conf
2021-12-03 09:39:50,106:DEBUG:certbot._internal.cli:Var pref_challs=dns,http (set by user).
2021-12-03 09:39:50,106:DEBUG:certbot._internal.cli:Var key_type=ecdsa (set by user).
2021-12-03 09:39:50,106:DEBUG:certbot._internal.cli:Var elliptic_curve=secp384r1 (set by user).
2021-12-03 09:39:50,106:DEBUG:certbot._internal.cli:Var webroot_path=/data/letsencrypt-acme-challenge (set by user).
2021-12-03 09:39:50,107:DEBUG:certbot._internal.cli:Var webroot_map={'webroot_path'} (set by user).
2021-12-03 09:39:50,107:DEBUG:certbot._internal.cli:Var webroot_path=/data/letsencrypt-acme-challenge (set by user).
2021-12-03 09:39:50,193:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): r3.o.lencr.org:80
2021-12-03 09:39:50,267:DEBUG:urllib3.connectionpool:http://r3.o.lencr.org:80 "POST / HTTP/1.1" 200 503
2021-12-03 09:39:50,270:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/archive/npm-4/cert5.pem is signed by the certificate's issuer.
2021-12-03 09:39:50,271:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/archive/npm-4/cert5.pem is: OCSPCertStatus.GOOD
2021-12-03 09:39:50,273:DEBUG:certbot.display.util:Notifying user: Certificate not yet due for renewal
2021-12-03 09:39:50,275:DEBUG:certbot._internal.plugins.selection:Requested authenticator webroot and installer None
2021-12-03 09:39:50,276:DEBUG:certbot.display.util:Notifying user:
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2021-12-03 09:39:50,276:DEBUG:certbot.display.util:Notifying user: The following certificates are not due for renewal yet:
2021-12-03 09:39:50,276:DEBUG:certbot.display.util:Notifying user:   /etc/letsencrypt/live/npm-11/fullchain.pem expires on 2022-03-01 (skipped)
  /etc/letsencrypt/live/npm-2/fullchain.pem expires on 2022-03-01 (skipped)
  /etc/letsencrypt/live/npm-20/fullchain.pem expires on 2022-03-03 (skipped)
  /etc/letsencrypt/live/npm-3/fullchain.pem expires on 2022-03-01 (skipped)
  /etc/letsencrypt/live/npm-4/fullchain.pem expires on 2022-03-01 (skipped)
2021-12-03 09:39:50,276:DEBUG:certbot.display.util:Notifying user: No renewals were attempted.
2021-12-03 09:39:50,276:DEBUG:certbot.display.util:Notifying user: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2021-12-03 09:39:50,277:DEBUG:certbot._internal.renewal:no renewal failures

[chaptergy]

Yeah, as you said there is nothing in there, since this is most likely not the log of when npm-8 was created. If you can replicate the issue please provide the logs of what happens when the error occurs.

[JohnGalt1717]

One would assume that it's a pretty straight forward repro though. Sorry I didn't get the logs, but it's a live system with external dependencies so after I reproduced the issue once I fixed it and put it back in production to minimize downtime.

[chaptergy]

It is much easier when you actually have a GoDaddy domain :P
Just let me know if you ever run into this issue again and have logs to help debug it.

[JohnGalt1717]

... almost certain it will happen with any DNS verification if you just do 2 separate wildcards....

[chaptergy]

Well, I am not able to reproduce it with other providers, requesting multiple wildcard certificates for the same domain, e.g. *.example.com works as expected, and restarting npm does not cause it to crash on boot.

[the1ts]

Yes, can confirm that Hetzner can have multiple wildcard certs for different domains without issues. Checked and the new cert is created fine.

[ch4ox]

I once had a similar problem where something like this happened and I had to fix paths in the database manually.

I think the steps I took were 1. creating a wildcard cert (Hetzner), 2. attaching this cert to a host, 3. deleting the cert without updating the host afterwards. A restart finally killed it for good.

Maybe something like that happened here as well?

[tree-white]

When I applied for a wildcard certificate, there was an error error, and after I tried to restart, there was a nginx: [emerg] cannot load certificate.

❯ Enabling IPV6 in hosts: /etc/nginx/conf.d
  ❯ /etc/nginx/conf.d/production.conf
  ❯ /etc/nginx/conf.d/default.conf
  ❯ /etc/nginx/conf.d/include/block-exploits.conf
  ❯ /etc/nginx/conf.d/include/force-ssl.conf
  ❯ /etc/nginx/conf.d/include/letsencrypt-acme-challenge.conf
  ❯ /etc/nginx/conf.d/include/assets.conf
  ❯ /etc/nginx/conf.d/include/ssl-ciphers.conf
  ❯ /etc/nginx/conf.d/include/proxy.conf
  ❯ /etc/nginx/conf.d/include/ip_ranges.conf
  ❯ /etc/nginx/conf.d/include/resolvers.conf
❯ Enabling IPV6 in hosts: /data/nginx
  ❯ /data/nginx/dead_host/1.conf
  ❯ /data/nginx/default_host/site.conf
  ❯ /data/nginx/proxy_host/14.conf
  ❯ /data/nginx/proxy_host/7.conf
  ❯ /data/nginx/proxy_host/13.conf
  ❯ /data/nginx/proxy_host/2.conf
  ❯ /data/nginx/proxy_host/9.conf
  ❯ /data/nginx/proxy_host/4.conf
  ❯ /data/nginx/proxy_host/11.conf
  ❯ /data/nginx/proxy_host/8.conf
  ❯ /data/nginx/proxy_host/5.conf
  ❯ /data/nginx/proxy_host/10.conf
  ❯ /data/nginx/proxy_host/12.conf
nginx: [emerg] cannot load certificate "/etc/docker/letsencrypt/live/npm-19/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/docker/letsencrypt/live/npm-19/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)

[spuxx-dev]

I'm running into the same issue. Did you find a solution @tree-white?

[spuxx-dev]

The issue was caused by a proxy_host that was assigned an ssl certificate that had been deleted. I managed to fix it by navigating to the data volume and into /nginx/proxy_host, and deleting the *.conf files that were referring to the deleted certificate.

[tree-white]

I'm running into the same issue. Did you find a solution @tree-white?

I forgot after a long time, but in the end I remembered redeployed and only applied for a wildcard certificate.