Internal error on SSL certificates when force SSL is active

[Mystery-X]

[12/2/2021] [3:03:23 PM] [SSL      ] › ✖  error     Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation  
Failed to renew certificate npm-2 with error: Some challenges have failed.
Failed to renew certificate npm-3 with error: Some challenges have failed.
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/npm-2/fullchain.pem (failure)
  /etc/letsencrypt/live/npm-3/fullchain.pem (failure)
2 renew failure(s), 0 parse failure(s)

    at ChildProcess.exithandler (node:child_process:397:12)
    at ChildProcess.emit (node:events:390:28)
    at maybeClose (node:internal/child_process:1064:16)
    at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5)
Connection Error: Error: read ECONNRESET
Connection Error: Error: read ECONNRESET
[12/2/2021] [3:54:36 PM] [SSL      ] › ℹ  info      Renewing Let'sEncrypt certificates for Cert #3: <**masked**>
[12/2/2021] [3:54:36 PM] [SSL      ] › ℹ  info      Command: certbot renew --force-renewal --config "/etc/letsencrypt.ini" --cert-name "npm-3" --preferred-challenges "dns,http" --no-random-sleep-on-renew --disable-hook-validation 
[12/2/2021] [3:54:39 PM] [Express  ] › ⚠  warning   Command failed: certbot renew --force-renewal --config "/etc/letsencrypt.ini" --cert-name "npm-3" --preferred-challenges "dns,http" --no-random-sleep-on-renew --disable-hook-validation 
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Failed to renew certificate npm-3 with error: Some challenges have failed.
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/npm-3/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)

Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

Duplicate relation "access_list" in a relation expression. You should use "a.[b, c]" instead of "[a.b, a.c]". This will cause an error in objection 2.0

When disabling the Force SSL option the renewal went flawless.

[12/2/2021] [3:56:34 PM] [SSL      ] › ℹ  info      Renewing Let'sEncrypt certificates for Cert #3: <**masked**>
[12/2/2021] [3:56:34 PM] [SSL      ] › ℹ  info      Command: certbot renew --force-renewal --config "/etc/letsencrypt.ini" --cert-name "npm-3" --preferred-challenges "dns,http" --no-random-sleep-on-renew --disable-hook-validation 
[12/2/2021] [3:56:40 PM] [SSL      ] › ℹ  info      - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/npm-3.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for <**masked**>

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations, all renewals succeeded: 
  /etc/letsencrypt/live/npm-3/fullchain.pem (success)

So to me it looks like NPM is also trying to forward the http request for cert renewal to SSL and thus it fails to complete the request.

[chaptergy]

Please provide us with the full letsencrypt logs. See #1271 (comment)

[Mystery-X]

It's not the full, but it contains the proof that it failed to access the file needed todo the verification.

2021-12-02 15:54:39,525:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/<**masked**> HTTP/1.1" 200 1353
2021-12-02 15:54:39,526:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Thu, 02 Dec 2021 15:54:39 GMT
Content-Type: application/json
Content-Length: 1353
Connection: keep-alive
Boulder-Requester: 122098528
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0101KB-iImdk_v4_E8qeaJBpzYY_-RvkALfB9wFV7ilE8Gc
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "<**masked**>"
  },
  "status": "invalid",
  "expires": "2021-12-09T15:54:37Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:connection",
        "detail": "Fetching https://<**masked**>/.well-known/acme-challenge/lKn4ocQjD6nyrS2_SZbE-Gw32s6uedE-jAo4mTYAcdY: Error getting validation data",
        "status": 400
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/<**masked**>/<**masked**>",
      "token": "lKn4ocQjD6nyrS2_SZbE-Gw32s6uedE-jAo4mTYAcdY",
      "validationRecord": [
        {
          "url": "http://<**masked**>/.well-known/acme-challenge/lKn4ocQjD6nyrS2_SZbE-Gw32s6uedE-jAo4mTYAcdY",
          "hostname": "<**masked**>",
          "port": "80",
          "addressesResolved": [
            "<**masked**>"
          ],
          "addressUsed": "<**masked**>"
        },
        {
          "url": "https://<**masked**>/.well-known/acme-challenge/lKn4ocQjD6nyrS2_SZbE-Gw32s6uedE-jAo4mTYAcdY",
          "hostname": "<**masked**>",
          "port": "443",
          "addressesResolved": [
            "<**masked**>"
          ],
          "addressUsed": "<**masked**>"
        }
      ],
      "validated": "2021-12-02T15:54:38Z"
    }
  ]
}
2021-12-02 15:54:39,526:DEBUG:acme.client:Storing nonce: <**masked**>
2021-12-02 15:54:39,526:INFO:certbot._internal.auth_handler:Challenge failed for domain <**masked**>
2021-12-02 15:54:39,526:INFO:certbot._internal.auth_handler:http-01 challenge for <**masked**>
2021-12-02 15:54:39,526:DEBUG:certbot._internal.display.obj:Notifying user: 
Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: <**masked**>
  Type:   connection
  Detail: Fetching https://<**masked**>/.well-known/acme-challenge/lKn4ocQjD6nyrS2_SZbE-Gw32s6uedE-jAo4mTYAcdY: Error getting validation data

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

2021-12-02 15:54:39,527:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/auth_handler.py", line 90, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/auth_handler.py", line 178, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

2021-12-02 15:54:39,527:DEBUG:certbot._internal.error_handler:Calling registered functions
2021-12-02 15:54:39,527:INFO:certbot._internal.auth_handler:Cleaning up challenges
2021-12-02 15:54:39,527:DEBUG:certbot._internal.plugins.webroot:Removing /data/letsencrypt-acme-challenge/.well-known/acme-challenge/lKn4ocQjD6nyrS2_SZbE-Gw32s6uedE-jAo4mTYAcdY
2021-12-02 15:54:39,527:DEBUG:certbot._internal.plugins.webroot:All challenges cleaned up
2021-12-02 15:54:39,528:ERROR:certbot._internal.renewal:Failed to renew certificate npm-3 with error: Some challenges have failed.
2021-12-02 15:54:39,529:DEBUG:certbot._internal.renewal:Traceback was:
Traceback (most recent call last):
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/renewal.py", line 475, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/main.py", line 1386, in renew_cert
    renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/main.py", line 122, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/renewal.py", line 335, in renew_cert
    new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/client.py", line 389, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/client.py", line 439, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort)
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/auth_handler.py", line 90, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/auth_handler.py", line 178, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

2021-12-02 15:54:39,529:DEBUG:certbot._internal.display.obj:Notifying user: 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2021-12-02 15:54:39,529:ERROR:certbot._internal.renewal:All renewals failed. The following certificates could not be renewed:
2021-12-02 15:54:39,529:ERROR:certbot._internal.renewal:  /etc/letsencrypt/live/npm-3/fullchain.pem (failure)
2021-12-02 15:54:39,529:DEBUG:certbot._internal.display.obj:Notifying user: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2021-12-02 15:54:39,529:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/opt/certbot/bin/certbot", line 8, in <module>
    sys.exit(main())
  File "/opt/certbot/lib/python3.7/site-packages/certbot/main.py", line 15, in main
    return internal_main.main(cli_args)
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/main.py", line 1574, in main
    return config.func(config, plugins)
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/main.py", line 1460, in renew
    renewal.handle_renewal_request(config)
  File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/renewal.py", line 501, in handle_renewal_request
    len(renew_failures), len(parse_failures)))
certbot.errors.Error: 1 renew failure(s), 0 parse failure(s)
2021-12-02 15:54:39,530:ERROR:certbot._internal.log:1 renew failure(s), 0 parse failure(s)

[chaptergy]

Are you using cloudflare? Does the same error occur if you disable cloudflare?

[Mystery-X]

No there is no cloudflare.
But due to your question I think I start to have an idea what's going on...
NPM is serving this website for internal use only on port 443, I've only opened port 80 to the outside because I was hopeing this was enought (like certbot) to fetch an SSL cert.
But I guess if you enable "Force SSL" it doesn't care if the traffic is going to /.well-known/acme-challenge or not, but instead redirects it always to the SSL port.

[Strugglechen1337]

Hello, i get this if i try to make a new certificate for my nginx proxy manager proxy host

Error: Command failed: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-14" --agree-tos --authenticator webroot --email "" --preferred-challenges "dns,http" --domains ""
Saving debug log to /var/log/letsencrypt/letsencrypt.log
An unexpected error occurred:
There were too many requests of a given type :: Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

at ChildProcess.exithandler (child_process.js:308:12)
at ChildProcess.emit (events.js:314:20)
at maybeClose (internal/child_process.js:1022:16)
at Process.ChildProcess._handle.onexit (internal/child_process.js:287:5)

can someone help me?
I use nginx proxy manager as docker version on unraid

[robertorubioguardia]

Hi,

Same here, but not just when force SSL is active but all the time. Can't generate nor renew SSL certificates.

Any help will be gratefully thanked.

app_1  | [12/9/2021] [9:12:17 PM] [Nginx    ] › ℹ  info      Reloading Nginx
app_1  | [12/9/2021] [9:12:17 PM] [SSL      ] › ℹ  info      Requesting Let'sEncrypt certificates for Cert #92: keylor.srhosting.net
app_1  | [12/9/2021] [9:12:17 PM] [SSL      ] › ℹ  info      Command: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-92" --agree-tos --authenticator webroot --email "soporte@servidoresrapidos.net" --preferred-challenges "dns,http" --domains "keylor.srhosting.net"
app_1  | [12/9/2021] [9:12:17 PM] [Nginx    ] › ℹ  info      Reloading Nginx
app_1  | [12/9/2021] [9:12:18 PM] [Express  ] › ⚠  warning   Command failed: certbot certonly --config "/etc/letsencrypt.ini" --cert-name "npm-92" --agree-tos --authenticator webroot --email "soporte@servidoresrapidos.net" --preferred-challenges "dns,http" --domains "keylor.srhosting.net"
app_1  | Another instance of Certbot is already running.
app_1  | Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /tmp/tmpyddaiksx/log or re-run Certbot with -v for more details.

[the1ts]

I don''t believe that force SSL is pushing /well-known/acme-challenge to SSL. I'm able to get the configured 404 error when hitting that path on HTTP as is done by the letsencrypt-acme-challenge.conf, any path outside that does redirect to SSL.

It may look like its forcing that URL to SSL if HSTS is turned on and your browser caches that first. This would not be the case for letsencrypt hitting your website for the challenge since its not designed for SSL communications but just plain HTTP so would ignore the HSTS header leaving it on the HTTP connection.

[erdoukki]

Same for me (at first)...!
I have checked twice all the Firewall / router redirection to my docker NPM / NextCloud...
I have now the check availability working (and green)...
But too much try on certificate renewal make it postpone... will try later

[Schlumpf9]

I have the same problem. When turning on force SSL then Certbot is not able to renew the certificate:

2022-04-26 06:56:14,572:INFO:certbot._internal.auth_handler:Challenge failed for domain XY
2022-04-26 06:56:14,572:INFO:certbot._internal.auth_handler:http-01 challenge for XY
2022-04-26 06:56:14,572:DEBUG:certbot._internal.display.obj:Notifying user:
Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: XY
Type: connection
Detail: IP: Fetching https://XY/well-known/acme-challenge/lqC8CqFhvzDci89waVFP_4-GgUWqqh273mA6Plv5naI: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

2022-04-26 06:56:14,572:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/auth_handler.py", line 105, in handle_authorizations
self._poll_authorizations(authzrs, max_retries, best_effort)
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/auth_handler.py", line 205, in _poll_authorizations
raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

2022-04-26 06:56:14,572:DEBUG:certbot._internal.error_handler:Calling registered functions
2022-04-26 06:56:14,572:INFO:certbot._internal.auth_handler:Cleaning up challenges
2022-04-26 06:56:14,572:DEBUG:certbot._internal.plugins.webroot:Removing /data/letsencrypt-acme-challenge/.well-known/acme-challenge/lqC8CqFhvzDci89waVFP_4-GgUWqqh273mA6Plv5naI
2022-04-26 06:56:14,573:DEBUG:certbot._internal.plugins.webroot:All challenges cleaned up
2022-04-26 06:56:14,573:ERROR:certbot._internal.renewal:Failed to renew certificate npm-9 with error: Some challenges have failed.
2022-04-26 06:56:14,573:DEBUG:certbot._internal.renewal:Traceback was:
Traceback (most recent call last):
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/renewal.py", line 485, in handle_renewal_request
main.renew_cert(lineage_config, plugins, renewal_candidate)
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/main.py", line 1441, in renew_cert
renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/main.py", line 127, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/renewal.py", line 345, in renew_cert
new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/client.py", line 424, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/client.py", line 476, in _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort)
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/auth_handler.py", line 105, in handle_authorizations
self._poll_authorizations(authzrs, max_retries, best_effort)
File "/opt/certbot/lib/python3.7/site-packages/certbot/_internal/auth_handler.py", line 205, in _poll_authorizations
raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

If i connect to the container and try to curl https://XY/well-known/acme-challenge/lqC8CqFhvzDci89waVFP_4-GgUWqqh273mA6Plv5naI I receive a 404 error so there is no firewall issue there. Requesting http will response with a redirect 301. If i turn off force SSL for the specific domain and try to renew the certificate everything works. So i can definitely agree that forcing SSL prevents certbot from cert renewal... Really annoying -.-

[AtryFox]

I have the same issue here, exactly as described above. As soon as I disable "Force SSL", renewing my certificates works without issues. The renew mechanism should disable "Force SSL" temporarily or add the /well-known/acme-challange/... path as a default rule where SSL is not forced.

[the1ts]

I did notice one difference in config over time. The include of force-ssl.conf is in the server section for newly created hosts, but in the location / section for older hosts. I can break currently working proxy hosts by moving the force-ssl.conf include into the server section, outside the location / section.
This change was in #1017, which fixes the custom locations ignoring the force-ssl.conf but appears to override the specific letsencrypt exception to force-ssl.
Therefore, I think the test for redirect needs to test both $scheme = "http" and not contains /.well-known/acme-challenge/
As you can't do multiple conditions in one if or nest them, I think this can be done with setting a variable on $scheme = http to H and concatenating a D to the same variable if outside /.well-known/acme-challenge/ so only do the return 301 if the variable = HD.

So we would have:

1. HTTP and letsencrypt ("H") don't redirect
2. HTTP and not letsencrypt ("HD") redirect
3. HTTPS and letsencrypt ("") don't redirect (already HTTPS)
4. HTTPS and not letsencrypt ("D") don't redirect (already HTTPS)

Guessing here, but we don't see this issues at first creation since the default_host is hit until the cert is obtained and the proxy_host config is written and nginx HUP'd.

[n0bbi]

Same here, if "Force SSL" is enabled, i'm not able to renew the letsencrypt-certificate.