Auto SSL certificate renewal failing using AWS Route53 DNS

[coreylane]

Checklist

• Have you pulled and found the error with jc21/nginx-proxy-manager:latest docker image?
  • Yes
• Are you sure you're not using someone else's docker image?
  • Yes
• Have you searched for similar issues (both open and closed)?
  • Yes

Describe the bug
Automatic SSL certificate renewal fails when using AWS Route53 DNS challenge, but renewing manually through NPM console works fine.

Based on initial review of the logs, it looks like the automated renewal command is not setting the required "AWS_CONFIG_FILE" environment variable:

[3/8/2022] [6:02:17 AM] [SSL      ] › ✖  error     Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation  
Failed to renew certificate npm-12 with error: Unable to locate credentials

Nginx Proxy Manager Version
v2.9.16

To Reproduce
Steps to reproduce the behavior:

1. Creates Let's Encrypt certificate using AWS Route53 DNS challenge
2. As certificate expiry date nears, the renewals fail

Expected behavior
SSL certificate is renewed as the expiry date approaches

Operating System
Unraid 6.10.0-rc2
Docker version 20.10.9, build c2ea9bc

Additional context
Logs

[3/8/2022] [5:54:15 AM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...
[3/8/2022] [6:02:17 AM] [SSL      ] › ✖  error     Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation  
Failed to renew certificate npm-12 with error: Unable to locate credentials
To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
AMAll renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/npm-12/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)

   at ChildProcess.exithandler (node:child_process:399:12)
   at ChildProcess.emit (node:events:520:28)
   at maybeClose (node:internal/child_process:1092:16)
   at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)

Workaround: Renewing through the console works as expected

[3/8/2022] [9:49:46 AM] [SSL      ] › ℹ  info      Renewing Let'sEncrypt certificates via Route 53 (Amazon) for Cert #12: *.darktower.one, darktower.one
[3/8/2022] [9:49:46 AM] [SSL      ] › ℹ  info      Command: AWS_CONFIG_FILE='/etc/letsencrypt/credentials/credentials-12' certbot renew --config "/etc/letsencrypt.ini" --cert-name "npm-12" --disable-hook-validation --no-random-sleep-on-renew 
[3/8/2022] [9:50:20 AM] [SSL      ] › ℹ  info      - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/npm-12.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for *.darktower.one and darktower.one

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations, all renewals succeeded: 
  /etc/letsencrypt/live/npm-12/fullchain.pem (success)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

[chaddm]

@coreylane I am having the same issue. Would you be kind enough to share the manual command, please? I could not find it. Thank you.

[coreylane]

@chaddm Try renewing through the NPM GUI, under SSL Certificates -> Renew Now

[chaddm]

@coreylane Thank you. It worked correctly.

[nmbgeek]

Same issue. Renewing manually through GUI fixed the issue which was the renewal at 4/25/2022 1:01AM

`[4/24/2022] [10:41:18 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[4/24/2022] [10:41:19 PM] [IP Ranges] › ℹ info Fetching IP Ranges from online services...
[4/24/2022] [10:41:19 PM] [IP Ranges] › ℹ info Fetching https://ip-ranges.amazonaws.com/ip-ranges.json
[4/24/2022] [10:41:19 PM] [IP Ranges] › ℹ info Fetching https://www.cloudflare.com/ips-v4
[4/24/2022] [10:41:19 PM] [IP Ranges] › ℹ info Fetching https://www.cloudflare.com/ips-v6
[4/24/2022] [10:41:19 PM] [Nginx ] › ℹ info Reloading Nginx
[4/24/2022] [10:41:20 PM] [Setup ] › ℹ info Logrotate completed.
[4/24/2022] [10:48:23 PM] [SSL ] › ✖ error Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation
Failed to renew certificate npm-1 with error: Unable to locate credentials
To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
All renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/npm-1/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)
at ChildProcess.exithandler (node:child_process:399:12)
at ChildProcess.emit (node:events:526:28)
at maybeClose (node:internal/child_process:1092:16)
at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)
[4/24/2022] [11:41:18 PM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[4/24/2022] [11:43:16 PM] [SSL ] › ✖ error Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation
Failed to renew certificate npm-1 with error: Unable to locate credentials
To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
All renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/npm-1/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)
at ChildProcess.exithandler (node:child_process:399:12)
at ChildProcess.emit (node:events:526:28)
at maybeClose (node:internal/child_process:1092:16)
at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)
[4/25/2022] [12:41:18 AM] [SSL ] › ℹ info Renewing SSL certs close to expiry...
[4/25/2022] [12:41:49 AM] [SSL ] › ✖ error Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation
Failed to renew certificate npm-1 with error: Unable to locate credentials
To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
All renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/npm-1/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)
at ChildProcess.exithandler (node:child_process:399:12)
at ChildProcess.emit (node:events:526:28)
at maybeClose (node:internal/child_process:1092:16)
at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)
Connection Error: Error: read ECONNRESET
[4/25/2022] [1:01:25 AM] [Express ] › ⚠ warning connect ECONNREFUSED 172.16.0.3:3306
[4/25/2022] [1:01:31 AM] [SSL ] › ℹ info Renewing Let'sEncrypt certificates via Route 53 (Amazon) for Cert #1: *.NOTMYDOMAIN.com
[4/25/2022] [1:01:31 AM] [SSL ] › ℹ info Command: AWS_CONFIG_FILE='/etc/letsencrypt/credentials/credentials-1' certbot renew --config "/etc/letsencrypt.ini" --cert-name "npm-1" --disable-hook-validation --no-random-sleep-on-renew
[4/25/2022] [1:02:15 AM] [SSL ] › ℹ info - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/npm-1.conf

---

Renewing an existing certificate for *.NOTMYDOMAIN.com

---

Congratulations, all renewals succeeded:
/etc/letsencrypt/live/npm-1/fullchain.pem (success)

• • • • • • • • • • -`

[matthew-larner]

I can replicate this issue. Works manually via the GUI.

[jonathanjuursema]

I have the same issue. I'm unable to verify the logs (it's been a week or two since the last manual renewal) but symptoms are the same. Let's Encrypt via Route53 DNS, get e-mail from Let's Encrypt that the cert was not automatically renewed, manual renewal saves the day.

[emkookmer]

I have the same Issue, Automatic renewal doesn't work, manual renewal does.

Below a snippet from the docker log:

Auto renewal

2022-06-17T09:47:57.071050992Z [6/17/2022] [9:47:57 AM] [SSL      ] › ℹ  info      Renewing SSL certs close to expiry...
2022-06-17T09:57:47.308169768Z [6/17/2022] [9:57:47 AM] [SSL      ] › ✖  error     Error: Command failed: certbot renew --non-interactive --quiet --config "/etc/letsencrypt.ini" --preferred-challenges "dns,http" --disable-hook-validation  
2022-06-17T09:57:47.309667756Z Failed to renew certificate npm-10 with error: Unable to locate credentials
2022-06-17T09:57:47.309676328Z To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
2022-06-17T09:57:47.309679394Z Failed to renew certificate npm-11 with error: Unable to locate credentials
2022-06-17T09:57:47.309687161Z To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
2022-06-17T09:57:47.309689779Z Failed to renew certificate npm-12 with error: Unable to locate credentials
2022-06-17T09:57:47.309692250Z To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
2022-06-17T09:57:47.309694859Z Failed to renew certificate npm-13 with error: Unable to locate credentials
2022-06-17T09:57:47.309697339Z To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
2022-06-17T09:57:47.309699886Z Failed to renew certificate npm-14 with error: Unable to locate credentials
2022-06-17T09:57:47.309702216Z To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
2022-06-17T09:57:47.309704851Z Failed to renew certificate npm-15 with error: Unable to locate credentials
2022-06-17T09:57:47.309707201Z To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
2022-06-17T09:57:47.309712085Z Failed to renew certificate npm-2 with error: Unable to locate credentials
2022-06-17T09:57:47.309714471Z To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
2022-06-17T09:57:47.309717083Z Failed to renew certificate npm-3 with error: Unable to locate credentials
2022-06-17T09:57:47.309719389Z To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
2022-06-17T09:57:47.309722585Z Failed to renew certificate npm-4 with error: Unable to locate credentials
2022-06-17T09:57:47.309726014Z To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
2022-06-17T09:57:47.309729790Z Failed to renew certificate npm-5 with error: Unable to locate credentials
2022-06-17T09:57:47.309733391Z To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
2022-06-17T09:57:47.309737128Z Failed to renew certificate npm-6 with error: Unable to locate credentials
2022-06-17T09:57:47.309740634Z To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
2022-06-17T09:57:47.309746021Z All renewals failed. The following certificates could not be renewed:
2022-06-17T09:57:47.309748402Z   /etc/letsencrypt/live/npm-10/fullchain.pem (failure)
2022-06-17T09:57:47.309751068Z   /etc/letsencrypt/live/npm-11/fullchain.pem (failure)
2022-06-17T09:57:47.309753448Z   /etc/letsencrypt/live/npm-12/fullchain.pem (failure)
2022-06-17T09:57:47.309755686Z   /etc/letsencrypt/live/npm-13/fullchain.pem (failure)
2022-06-17T09:57:47.309758011Z   /etc/letsencrypt/live/npm-14/fullchain.pem (failure)
2022-06-17T09:57:47.309760241Z   /etc/letsencrypt/live/npm-15/fullchain.pem (failure)
2022-06-17T09:57:47.309762491Z   /etc/letsencrypt/live/npm-2/fullchain.pem (failure)
2022-06-17T09:57:47.309764835Z   /etc/letsencrypt/live/npm-3/fullchain.pem (failure)
2022-06-17T09:57:47.309767558Z   /etc/letsencrypt/live/npm-4/fullchain.pem (failure)
2022-06-17T09:57:47.309769795Z   /etc/letsencrypt/live/npm-5/fullchain.pem (failure)
2022-06-17T09:57:47.309772019Z   /etc/letsencrypt/live/npm-6/fullchain.pem (failure)
2022-06-17T09:57:47.309774393Z 11 renew failure(s), 0 parse failure(s)
2022-06-17T09:57:47.309776639Z 
2022-06-17T09:57:47.309778811Z     at ChildProcess.exithandler (node:child_process:399:12)
2022-06-17T09:57:47.309781098Z     at ChildProcess.emit (node:events:520:28)
2022-06-17T09:57:47.309783608Z     at maybeClose (node:internal/child_process:1092:16)
2022-06-17T09:57:47.309785934Z     at Process.ChildProcess._handle.onexit (node:internal/child_process:302:5)

Manual renewal

2022-06-17T11:32:30.627108035Z [6/17/2022] [11:32:30 AM] [SSL      ] › ℹ  info      Renewing Let'sEncrypt certificates via Route 53 (Amazon) for Cert #10: *.domain.tld, domain.tld
2022-06-17T11:32:30.628592566Z [6/17/2022] [11:32:30 AM] [SSL      ] › ℹ  info      Command: AWS_CONFIG_FILE='/etc/letsencrypt/credentials/credentials-10' certbot renew --config "/etc/letsencrypt.ini" --cert-name "npm-10" --disable-hook-validation --no-random-sleep-on-renew 
2022-06-17T11:33:34.021023256Z [6/17/2022] [11:33:34 AM] [SSL      ] › ℹ  info      - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2022-06-17T11:33:34.021050555Z Processing /etc/letsencrypt/renewal/npm-10.conf
2022-06-17T11:33:34.021055186Z - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2022-06-17T11:33:34.021059019Z Renewing an existing certificate for *.domain.tld and domain.tld
2022-06-17T11:33:34.021062747Z 
2022-06-17T11:33:34.021066108Z - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2022-06-17T11:33:34.021069739Z Congratulations, all renewals succeeded: 
2022-06-17T11:33:34.021072948Z   /etc/letsencrypt/live/npm-10/fullchain.pem (success)
2022-06-17T11:33:34.021076289Z - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

NOTE: I've anonymized the domain names, but the structure is the same

[matthew-larner]

I'm still experiencing this issue on the latest version. Is anyone able to fix this issue?

[coreylane]

I'm still experiencing this issue on the latest version. Is anyone able to fix this issue?

@matthew-larner Renewing through the NPM GUI works fine for me, I posted a screenshot in an earlier comment. If this doesn't work you may have an issue with your AWS credentials. Perhaps the User doesn't have required permissions to route53. What do your logs say?

Required AWS permissions:

route53:ChangeResourceRecordSets
route53:ListHostedZones
route53:GetChange

[matthew-larner]

@coreylane renewing via the GUI works fine. Automatic renewals don't though. Since I have 50+ entries it's a real pain having to renew manually.

How much effort is required to fix the auto renewals?

Thanks for you help on this.

[emkookmer]

It looks like the code use for auto renewal isnt using the same renewal functions as the UI, but a simple command to certbot

nginx-proxy-manager/backend/internal/certificate.js

Lines 42 to 106 in 14b889a

	processExpiringHosts: () => {
	if (!internalCertificate.intervalProcessing) {
	internalCertificate.intervalProcessing = true;
	logger.info('Renewing SSL certs close to expiry...');
	const cmd = certbotCommand + ' renew --non-interactive --quiet ' +
	'--config "' + letsencryptConfig + '" ' +
	'--preferred-challenges "dns,http" ' +
	'--disable-hook-validation ' +
	(letsencryptStaging ? '--staging' : '');
	return utils.exec(cmd)
	.then((result) => {
	if (result) {
	logger.info('Renew Result: ' + result);
	}
	return internalNginx.reload()
	.then(() => {
	logger.info('Renew Complete');
	return result;
	});
	})
	.then(() => {
	// Now go and fetch all the letsencrypt certs from the db and query the files and update expiry times
	return certificateModel
	.query()
	.where('is_deleted', 0)
	.andWhere('provider', 'letsencrypt')
	.then((certificates) => {
	if (certificates && certificates.length) {
	let promises = [];
	certificates.map(function (certificate) {
	promises.push(
	internalCertificate.getCertificateInfoFromFile('/etc/letsencrypt/live/npm-' + certificate.id + '/fullchain.pem')
	.then((cert_info) => {
	return certificateModel
	.query()
	.where('id', certificate.id)
	.andWhere('provider', 'letsencrypt')
	.patch({
	expires_on: moment(cert_info.dates.to, 'X').format('YYYY-MM-DD HH:mm:ss')
	});
	})
	.catch((err) => {
	// Don't want to stop the train here, just log the error
	logger.error(err.message);
	})
	);
	});
	return Promise.all(promises);
	}
	});
	})
	.then(() => {
	internalCertificate.intervalProcessing = false;
	})
	.catch((err) => {
	logger.error(err);
	internalCertificate.intervalProcessing = false;
	});
	}
	},

It should use this function to renew the certificates

nginx-proxy-manager/backend/internal/certificate.js

Lines 999 to 1028 in 14b889a

	renewLetsEncryptSslWithDnsChallenge: (certificate) => {
	const dns_plugin = dnsPlugins[certificate.meta.dns_provider];
	if (!dns_plugin) {
	throw Error(`Unknown DNS provider '${certificate.meta.dns_provider}'`);
	}
	logger.info(`Renewing Let'sEncrypt certificates via ${dns_plugin.display_name} for Cert #${certificate.id}: ${certificate.domain_names.join(', ')}`);
	let mainCmd = certbotCommand + ' renew ' +
	'--config "' + letsencryptConfig + '" ' +
	'--cert-name "npm-' + certificate.id + '" ' +
	'--disable-hook-validation ' +
	'--no-random-sleep-on-renew ' +
	(letsencryptStaging ? ' --staging' : '');
	// Prepend the path to the credentials file as an environment variable
	if (certificate.meta.dns_provider === 'route53') {
	const credentialsLocation = '/etc/letsencrypt/credentials/credentials-' + certificate.id;
	mainCmd = 'AWS_CONFIG_FILE=\'' + credentialsLocation + '\' ' + mainCmd;
	}
	logger.info('Command:', mainCmd);
	return utils.exec(mainCmd)
	.then(async (result) => {
	logger.info(result);
	return result;
	});
	},

I've "quick" fixed my installation by hardcoding the aws credentials:

const cmd = certbotCommand + ' renew --non-interactive --quiet ' +
        '--config "' + letsencryptConfig + '" ' +
        '--preferred-challenges "dns,http" ' +
        '--disable-hook-validation ' +
        (letsencryptStaging ? '--staging' : '');

let mainCmd = 'AWS_CONFIG_FILE=\'/etc/letsencrypt/credentials/credentials-3\' '+cmd

return utils.exec(/*cmd*/mainCmd)
        .then((result) => {

[matthew-larner]

@jc21 if I can push a fix for this would you support a PR?

[miztertea]

Still having this issue. I was able to resolve by passing the credentials as an environment variable in docker-compose:

environment: - AWS_CONFIG_FILE=/etc/letsencrypt/credentials/credentials-1

[jangeador]

Just to thank @miztertea for contributing a solution, and to confirm that the env variable solved the problem for me as well.