Dnscrypt-proxy no longer works after reboot

[shapiro125]

Describe the bug

After a reboot of the server, dnscrypt-proxy no longer works on Vultr. I believe it's related to #956, but that one was closed and the problem is still occuring for me.
To Reproduce

Steps to reproduce the behavior:

1. Run algo installation script
2. Dnscrypt-proxy works as expected.
3. If server needs to restart (unattended-reboot) or is rebooted, dnscrypt-proxy fails to resolve.

● dnscrypt-proxy.service - DNSCrypt-proxy client
   Loaded: loaded (/lib/systemd/system/dnscrypt-proxy.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/dnscrypt-proxy.service.d
           └─99-capabilities.conf
   Active: failed (Result: exit-code) since Tue 2019-03-12 23:33:53 UTC; 4s ago
     Docs: https://github.com/jedisct1/dnscrypt-proxy/wiki
  Process: 1696 ExecStart=/usr/bin/dnscrypt-proxy --config /etc/dnscrypt-proxy/dnscrypt-proxy.toml (code=exited, status=255)
 Main PID: 1696 (code=exited, status=255)

Mar 12 23:33:53 London systemd[1]: Started DNSCrypt-proxy client.
Mar 12 23:33:53 London dnscrypt-proxy[1696]: Unable to use source [public-resolvers]: [read udp 127.0.0.1:37798->127.0.0.53:53: read: connection refused]
Mar 12 23:33:53 London dnscrypt-proxy[1696]: No servers configured
Mar 12 23:33:53 London systemd[1]: dnscrypt-proxy.service: Main process exited, code=exited, status=255/n/a
Mar 12 23:33:53 London systemd[1]: dnscrypt-proxy.service: Failed with result 'exit-code'.

Expected behavior

dnscrypt-proxy restarts after a reboot.

[shapiro125]

Just as some background, the issue scaleway was having in #956 is now in Vultr on 18.04 -- #956 (comment)

/etc/network/interfaces.d is empty after restart

[davidemyers]

/etc/network/interfaces.d is no longer used on 18.04.

It looks like maybe systemd-resolved is not ready to receive DNS requests when dnscrypt-proxy starts up.

Do Vultr instances use DHCP? Could you post the contents of /etc/netplan/50-cloud-init.yaml (or whatever files you find in /etc/netplan)? You can obscure any addresses in that file if you like. Maybe the DHCP server is slow to respond.

[davidemyers]

Here's something to try. Edit /lib/systemd/system/dnscrypt-proxy.service and change:

Before=nss-lookup.target
Wants=nss-lookup.target

to:

After=nss-lookup.target

Then reboot.

[shapiro125]

Thanks @davidemyers. I also just tried re-deploying to EC2, and the same problem persists.

In /etc/netplan I have two files -- 01-netcfg.yaml which is empty and 10-ens3.yaml which is

network:
  version: 2
  renderer: networkd
  ethernets:
    ens3:
      dhcp4: yes

Tried your After=nss-lookup.target trick, but still getting

● dnscrypt-proxy.service - DNSCrypt-proxy client
   Loaded: loaded (/lib/systemd/system/dnscrypt-proxy.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/dnscrypt-proxy.service.d
           └─99-capabilities.conf
   Active: failed (Result: exit-code) since Wed 2019-03-13 00:58:43 UTC; 1min 9s ago
     Docs: https://github.com/jedisct1/dnscrypt-proxy/wiki
  Process: 1297 ExecStart=/usr/bin/dnscrypt-proxy --config /etc/dnscrypt-proxy/dnscrypt-proxy.toml (code=exited, status=255)
 Main PID: 1297 (code=exited, status=255)

Mar 13 00:58:43 London systemd[1]: Started DNSCrypt-proxy client.
Mar 13 00:58:43 London dnscrypt-proxy[1297]: Unable to use source [public-resolvers]: [read udp 127.0.0.1:48349->127.0.0.53:53: read: connection ref
Mar 13 00:58:43 London dnscrypt-proxy[1297]: No servers configured
Mar 13 00:58:43 London systemd[1]: dnscrypt-proxy.service: Main process exited, code=exited, status=255/n/a
Mar 13 00:58:43 London systemd[1]: dnscrypt-proxy.service: Failed with result 'exit-code'.
Mar 13 00:58:43 London systemd[1]: dnscrypt-proxy.service: Start request repeated too quickly.
Mar 13 00:58:43 London systemd[1]: dnscrypt-proxy.service: Failed with result 'exit-code'.
Mar 13 00:58:43 London systemd[1]: Failed to start DNSCrypt-proxy client.

Maybe it's a firewall issue? It starts, but then can't reach the public resolvers for some reason.

[davidemyers]

Does dnscrypt-proxy start properly if you start it manually after booting (systemctl start dnscrypt-proxy)?

[jackivanov]

OK, seems like a bug. We need to start dnscrypt-proxy after systemd-resolved

[shapiro125]

Ok, thanks all! @davidemyers -- no systemctl start dnscrypt-proxy doesn't help either.

[jackivanov]

For now systemctl restart dnscrypt-proxy should help