Apply exponential backoff to code-server reconnection

[jawnsy]

Current behavior

When code-server loses the connection to the server, it attempts to re-connect at 30 second intervals repeatedly until the connection succeeds. If there is a large-scale event causing an outage while clients have code-server open, then this can cause repeated spikes in traffic on the server.

Steps to reproduce

1. Open VSCode via master.cdr.dev
2. Rebuild your environment while VSCode is open

While the environment is rebuilding, we just see this message every 30 seconds, counting down to zero and then restarting back at 30 seconds:

Proposed solution

We should consider applying a randomized exponential backoff scheme (e.g. try to reconnect in 10 seconds, then 20 seconds, then 40 seconds) to avoid coordinated events of large-scale deployments (hundreds or thousands of clients) hammering the server by attempting to reconnect at approximately the same time. This also allows us to begin with much smaller intervals to begin with (e.g. 5 seconds) making the reconnection much faster in the common case of a transient network issue.

[jsjoeio]

Seems like a valid point! Though I would imagine this behavior is from VS Code and not specific to code-server - though I hope @code-asher will correct me if that's wrong.

[code-asher]

Yup it is from VS Code, maybe it's something we could patch upstream! But we've patched this connection logic locally before (those changes have since been removed since they were added upstream) so it wouldn't be a first haha. *Edit*: forgot to say, this seems like a great idea to me.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no activity occurs in the next 5 days.