"Timeout when waiting for search string OpenSSH in XX.XX.XX.XX:4160" failure when deploying to Vultr

[Imfae]

Describe the bug

When attempting to install algo on Vultr vps from Windows 10, I'm repeatedly confronted with the error "fatal: [localhost]: FAILED! => {"changed": false, "elapsed": 321, "msg": "Timeout when waiting for search string OpenSSH in XX.XX.XX.XX:4160"}. It won't resolve after I followed the troubleshooting guide and removed all cloud firewalls.

To Reproduce

Steps to reproduce the behavior:

1. Set up account in Vultr.
2. Install WLS in Windows 10.
3. Install algo.
4. Install algo dependencies.
5. Run ./algo

Expected behavior

Algo set up with no issue.

Additional context

I've set up algo from the same computer with the same cloud service provider before, but destroyed the server after Ubuntu 19.04 was no longer supported by Vultr. So I don't think this issue is caused by basic mistakes like ssh service not opened.

Full log

`[DEPRECATION WARNING]: Ansible will require Python 3.8 or newer on the controller starting with Ansible 2.12. Current version: 3.6.7 (default, Oct 22 2018, 11:32:17)
[GCC 8.2.0]. This feature will be removed from ansible-core in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [localhost] *******************************************************************************************************************************************************
TASK [Gathering Facts] *************************************************************************************************************************************************ok: [localhost]

TASK [Playbook dir stat] ***********************************************************************************************************************************************ok: [localhost]

TASK [Ensure Ansible is not being run in a world writable directory] ***************************************************************************************************ok: [localhost] => {
"changed": false,
"msg": "All assertions passed"
}
[WARNING]: The value '' is not a valid IP address or network, passing this value to ipaddr filter might result in breaking change in future.

TASK [Ensure the requirements installed] *******************************************************************************************************************************ok: [localhost]

TASK [Set required ansible version as a fact] **************************************************************************************************************************ok: [localhost] => (item=ansible-core==2.11.3)

TASK [Verify Python meets Algo VPN requirements] ***********************************************************************************************************************ok: [localhost] => {
"changed": false,
"msg": "All assertions passed"
}

TASK [Verify Ansible meets Algo VPN requirements] **********************************************************************************************************************ok: [localhost] => {
"changed": false,
"msg": "All assertions passed"
}
[WARNING]: Found variable using reserved name: no_log

PLAY [Ask user for the input] ******************************************************************************************************************************************
TASK [Gathering Facts] *************************************************************************************************************************************************ok: [localhost]
[Cloud prompt]
What provider would you like to use?
1. DigitalOcean
2. Amazon Lightsail
3. Amazon EC2
4. Microsoft Azure
5. Google Compute Engine
6. Hetzner Cloud
7. Vultr
8. Scaleway
9. OpenStack (DreamCompute optimised)
10. CloudStack (Exoscale optimised)
11. Linode
12. Install to existing Ubuntu 18.04 or 20.04 server (for more advanced users)

Enter the number of your desired provider
:
7^M
TASK [Cloud prompt] ****************************************************************************************************************************************************ok: [localhost]

TASK [Set facts based on the input] ************************************************************************************************************************************ok: [localhost]
[VPN server name prompt]
Name the vpn server
[algo]
:
algo^M
TASK [VPN server name prompt] ******************************************************************************************************************************************ok: [localhost]
[Cellular On Demand prompt]
Do you want macOS/iOS clients to enable "Connect On Demand" when connected to cellular networks?
[y/N]
:
n^M
TASK [Cellular On Demand prompt] ***************************************************************************************************************************************ok: [localhost]
[Wi-Fi On Demand prompt]
Do you want macOS/iOS clients to enable "Connect On Demand" when connected to Wi-Fi?
[y/N]
:
n^M
TASK [Wi-Fi On Demand prompt] ******************************************************************************************************************************************ok: [localhost]
[Retain the PKI prompt]
Do you want to retain the keys (PKI)? (required to add users in the future, but less secure)
[y/N]
:
n^M
TASK [Retain the PKI prompt] *******************************************************************************************************************************************ok: [localhost]
[DNS adblocking prompt]
Do you want to enable DNS ad blocking on this VPN server?
[y/N]
:
y^M
TASK [DNS adblocking prompt] *******************************************************************************************************************************************ok: [localhost]
[SSH tunneling prompt]
Do you want each user to have their own account for SSH tunneling?
[y/N]
:
y^M
TASK [SSH tunneling prompt] ********************************************************************************************************************************************ok: [localhost]

TASK [Set facts based on the input] ************************************************************************************************************************************ok: [localhost]

PLAY [Provision the server] ********************************************************************************************************************************************
TASK [Gathering Facts] *************************************************************************************************************************************************ok: [localhost]

--> Please include the following block of text when reporting issues:

Algo running on: Ubuntu 18.04.2 LTS (Virtualized: wsl)
Created from git fork. Last commit: de1e909 Update cloud-linode.md (#14348)
Python 3.6.7
Runtime variables:
algo_provider "vultr"
algo_ondemand_cellular "False"
algo_ondemand_wifi "False"
algo_ondemand_wifi_exclude "X251bGw="
algo_dns_adblocking "True"
algo_ssh_tunneling "True"
wireguard_enabled "True"
dns_encryption "True"

TASK [Display the invocation environment] ******************************************************************************************************************************changed: [localhost -> localhost]

TASK [Install the requirements] ****************************************************************************************************************************************ok: [localhost -> localhost]

TASK [Generate the SSH private key] ************************************************************************************************************************************ok: [localhost]

TASK [Generate the SSH public key] *************************************************************************************************************************************ok: [localhost]

TASK [Copy the private SSH key to /tmp] ********************************************************************************************************************************ok: [localhost -> localhost]

TASK [Include a provisioning role] *************************************************************************************************************************************[cloud-vultr : pause]
Enter the local path to your configuration INI file
(https://trailofbits.github.io/algo/cloud-vultr.html):
:
^M
TASK [cloud-vultr : pause] *********************************************************************************************************************************************ok: [localhost]

TASK [cloud-vultr : Set the token as a fact] ***************************************************************************************************************************ok: [localhost]

TASK [cloud-vultr : Get regions] ***************************************************************************************************************************************ok: [localhost]

TASK [cloud-vultr : Format regions] ************************************************************************************************************************************ok: [localhost]

TASK [cloud-vultr : Set regions as a fact] *****************************************************************************************************************************ok: [localhost]

TASK [cloud-vultr : Set default region] ********************************************************************************************************************************ok: [localhost]
[cloud-vultr : pause]
What region should the server be located in?
(https://www.vultr.com/locations/):
1. Sydney
2. São Paulo
3. Toronto
4. Frankfurt
5. Paris
6. London
7. Tokyo
8. Seoul
9. Mexico City
10. Amsterdam
11. Stockholm
12. Singapore
13. Atlanta
14. Chicago
15. Dallas
16. Los Angeles
17. Miami
18. New Jersey
19. Seattle
20. Silicon Valley

Enter the number of your desired region
[18]
:
11^M
TASK [cloud-vultr : pause] *********************************************************************************************************************************************ok: [localhost]

TASK [cloud-vultr : Set the desired region as a fact] ******************************************************************************************************************ok: [localhost]

TASK [cloud-vultr : Creating a firewall group] *************************************************************************************************************************changed: [localhost]

TASK [cloud-vultr : Creating firewall rules] ***************************************************************************************************************************changed: [localhost] => (item={'protocol': 'tcp', 'port': 4160, 'ip': 'v4', 'cidr': '0.0.0.0/0'})
changed: [localhost] => (item={'protocol': 'tcp', 'port': 4160, 'ip': 'v6', 'cidr': '::/0'})
changed: [localhost] => (item={'protocol': 'udp', 'port': 500, 'ip': 'v4', 'cidr': '0.0.0.0/0'})
changed: [localhost] => (item={'protocol': 'udp', 'port': 500, 'ip': 'v6', 'cidr': '::/0'})
changed: [localhost] => (item={'protocol': 'udp', 'port': 4500, 'ip': 'v4', 'cidr': '0.0.0.0/0'})
changed: [localhost] => (item={'protocol': 'udp', 'port': 4500, 'ip': 'v6', 'cidr': '::/0'})
changed: [localhost] => (item={'protocol': 'udp', 'port': 51820, 'ip': 'v4', 'cidr': '0.0.0.0/0'})
changed: [localhost] => (item={'protocol': 'udp', 'port': 51820, 'ip': 'v6', 'cidr': '::/0'})

TASK [cloud-vultr : Upload the startup script] *************************************************************************************************************************ok: [localhost]

TASK [cloud-vultr : Creating a server] *********************************************************************************************************************************changed: [localhost]

TASK [cloud-vultr : set_fact] ******************************************************************************************************************************************ok: [localhost]

TASK [Set subjectAltName as a fact] ************************************************************************************************************************************ok: [localhost]

TASK [Add the server to an inventory group] ****************************************************************************************************************************changed: [localhost]

TASK [Additional variables for the server] *****************************************************************************************************************************changed: [localhost]

TASK [Wait until SSH becomes ready...] *********************************************************************************************************************************fatal: [localhost]: FAILED! => {"changed": false, "elapsed": 321, "msg": "Timeout when waiting for search string OpenSSH in XX.XX.XX.XX:4160"}

TASK [include_tasks] ***************************************************************************************************************************************************included: /home/dust/algo/playbooks/rescue.yml for localhost

TASK [debug] ***********************************************************************************************************************************************************ok: [localhost] => {
"fail_hint": [
"Sorry, but something went wrong!",
"Please check the troubleshooting guide.",
"https://trailofbits.github.io/algo/troubleshooting.html"
]
}

TASK [Fail the installation] *******************************************************************************************************************************************fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed as requested from task"}

PLAY RECAP *************************************************************************************************************************************************************localhost : ok=41 changed=6 unreachable=0 failed=1 skipped=1 rescued=1 ignored=0`

[davidemyers]

I can recreate this, in both the Stockholm and Atlanta regions. Vultr has made some changes to their Ubuntu Server configuration that break Algo. The main change is that they're enabling the UFW firewall by default and only opening port 22. Algo cannot coexist with other host-based firewalls.

Vultr is also now using cloud-init, but I don't think that actually breaks Algo.

As an additional complication the default Vultr DNS server is unreachable from the instance I created in Stockholm, but works from Atlanta. That will also break Algo.

Please change the title of this issue to something to indicate it is specific to Vultr, such as "Cannot deploy to Vultr".

As a workaround, you can edit the file files/cloud-init/base.sh and add to the bottom:

ufw disable

But it still might not work in Stockholm right now.

Ping @jackivanov.

[Imfae]

After adding ufw disable to files/cloud-init/base.sh, my attempt to install on Vultr's Stockholm server resulted in:

What region should the server be located in?
(https://www.vultr.com/locations/):
1. Sydney
2. São Paulo
3. Toronto
4. Frankfurt
5. Paris
6. London
7. Tokyo
8. Seoul
9. Mexico City
10. Amsterdam
11. Stockholm
12. Singapore
13. Atlanta
14. Chicago
15. Dallas
16. Los Angeles
17. Miami
18. New Jersey
19. Seattle
20. Silicon Valley

Enter the number of your desired region
[18]
:
11^M
TASK [cloud-vultr : pause] *********************************************************************************************************************************************
ok: [localhost]

TASK [cloud-vultr : Set the desired region as a fact] ******************************************************************************************************************
ok: [localhost]

TASK [cloud-vultr : Creating a firewall group] *************************************************************************************************************************
ok: [localhost]

TASK [cloud-vultr : Creating firewall rules] ***************************************************************************************************************************
ok: [localhost] => (item={'protocol': 'tcp', 'port': 4160, 'ip': 'v4', 'cidr': '0.0.0.0/0'})
ok: [localhost] => (item={'protocol': 'tcp', 'port': 4160, 'ip': 'v6', 'cidr': '::/0'})
ok: [localhost] => (item={'protocol': 'udp', 'port': 500, 'ip': 'v4', 'cidr': '0.0.0.0/0'})
ok: [localhost] => (item={'protocol': 'udp', 'port': 500, 'ip': 'v6', 'cidr': '::/0'})
ok: [localhost] => (item={'protocol': 'udp', 'port': 4500, 'ip': 'v4', 'cidr': '0.0.0.0/0'})
ok: [localhost] => (item={'protocol': 'udp', 'port': 4500, 'ip': 'v6', 'cidr': '::/0'})
ok: [localhost] => (item={'protocol': 'udp', 'port': 51820, 'ip': 'v4', 'cidr': '0.0.0.0/0'})
ok: [localhost] => (item={'protocol': 'udp', 'port': 51820, 'ip': 'v6', 'cidr': '::/0'})

TASK [cloud-vultr : Upload the startup script] *************************************************************************************************************************
changed: [localhost]

TASK [cloud-vultr : Creating a server] *********************************************************************************************************************************
changed: [localhost]

TASK [cloud-vultr : set_fact] ******************************************************************************************************************************************
ok: [localhost]

TASK [Set subjectAltName as a fact] ************************************************************************************************************************************
ok: [localhost]

TASK [Add the server to an inventory group] ****************************************************************************************************************************
changed: [localhost]

TASK [Additional variables for the server] *****************************************************************************************************************************
changed: [localhost]

TASK [Wait until SSH becomes ready...] *********************************************************************************************************************************
ok: [localhost]

TASK [Linux | set OS specific facts] ***********************************************************************************************************************************
ok: [localhost]

TASK [Set config paths as facts] ***************************************************************************************************************************************
ok: [localhost]

TASK [Update config paths] *********************************************************************************************************************************************
changed: [localhost]

TASK [debug] ***********************************************************************************************************************************************************
ok: [localhost] => {
"IP_subject_alt_name": "XX.XX.XX.XX"
}

TASK [Wait 600 seconds for target connection to become reachable/usable] ***********************************************************************************************
ok: [localhost -> XX.XX.XX.XX] => (item=XX.XX.XX.XX)

PLAY [Configure the server and install required software] **************************************************************************************************************

TASK [Wait until the cloud-init completed] *****************************************************************************************************************************
ok: [XX.XX.XX.XX]

TASK [Ensure the config directory exists] ******************************************************************************************************************************
ok: [XX.XX.XX.XX -> localhost]

TASK [Dump the ssh config] *********************************************************************************************************************************************
ok: [XX.XX.XX.XX -> localhost]

TASK [common : Check the system] ***************************************************************************************************************************************
ok: [XX.XX.XX.XX]

TASK [common : include_tasks] ******************************************************************************************************************************************
included: /home/dust/algo/roles/common/tasks/ubuntu.yml for XX.XX.XX.XX

TASK [common : Gather facts] *******************************************************************************************************************************************
ok: [XX.XX.XX.XX]

Installation seems unable to continue after TASK [common : Gather facts].

I managed to install algo on Vultr's Amsterdam server instead. But when I try to connect via wireguard, log shows:

Startup complete
Sending handshake initiation to peer 1 (XX.XX.XX.XX:51820)
Handshake for peer 1 (XX.XX.XX.XX:51820) did not complete after 5 seconds, retrying (try 2)
Sending handshake initiation to peer 1 (XX.XX.XX.XX:51820)
Handshake for peer 1 (XX.XX.XX.XX:51820) did not complete after 5 seconds, retrying (try 2)

I'm unable to connect to the vpn because traffic consists of 'sending' and no 'receiving.'

[Imfae]

Nevermind the last part. I consulted #1594 and changed the wireguard port to port 53. Now I'm able to connect with the server just fine.

I am curious though, will the ufw disable affect the security of my vpn in any way?

[davidemyers]

I am curious though, will the ufw disable affect the security of my vpn in any way?

No, Algo puts its own set of strict firewall rules in place.

[Imfae]

Thanks!