Why am I receiving the errors UPSTREAM_ERROR, UPSTREAM_NOT_FOUND, or CLIENT_UNAUTHORIZED when accessing my Lightsail instance using the browser-based SSH console?

Last updated: 2021-10-12

I'm receiving one of the following error messages when connecting to my Amazon Lightsail instance using the browser-based SSH console:

  • Your instance encountered an error and has closed the connection. Try again or contact customer support. UPSTREAM_ERROR [515]
  • An error occurred and we were unable to connect or stay connected to your instance. If this instance has just started up, try again in a minute or two. UPSTREAM_NOT_FOUND [519]
  • Login failed. If this instance has just started up, try again in a minute or two. CLIENT_UNAUTHORIZED [769]

How do I troubleshoot these errors?

Short description

The UPSTREAM_ERROR [515] and UPSTREAM_NOT_FOUND [519] errors indicate that your Lightsail instance is can't be reached when using SSH. The following are common reasons for these errors:

  • Instance boot failures, instance status check failures. or resource over-utilization on the instance.
  • An OS-level firewall is blocking SSH port access.
  • The default SSH port (22) is changed to a different one.
  • The SSH service is down.

The CLIENT_UNAUTHORIZED [769] error indicates that there is an SSH authentication issue in your Lightsail instance. The following are common reasons for this error:

  • Misconfiguration of the Lightsail system key in /etc/ssh/lightsail_instance_ca.pub.
  • You upgraded your Ubuntu instance from an older version to 20.04 or later.

Resolution

Instance boot failures, instance status check failures, or resource over-utilization on the instance

View the status check metrics of your instance to determine if the instance failed the system status check or the instance status check.

System status check

If the system status check failed, then the instance's underlying hardware is unhealthy. To fix the issue, stop and then start the instance. This migrates the instance to healthy hardware.

Warning: Be aware that the public IP address of the instance changes on every stop and start of the instance. If you want the IP address to stay the same on every stop and start of the instance, attach a static IP address.

Instance status check

If the instance status check failed, then an operating system-level issues might be causing boot errors. Or, the instance's resources, such as CPU or memory, might be over-utilized. For troubleshooting steps, see How can I troubleshoot common issues that cause my Lightsail instance to be unresponsive?

An OS-level firewall is blocking SSH port access

The browser-based SSH access uses the Amazon internal IP address to connect to your Lightsail Instance. If OS-level firewalls, such as Iptables, UFW, and so on, or access control files such as /etc/hosts.deny (that is, Hosts Access Control in cPanel based Instances) allow SSH access only to a set of IP addresses and deny all others, then you receive the UPSTREAM_NOT_FOUND [519] or UPSTREAM_ERROR [515] error. The same issue occurs if the firewall is blocking SSH access completely.

To resolve this issue, do one of the following:

  • If SSH access to the instance using the terminal or PuTTY application works, then log in to the instance and remove the deny rules from firewall and /etc/hosts.deny file.
  • If you have a cPanel instance, then access the WHM console to remove the firewall rules.
  • If you can't connect to the instance by using SSH from the terminal or PuTTY application, then deactivate the firewall using a launch script. The launch script can be added only while launching an instance. To use a launch script to deactivate OS-level firewalls, Iptables, and UFW, do the following:

1.    Open the Amazon Lightsail console.

2.    Create a manual snapshot of the instance.

3.    On the Snapshots tab, under Manual snapshots, select the three dots next to the new snapshot.

4.    Select Create new instance.

5.    Select the same Availability Zone as the previous instance.

6.    Select Add launch script, and then add the following script.

Note: The following example deactivates the UFW firewall, flushes all the iptable chains or firewall rules, and deactivates the /etc/hosts.deny file by renaming it.

sudo ufw disable
sudo iptables -F
sudo mv /etc/hosts.deny /etc/hosts.deny_backup
sudo touch /etc/hosts.deny
sudo systemctl enable sshd
sudo systemctl restart sshd

7.    Choose a new instance plan, or use the same plan as the previous instance.

8.    Enter a name for the instance, and then select Create instance.

After the new instance starts running, wait for 10 to 15 minutes, and then try to connect to it using the browser-based SSH console.

Note: If the previous instance had a static IP address, you can use it on the new instance. Detach the static IP address, and then attach it to the new instance from the Networking tab. For more information, see Static IP addresses in Amazon Lightsail.

The SSH service is down

If the SSH service isn't running or active on the instance, then the SSH connection fails and you receive the UPSTREAM_NOT_FOUND [519] error. To troubleshoot this issue, configure the AWS Systems Manager Session Manager service for your Lightsail instance. After configuring Session Manager, access the Instance without the SSH service, and then fix the SSH issue.

Basic troubleshooting steps for SSH issues include:

  • Review the SSH authentication logs in /var/log/auth.log or /var/log/secure file, depending on the OS distribution, to identify the errors.
  • Test the SSH configuration file syntax and then correct any errors.
sudo sshd -t
sudo systemctl restart sshd

Misconfiguration of the Lightsail system key /etc/ssh/lightsail_instance_ca.pub

Lightsail uses the system key /etc/ssh/lightsail_instance_ca.pub to activate browser-based SSH access. If this file is missing, then SSH authentication fails and you receive the CLIENT_UNAUTHORIZED [769]error. The same error occurs when the Lightsail system key isn't specified in the TrustedUserCAKeys parameter in the SSH configuration file /etc/ssh/sshd_config.

If the SSH connection to the instance through the terminal or PuTTY application works, then log in to the instance and confirm that the /etc/ssh/sshd_config file exists and that it contains the ssh-rsa key.

If the file is missing, then recreate the file using the following steps:

1.    Run the following command and confirm that you get the ssh-rsa key in the command output:

sudo cat /var/lib/cloud/instance/user-data.txt | grep ^ssh-rsa

2.    If the command returns the ssh-rsa key in the output, then run the following commands to copy it to /etc/ssh/lightsail_instance_ca.pub:

sudo sh -c "cat /var/lib/cloud/instance/user-data.txt | grep ^ssh-rsa > /etc/ssh/lightsail_instance_ca.pub"
    sudo sh -c "echo >> /etc/ssh/sshd_config" 
    sudo sh -c "echo 'TrustedUserCAKeys /etc/ssh/lightsail_instance_ca.pub' >> /etc/ssh/sshd_config"
    sudo systemctl restart sshd

3.    If you don't get an ssh-rsa key from step 1, or if you can't SSH to the instance using the terminal or PuTTY application, then create a manual snapshot of the instance and launch a new instance with that snapshot. The Lightsail system key is added to the server automatically at instance launch.

You've upgraded from an older Ubuntu version to 20.04 or later

For instances running Ubuntu 20.04 or later, you must allow certificate authorities (CAs) to use the ssh-rsa algorithm to sign certificates. Otherwise, authentication fails and you receive the CLIENT_UNAUTHORIZED [769] error. CAs are allowed to use the ssh-rsa algorithm by default in Ubuntu 20.04 instances. However, if you upgrade your older Ubuntu version to 20.04, then you must manually allow the CAs. To do this, do the following:

1.    Connect to the instance using an SSH client such as the terminal or PuTTY application.

2.    Open the SSH configuration file (/etc/ssh/sshd_config).

2.    Add ssh_rsa in the CASignatureAlgorithms line. The following example shows the CASignatureAlgorithms line after modifying it to add ssh_rsa:

CASignatureAlgorithms ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa

3.    Run the following command to restart the sshd service:

$ sudo systemctl restart sshd

Did this article help?


Do you need billing or technical support?