How do I resolve an internal error when activating my Storage Gateway?

Last updated: 2020-10-05

I'm trying to activate my gateway on AWS Storage Gateway, but I'm getting an internal error. How can I troubleshoot this?

Short description

A request to Storage Gateway traverses two network paths. First, the activation request sent by a client connects to the gateway's virtual machine (VM) or Amazon Elastic Compute Cloud (Amazon EC2) instance over port 80. Second, if the gateway successfully receives the activation request, then the gateway communicates with the Storage Gateway endpoints to receive an activation key. If the gateway can't reach those Storage Gateway endpoints, then the service responds to the client with an internal error message.

Resolution

If using a public endpoint to activate your gateway

1.    Verify that you opened the required ports. For gateways deployed on-premises, check that the ports are open on your local firewall. For gateways deployed on an Amazon EC2 instance, check that the ports are open on the instance's security group. To confirm that the ports are open, run a telnet command on the public endpoint from a server that's in the same subnet as the gateway. For example, the following telnet commands test the connection to port 443:

telnet d4kdq0yaxexbo.cloudfront.net 443
telnet storagegateway.region.amazonaws.com 443
telnet dp-1.storagegateway.region.amazonaws.com 443
telnet proxy-app.storagegateway.region.amazonaws.com 443
telnet client-cp.storagegateway.region.amazonaws.com 443
telnet anon-cp.storagegateway.region.amazonaws.com 443

To confirm that the gateway itself can reach the endpoint, access the gateway's local VM console (for gateways deployed on-premises), or SSH to the gateway's instance (for gateways deployed on Amazon EC2). Then, run a network connectivity test. Confirm that the test returns [PASSED].

Note: The default login user name for the gateway console is admin, and the default password is password.

2.    Confirm that there's no SSL inspection, deep packet inspection, or any other form of firewall security that modifies packets sent from the gateway to the public endpoints. The SSL handshake fails if the SSL certificate is modified from what the activation endpoint expects. To confirm that there's no SSL inspection in progress, run an OpenSSL command on the main activation endpoint (anon-cp.storagegateway.region.amazonaws.com) on port 443 from a machine that's in the same subnet as the gateway:

Note: Replace region with your activation Region.

$ openssl s_client -connect  anon-cp.storagegateway.region.amazonaws.com:443 -servername anon-cp.storagegateway.region.amazonaws.com

If there's no SSL inspection in progress, then the command returns a response similar to the following:

$ openssl s_client -connect anon-cp.storagegateway.us-east-2.amazonaws.com:443 -servername anon-cp.storagegateway.us-east-2.amazonaws.com
CONNECTED(00000003)
depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify return:1
depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
verify return:1
depth=0 CN = anon-cp.storagegateway.us-east-2.amazonaws.com
verify return:1
---
Certificate chain
 0 s:/CN=anon-cp.storagegateway.us-east-2.amazonaws.com
   i:/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon
 1 s:/C=US/O=Amazon/OU=Server CA 1B/CN=Amazon
   i:/C=US/O=Amazon/CN=Amazon Root CA 1
 2 s:/C=US/O=Amazon/CN=Amazon Root CA 1
   i:/C=US/ST=Arizona/L=Scottsdale/O=Starfield Technologies, Inc./CN=Starfield Services Root Certificate Authority - G2
 3 s:/C=US/ST=Arizona/L=Scottsdale/O=Starfield Technologies, Inc./CN=Starfield Services Root Certificate Authority - G2
   i:/C=US/O=Starfield Technologies, Inc./OU=Starfield Class 2 Certification Authority
---

If there is an ongoing SSL inspection, then the response shows an altered certificate chain, similar to the following:

$ openssl s_client -connect  anon-cp.storagegateway.ap-southeast-1.amazonaws.com:443 -servername anon-cp.storagegateway.ap-southeast-1.amazonaws.com
CONNECTED(00000003)
depth=0 DC = com, DC = amazonaws, OU = AWS, CN = anon-cp.storagegateway.ap-southeast-1.amazonaws.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 DC = com, DC = amazonaws, OU = AWS, CN = anon-cp.storagegateway.ap-southeast-1.amazonaws.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/DC=com/DC=amazonaws/OU=AWS/CN=anon-cp.storagegateway.ap-southeast-1.amazonaws.com
   i:/C=IN/O=Company/CN=Admin/ST=KA/L=New town/OU=SGW/emailAddress=admin@company.com
---

The activation endpoint accepts SSL handshakes only if it recognizes the SSL certificate. This means that the gateway's outbound traffic to the endpoints must be exempt from any SSL inspection or deep packet inspection performed by any firewalls in your network.

3.    Confirm that your gateway is synchronizing time properly. Excessive time skews can cause SSL handshake errors. Use the gateway's local VM console to check your gateway's time synchronization. The time skew should be no larger than 60 seconds.

Note: The System Time Management option isn't available on gateways that are hosted on an EC2 instance. Confirm that the instance can connect to the NTP server pool list over ports UDP and TCP 123:

0.amazon.pool.ntp.org
1.amazon.pool.ntp.org
2.amazon.pool.ntp.org
3.amazon.pool.ntp.org

If using an Amazon Virtual Private Cloud (Amazon VPC) endpoint to activate your gateway

1.    Verify that you opened the required ports within your local firewall (for gateways deployed on-premises) or security group (for gateways deployed in Amazon EC2). The ports required for connecting a gateway to a Storage Gateway VPC endpoint are different from those required when you connect a gateway to the public endpoints. The following ports are required for connecting to a Storage Gateway VPC endpoint:

  • TCP 443
  • TCP 1026
  • TCP 1027
  • TCP 1028
  • TCP 1031
  • TCP 2222

Additionally, check the security group that's attached to your Storage Gateway VPC endpoint. The default security group attached to the endpoint might not allow the required ports. Create a new security group that allows traffic from your gateway's IP address range over the required ports. Then, attach that security group to the VPC endpoint.

Note: Use the Amazon VPC console to verify the security group that's attached to the VPC endpoint. View your Storage Gateway VPC endpoint from the console, and then choose the Security Groups tab.

To confirm that the required ports are open, run telnet commands on the Storage Gateway VPC Endpoint from a server that's in the same subnet as the gateway. You can run the tests on the first DNS name that doesn't specify an Availability Zone. For example, the following telnet commands test the required port connections using the DNS name vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:

telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 443
telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 1026
telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 1027
telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 1028
telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 1031
telnet vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com 2222

2.    Confirm that there's no SSL inspection, deep packet inspection, or any other form of firewall security that modifies packets sent from the gateway to your Storage Gateway VPC endpoint. The SSL handshake fails if the SSL certificate is modified from what the activation endpoint expects. To confirm that there's no SSL inspection in progress, run an OpenSSL command on your Storage Gateway VPC endpoint from a machine that's in the same subnet as the gateway. Run the command for each required port:

$ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:443 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com

$ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1026 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com

$ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1027 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com

$ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1028 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com

$ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1031 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com

$ openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:2222 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com

If there's no SSL inspection in progress, the command returns a response similar to the following:

openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1027 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com
CONNECTED(00000005)
depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify return:1
depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
verify return:1
depth=0 CN = anon-cp.storagegateway.us-east-1.amazonaws.com
verify return:1
---
Certificate chain
 0 s:CN = anon-cp.storagegateway.us-east-1.amazonaws.com
   i:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
 1 s:C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
   i:C = US, O = Amazon, CN = Amazon Root CA 1
 2 s:C = US, O = Amazon, CN = Amazon Root CA 1
   i:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
 3 s:C = US, ST = Arizona, L = Scottsdale, O = "Starfield Technologies, Inc.", CN = Starfield Services Root Certificate Authority - G2
   i:C = US, O = "Starfield Technologies, Inc.", OU = Starfield Class 2 Certification Authority
---

If there is an ongoing SSL inspection, the response shows an altered certificate chain, similar to the following:

openssl s_client -connect vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com:1027 -servername vpce-1234567e1c24a1fe9-62qntt8k.storagegateway.us-east-1.vpce.amazonaws.com
CONNECTED(00000005)
depth=2 C = US, O = Amazon, CN = Amazon Root CA 1
verify return:1
depth=1 C = US, O = Amazon, OU = Server CA 1B, CN = Amazon
verify return:1
depth=0 DC = com, DC = amazonaws, OU = AWS, CN = anon-cp.storagegateway.us-east-1.amazonaws.com
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/DC=com/DC=amazonaws/OU=AWS/CN=anon-cp.storagegateway.us-east-1.amazonaws.com
   i:/C=IN/O=Company/CN=Admin/ST=KA/L=New town/OU=SGW/emailAddress=admin@company.com
---

The activation endpoint accepts SSL handshakes only if it recognizes the SSL certificate. This means that the gateway's outbound traffic to your VPC endpoint over all required ports must be exempt from any SSL inspection or deep packet inspection performed by any firewalls in your network.

3.    Confirm that your gateway is synchronizing time properly. Excessive time skews can cause SSL handshake errors. Use the gateway's local VM console to check your gateway's time synchronization. The time skew should be no larger than 60 seconds.

Note: The System Time Management option isn't available on gateways that are hosted on an EC2 instance. Confirm that the instance can connect to the NTP server pool list over ports UDP and TCP 123:
0.amazon.pool.ntp.org
1.amazon.pool.ntp.org
2.amazon.pool.ntp.org
3.amazon.pool.ntp.org

4.    Before activation, if you have an HTTP proxy on Amazon EC2 configured on the on-premises gateway VM as a Squid proxy on port 3128, then confirm the following:

  • The security group attached to the HTTP proxy on Amazon EC2 must have an inbound rule that allows Squid proxy traffic on port 3128 from the gateway VM's IP address.
  • The security group attached to the Storage Gateway VPC endpoint must have inbound rules that allow traffic on ports 1026-1028, 1031, 2222, and 443 from the IP address of the HTTP proxy on Amazon EC2.

If using a public endpoint to activate your gateway when there is a Storage Gateway VPC endpoint in the same VPC

Confirm that the Enable Private DNS Name setting is not enabled on your Storage Gateway VPC endpoint. If this setting is enabled, you can't activate any gateways from that VPC to the public endpoint. Follow these steps to disable the private DNS name option:

  1. Open the Amazon VPC console.
  2. In the navigation pane, choose Endpoints.
  3. Choose your Storage Gateway VPC endpoint.
  4. Choose Actions.
  5. Choose Manage Private DNS Names.
  6. For Enable Private DNS Name, uncheck Enable for this Endpoint.
  7. Choose Modify Private DNS Names to save the setting.

Did this article help?


Do you need billing or technical support?