How do I troubleshoot problems with creating or connecting to an AWS Glue development endpoint?

Most problems with AWS Glue development endpoints fall into one of the following categories. Troubleshoot in this order:

1. AWS Identity and Access Management (IAM) roles and permissions
2. Networking
3. SSH keys
   

IAM roles and permissions

1. Be sure that the AWSGlueServiceRole is attached to the IAM policy and that the policy allows AWS Glue to assume this role ("svc:glue.amazonaws.com").
2. If you can't create the development endpoint in a VPC, be sure that the IAM policy allows the following actions:
   ec2:DescribeVpcAttribute
   ec2:DescribeSubnets
   ec2:DescribeSecurityGroups
   ec2:DescribeNetworkInterfaces
   ec2:DeleteNetworkInterface
   ec2:CreateNetworkInterface
   ec2:DescribeRouteTables
   ec2:DescribeVpcEndpoints
3. If the development endpoint uses Amazon Simple Storage Service (Amazon S3) for the source and target, confirm that the IAM policy allows the following actions:
   s3:ListBucket
   s3:GetObject
   s3:ListBucket
   s3:PutObject
   s3:DeleteObject
4. If you specified Amazon S3 paths for the Python library and the dependent jars, confirm that the IAM policy allows access to both paths.
5. If you're using the AWS Glue Data Catalog as the Apache Hive metastore, be sure that the IAM policy allows the "glue:CreateDatabase" action and doesn't include an explicit deny. For more information, see Determining Whether a Request Is Allowed or Denied Within an Account.

If you're still not able to connect to the development endpoint, troubleshoot your networking settings.

Networking

If you created the development endpoint in a VPC:

1. If you're using a custom DNS, be sure that each available IP address in the subnet has an A record and a PTR record.
2. Be sure that the enableDnsHostnames and enableDnsSupport attributes are set to true. For more information, see Setting Up DNS in Your VPC.

Check the route table and security group associated with the subnet for the endpoint:

1. Confirm that the route table has a route to the Amazon S3 endpoint.
2. Confirm that the route table has a route to an internet gateway. This is required to access the development endpoint from a local machine.
3. Confirm that the security group has a self-referencing inbound rule for all TCP ports.
4. Confirm that security group has an outbound rule that allows all traffic or an outbound rule that to a self-referencing security group. For more information, see Setting Up Your Network for a Development Endpoint.

If you're having trouble connecting to the development endpoint using SSH:

1. Be sure that the security group has an inbound rule that allows port 22 from the machine that you're using to connect to.
2. When you launch a development endpoint in a VPC, AWS Glue uses private addresses. If you try to connect to the endpoint using SSH from outside the VPC, the connection fails even if the subnet has an internet gateway and route. To resolve this problem, associate an Elastic IP address to the development endpoint.
   Note: To connect with an Elastic IP address, the subnet must have a route to an internet gateway. To connect to a development endpoint in a private subnet, first connect to a bastion host using SSH. Then, connect to the development endpoint using SSH.

If you still can't connect to the development endpoint using SSH, check your SSH keys.

SSH keys

The following steps are for troubleshooting problems with SSH keys. If you're having trouble connecting to a development endpoint using SSH, see the Networking section in this article.

1. Confirm that the private key has a .pem extension. For example, if your key file is id_rsa, rename it to id_rsa.pem. If the private key doesn't have a .pem extension, then you get the error: "Connection blocked because server only allows public key authentication. Please contact your network administrator."
2. If you're having trouble converting a .ppk file to .pem, follow the steps at How do I convert a .pem file into a .ppk, and vice versa, on Windows and Linux?
3. If you're having trouble creating the development endpoint, be sure that the public key doesn't have white spaces or newline characters. Public keys generated with PuTTYgen often contain these characters. To resolve this problem, copy the public key to a text editor. Then, remove white spaces and newline characters. When you do this, the entire SSH key is on a single line. Use the modified key to create the development endpoint.