How do I troubleshoot the AWS Glue error "VPC S3 endpoint validation failed for SubnetId"?

Last updated: 2021-08-09

My AWS Glue job fails with the error "JobRunId:jr_xx failed to execute with exception VPC S3 endpoint validation failed for SubnetId. Reason: Could not find S3 endpoint or NAT gateway for subnetId".

-or-

My AWS Glue Crawler fails with the error "VPC S3 endpoint validation failed for SubnetId. Reason: Could not find S3 endpoint or NAT gateway for subnetId".

Short description

You get this error when the following conditions are true:

  • You're using an AWS Glue connection with your AWS Glue job or AWS Glue crawler.
  • The subnet configured for your AWS Glue connection doesn't have an Amazon Simple Storage Service (Amazon S3) Virtual Private Cloud (VPC) endpoint or a NAT gateway setup.

AWS Glue creates elastic network interfaces in the subnet specified in the AWS Glue connection. These elastic network interfaces are used to connect to your data sources and Amazon S3. The elastic network interfaces are assigned a private IP address from the IP address range within the specified subnet. Because no public IP addresses are used, AWS Glue can't directly use the internet to connect to your data sources or Amazon S3 within your VPC. For more information, see Setting up your environment to access data stores.

Resolution

Be sure that the subnet configured for your AWS Glue connection has an Amazon S3 VPC gateway endpoint or a route to a NAT gateway in the subnet's route table. For more information, see Error: Could Not Find S3 Endpoint or NAT Gateway for subnetId in VPC. If your AWS Glue job must access the public internet, then check and confirm that the subnet is configured with a NAT gateway.

Setting up an S3 VPC gateway endpoint

To set up an S3 VPC gateway endpoint, do the following:

  1. Open the Amazon VPC console.
  2. In the navigation pane, choose Endpoints.
  3. Choose Create Endpoint.
  4. For Service Name, select com.amazonaws.us-east-1.s3. Be sure that the Type column indicates Gateway.
    Note:
    Be sure to replace us-east-1 with the AWS Region of your choice.
  5. For VPC, select the VPC where you want to create the endpoint.
  6. For Configure route tables, a route to the S3 VPC endpoint is automatically added.
  7. For Policy, you can leave the default option Full Access.
  8. Choose Create Endpoint.

For more information, see Create a gateway endpoint.

If you are setting up a more restrictive policy, then be sure to allow your AWS Glue Job or AWS Glue crawler's AWS Identity and Access Management (IAM) role to access the required S3 buckets and objects.

Setting up a NAT gateway

To set up a NAT gateway, do the following:

Note: Be sure that the VPC that's used for the AWS Glue connection has at least one public subnet. For more information, see VPC with public and private subnets (NAT).

  1. Open the Amazon VPC console.
  2. In the navigation pane, choose NAT Gateways.
  3. Choose Create NAT gateway.
  4. For Name, enter the name of the gateway that you want to create.
  5. For Subnet, select a public subnet in the VPC that you specified in the AWS Glue connection.
  6. For Connectivity type, you can leave the default option Public to create a public NAT gateway.
  7. Choose Create NAT gateway.

After setting up the gateway, you must do the following:

  • Modify the route table associated with the subnet that's specified in the AWS Glue connection.
  • Add a route to the NAT gateway that's created.

To modify the route table and add a route to the NAT gateway, do the following:

  1. In the navigation pane, choose Route Tables.
  2. Select the route table associated with the subnet that you want to modify.
  3. Choose Actions, and then choose Edit routes.
  4. In the Edit routes page, choose Add route.
  5. In the new route that appears, do the following:
    For Destination, enter 0.0.0.0/0.
    For Target, select NAT Gateway, and then select the NAT gateway that you created. Note: You must select these options to make sure that all the subnet traffic, such as internet bound traffic, are sent to the NAT gateway.
  6. Choose Save changes.

After setting up the S3 VPC endpoint or NAT gateway in your subnet, test your respective AWS Glue connection. Rerun your AWS Glue job or AWS Glue crawler after the test connection is successful.


Did this article help?


Do you need billing or technical support?