Launch EI accelerators in minutes with the Amazon Elastic Inference setup tool for EC2
The Amazon Elastic Inference (EI) setup tool is a Python script that enables you to quickly get started with EI.
Elastic Inference allows you to attach low-cost GPU-powered acceleration to Amazon EC2 and Amazon SageMaker instances to reduce the cost of running deep learning inference by up to 75 percent. If you are using EI for the first time, there are a number of dependencies that must be set up: Amazon Web Services (AWS) PrivateLink VPC Endpoints, IAM policies, and security group rules. To accelerate this, The EI setup script makes it easy for you to get started by creating the necessary resources to help you launch EI accelerators in minutes. In this blog post I describe how to use the script, functionalities of the script, and what to expect when you run it.
At a high level, the script does the following:
- Creates an IAM role for the instance with an IAM policy that lets you connect to the AWS Elastic Inference service.
- Creates a security group with the necessary ingress and egress rules to allow the instance to communicate with the accelerator.
- Creates an AWS PrivateLink VPC Endpoint within your desired subnet.
- Launches the desired EC2 instance with an EI accelerator using the latest AWS Deep Learning AMI (DLAMI) for the chosen operating system
To set up EI, run the script linked below. It depends on the following entities:
- Python 3 installed on your local machine where you expect to run the tool.
- The AWS SDK for Python (Boto3).
- An Amazon VPC in the Region where you are launching the instance (could be your default VPC).
- Subnet where you’d like to launch the instance.
- EC2 Key Pair.
- AWS credentials.
With these in place, download the amazonei_setup.py script from GitHub to your local machine and run it from your terminal using following command:
What the tool creates on your behalf
The script creates following AWS resources:
- Instance role with an Amazon EI Policy. This role is created the first time the script is run. In all subsequent runs, script reuses this IAM role. If this role is deleted, script recreates the role next time it is run. The IAM role has following properties:
- Role name: Amazon-Elastic-Inference-Connect-Role
- Policy name: Amazon-Elastic-Inference-Connect-Policy
- Instance profile name: Amazon-Elastic-Inference-Instance-Profile
The policy description is as follows:
- Security Group (SG). The security group associated with the EC2 instance should allow inbound traffic to port 443 as required by Amazon EI service. You also need inbound rules that allow traffic to port 22 for SSH. If a security group matching these rules is found, it is used. However, if no matching SG is found, a new SG with required rules is created. The outbound rules are set to allow traffic to all ports. The new SG name is amazon_ei_security_group, with the description Security Group for accessing Amazon EI service.
- Interface VPC endpoint (AWS PrivateLink). The script scans for existing endpoint associated with Amazon EI service for the Region and VPC that you chose. For example, for the us-west-2 Region, the script looks for the endpoint with name amazonaws.us-west-2.elastic-inference.runtime in the given VPC ID. If the endpoint is not found, the script creates one. Also, the script sets following attributes of the VPC endpoint to true, as required by Amazon EI:
- The script modifies the endpoint and add SG and chosen subnet if they are missing from the discovered endpoint
- The script discovers latest AWS DLAMI based on the operating system chosen by the user.
- If all steps succeed, the script launches an instance and reports the instance ID.
- The script tries to obtain public DNS name after the instance is launched and is in running state.
- Even if the instance is running, it may not be ready for accepting SSH connection and users may want to wait until the instance is fully initialized. EC2 console or AWS CLI can be used to query the initialization state, using the instance ID that is reported by the script for the newly launched instance.
What to expect when you run the tool
The example here illustrates what to expect when you run the script.
- Launch the script. The script can be launched from the command prompt as:
- $ python amazonei_setup.py –region us-west-2 –instance-type m5.xlargeAWS credentials are required to create or modify AWS resources. It uses Boto3, AWS SDK for Python. In order to be able to configure and manage AWS resources, the script needs user credentials. If the script is run without appropriate credentials, it reports the error below:
The solution is to configure AWS credentials using one of the methods described in the Amazon Boto3 documentation. After the credentials are in place, the script is able to proceed.
- Choose Operating System. The script prints informative message and prompts for choosing the OS. It also informs that entering ‘q’ causes the script to exit. Choose ‘1’ for the next step.
- Choose Accelerator size. The script discovered latest DL AMI for Ubuntu, it also discovered one key pair. If it discovers multiple key pairs, it lists those and ask the user to choose desired key pair by typing its index. In general if there are multiple eligible inputs, the script shows them as indexed list and let the user choose an item by typing its index. Thus, script lists supported accelerator sizes and lets user choose.
- Choose VPC. As illustrated, user chose option ‘1’ for Accelerator size and the script confirmed the Accelerator size chosen and proceeded to discover IAM role. Subsequently, it presents list of available VPCs.
- Launch an instance. Once user chooses the VPC ID, the script found a security group with matching inbound rules associated with chosen VPC, it also found one subnet associated with the chosen VPC ID. Additionally it found VPC endpoint for Amazon EI service. As the script has all the details to launch an EC2 instance, the script summarizes all the parameters it uses to launch the instance.
- Launch and wait for the instance to reach running state. As the user typed ‘y’, the script proceeded to launch the instance. The script also printed probable SSH command. The script infers the SSH command based on the OS type, key pair chosen, and the public DNS name. Actual command differs based on location of pem file. The script also warns that the instance may not be immediately accessible via SSH, even though it is in running state. The instance needs to be initialized fully, specifically the SSH daemon needs to be started before it can accept SSH connections. If the pem file is correctly located the user should be able to access the instance and proceed with using Amazon Elastic Inference.
The setup script simplifies your launch of an EC2 instance with EI. It ensures that all settings are correctly configured and instance is launched with requisite permissions to use EI. If you have any feedback about this blog post, feel free to use the comment section on this page.
About the Authors
Eftiquar Shaikh is Senior Software engineer with AWS AI. He works on building AWS services in AI space. When he is not programming, he likes to read, run and travel.
Satadal Bhattacharjee is Principal Product Manager with AWS AI. He leads the Machine Learning Engine PM team working on projects such as SageMaker Neo, AWS Deep Learning AMIs, and AWS Elastic Inference. For fun outside work, Satadal loves to hike, coach robotics teams, and spend time with his family and friends.