In the video on the left, Rendy shows you how
to stop an EC2 instance with AWS Data Pipeline

 

In the video on the right, Rendy walks you through
creating an AWS Data Pipeline role

stop-start-ec2-instances-rendy
aws-kc-slate

What's a best practice to programmatically stop and start my Amazon Elastic Compute Cloud (Amazon EC2) instances at scheduled intervals without terminating the instances?

You can use AWS Data Pipeline to programmatically start and stop your EC2 instances at scheduled instances. Data Pipeline uses AWS technologies and runs AWS Command Line Interface (AWS CLI) commands on a set schedule with no external dependencies. Data Pipeline writes logs to Amazon Simple Storage Service (Amazon S3), and runs in the context of an IAM role, which eliminates key management requirements. Data Pipeline is also cost effective. For example, the Data Pipeline free tier can be used to stop and start instances once per day. For more information, see AWS Data Pipeline Pricing.

Note: An Amazon EC2 t1.micro instance is started as the host environment for execution of a data pipeline. EC2 instances started for this purpose run for a default timeout period of 50 minutes. All resources used to host execution of a data pipeline accrue to your account. Executing pipelines to stop and restart an EC2 instance for 100 minutes or less uses the same amount of resources as are used by letting an EC2 t1.micro instance continue to run.

You should stop and restart one or more EC2 t1.micro or larger instances for more than 100 minutes (50 minutes to stop, and 50 minutes to start) to be sure that the method described in this article doesn't consume more resources than are conserved.

Data Pipeline requires that you create a customer managed policy for the DataPipelineDefaultResourceRole role if the default policy associated with this role is the AmazonEC2RoleforDataPipelineRole AWS managed policy, because AWS managed policies are read-only. If the policy attached to your DataPipelineDefaultResourceRole is not an AWS managed policy, then you don't need to create a customer managed policy. Instead, you can edit the policy assigned to the DataPipelineDefaultResourceRole role.

Follow these procedures to set up a Data Pipeline that can programmatically start and stop EC2 instances at specified times:

Create and validate a Custom Policy for the DataPipelineDefaultResourceRole

1.   In the AWS Management Console, in the Security and Identity section, choose Identity & Access Management (IAM) to open the Identity and Access Management dashboard.

2.   Choose Policies.

3.   Choose Create Policy to open the Create Policy page.

4.   Choose the option to Create Your Own Policy.

5.   Enter a policy name to associate with the DataPipelineDefaultResourceRole, such as "DataPipelineDefaultResourceRole_EC2_Policy".

6.   Enter a description for the policy, such as "Policy associated with the DataPipelineDefaultResourceRole when starting and stopping EC2 Instances with Data Pipeline."

7.   Enter the following information into the Policy Document section for the new policy:

{
   "Version": "2012-10-17",
  "Statement": [
      { 
        "Effect": "Allow",
        "Action": [
            "s3:*", 
            "ec2:Describe*",
            "ec2:Start*", 
            "ec2:RunInstances",
            "ec2:Stop*", 
            "datapipeline:*",
            "cloudwatch:*" 
          ],
          "Resource": [
              "*" 
          ]
      }
    ]
}

Note: It's a best practice that you apply the same permissions described here to any customer-managed policy currently associated with the "DataPipelineDefaultResourceRole" role.

8.   Choose Validate Policy.

9.   After the policy is validated, choose Create Policy to create the new policy.

Attach the policy to the DataPipelineDefaultResourceRole role

1.   Enter "DataPipeline" or another appropriate prefix for the Policy Type filter expression. Check the box next to this policy, and then choose the Attach option from the Policy Actions drop-down menu.

2.   On the Attach Policy page, enter the filter expression "datapipeline," and then select the check box next to the DataPipelineDefaultResourceRole entry that is returned by the filter.

3.   Choose Attach Policy at the bottom of the page to associate the new policy with the DataPipelineDefaultResourceRole.

Create the Data Pipelines to run AWS CLI commands

1.   Open the Data Pipeline console.

2.   Choose Create New Pipeline and enter the following information to create two pipelines:
Name: For example, Start EC2 instances and Stop EC2 instances.
Description: Provide relevant details about the pipeline as needed.
Source: Choose Build using template and choose the template Run AWS CLI command.
AWS CLI command: Specify what the pipeline does. Create two pipelines, one to run the aws ec2 start-instances command and another to run the aws ec2 stop-instances command.

Note: Both ec2 start-instances and ec2 stop-instances require valid values for the --region parameter.

For example, you can use the following command to start the specified EC2 instances:

aws ec2 start-instances --instance-ids i-abcd1234 i-987a654b i-ba154f3c --region us-east-x 

This command provides the syntax used to stop the same EC2 instances you started:

aws ec2 stop-instances --instance-ids i-abcd1234 i-987a654b i-ba154f3c --region us-east-x 

Important If any instance ids passed to the --instance-ids parameter do not exist, the entire command fails and no instances are stopped or started. This will be a problem if any of the specified instance ids have been terminated. For example, if the instance id i-abcd1234 was terminated, the aws ec2 start-instances example doesn't start any of the designated instances. Consider issuing separate, semicolon-delimited commands to guard against this scenario:

aws ec2 start-instances --instance-ids i-abcd1234 --region us-east-x;
aws ec2 start-instances --instance-ids i-987a654b --region us-east-x;
aws ec2 start-instances --instance-ids i-ba154f3c --region us-east-x

For more information about using the AWS CLI to start and stop EC2 instances, see start-instances and stop-instances in the AWS CLI documentation.

Configure the Data Pipeline schedule and logging

1.   Configure each Data Pipeline schedule with appropriate scheduling information. For more information, see Scheduling Pipelines.

2. Enable pipeline-level logging at pipeline creation by specifying an Amazon S3 location in either the console or with a pipelineLogUri in the default object in the SDK or CLI. For more information see Viewing Pipeline Logs.

Note: Data Pipeline logging is not mandatory. But if it's not enabled, the console displays a warning when you perform Data Pipeline validation.

3.   Set security access options:
- IAM Roles: Choose Custom Pipeline Role: DataPipelineDefaultRole
- EC2 Instance Role: Choose DataPipelineDefaultResourceRole

Note: Data Pipeline creates the necessary IAM roles for you.

Update role permissions and activate the pipelines

1.   In the AWS Management Console, choose IAM, and then choose Roles.
– If you are using a non-managed policy, select the DataPipelineDefaultResourceRole role and edit the associated policy as described in step 7 of "Create and validate a Custom Policy for the DataPipelineDefaultResourceRole."
– Otherwise, associate the DataPipelineDefaultResourceRole with a custom policy as described in Creating a Custom Policy for the DataPipelineDefaultResourceRole. For more information, see Editing Your Pipeline.

2.   Choose Activate in the console to activate the pipelines. You can monitor Data Pipeline activities in the console to verify that actions are completed successfully and on schedule.


Did this page help you? Yes | No

Back to the AWS Support Knowledge Center

Need help? Visit the AWS Support Center.

Published: 2015-05-21

Updated: 2018-08-24