AWS Management Tools Blog

How to secure infrequently used EC2 instances with AWS Systems Manager

Many organizations have predictable spikes in the usage of their applications and services. For example, retailers see large spikes in usage during Black Friday or Cyber Monday. The beauty of Amazon Elastic Compute Cloud (Amazon EC2) is that it allows customers to quickly scale up their compute power to meet these demands. However, some customers might require more time-consuming setup for their software running on EC2 instances. Instead of creating and terminating instances to meet demand, these customers turn off instances and then turn them on again when they are needed. Eventually the patches on those instances become out of date, and they require updates.

In this blog, I cover how to leverage AWS Step Functions and AWS Systems Manager Maintenance Windows to start an EC2 instance, apply any outstanding patches using Patch Manager, then stop the EC2 instance. I also cover the necessary steps to manage these resources using AWS CloudFormation. A Maintenance Window lets you define a schedule for when to perform potentially disruptive actions on your instances, such as patching an operating system (OS), updating drivers, or installing software. Each Maintenance Window has a schedule, a duration, a set of registered targets, and a set of registered tasks. AWS Step Functions are part of the AWS Serverless Platform. They make it simple to orchestrate AWS Lambda functions for serverless applications. Finally, CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion.

Availability

Although AWS Systems Manager is currently available in all AWS Regions, CloudFormation support for Systems Manager is only available in some Regions. For a list of available Regions, please check the CloudFormationResourceSpecification.json for a given Region, which can be seen here. Similarly, Step Functions are only available in some Regions as well. The list of available Regions can be seen here.

Setup

CloudFormation works on your behalf to manage AWS resources. The CloudFormation template for this post, which outlines the resources that will be provisioned into an AWS CloudFormation stack, can be run as-is, there is no additional setup required to get started. The template creates the following resources:

  • An EC2 instance
    • The instance we are going to patch
  • A Systems Manager Maintenance Window
    • A Maintenance Window representing a timeframe in which to run our patching operation
  • A Systems Manager Maintenance Window target
    • A target on which the task will act, in this case it will be our EC2 instance
  • A Systems Manager Maintenance Window task
    • The task that we want to execute, in this case an AWS Step Function
  • A series of AWS Lambda functions
    • One Lambda function for each of the following actions: start, stop, patch, and poll our instance as well as poll the patching activity
  • An AWS Step Function
    • Manages the workflow of the Lambda Functions
  • IAM roles
    • Give proper access to all of the above mentioned resources

Here is a link (JSON)/(YAML) to the sample template. I will walk through each piece of the template step-by-step. I’ll discuss some of the common issues to look out for when trying to run this template on your own account.

CloudFormation template walkthrough

IAM roles

A Maintenance Window executes a set of tasks on a schedule without requiring user interaction through the AWS Management Console or the AWS SDK/CLI. For this blog post, the task will be a Step Function. The Maintenance Window will require an IAM role that grants it the appropriate permissions to execute the Step Function. Also, the Lambda functions and Step Function require their own roles based on the resources they are accessing. These are the IAM roles created for this blog post:

  • InstanceStateActionRoleForLambda
    • The InstanceStateActionRoleForLambda provides the necessary access to allow the Starter, Stopper, and InstancePoller Lambda functions to call an EC2 instance.
  • InstancePatchingRoleForLambda
    • The InstancePatchingRoleForLambda provides the necessary access to allow the Patcher and PatchPoller Lambda functions to call Systems Manager.
  • StepFunctionRole
    • The StepFunctionRole allows your Step Function to call each of the Lambda functions.

Lambda functions

Here is a list and short description of the Lambda functions created for this blog post:

  • InstancePatcherLambda
    • Uses AWS Systems Manager Patch Manager to scan and install any patches based on the Patch Baseline that applies to the instance (the Patch Baseline associated with the instance’s Patch Group, if defined, otherwise the default Patch Baseline for the instance’s operating system).
  • InstancePatchPollerLambda
    • Polls Systems Manager to verify the status of the patching command.
  • InstanceStarterFunction
    • Starts the EC2 instance.
  • InstanceStopperLambda
    • Stops the EC2 instance.
  • InstancePollerLambda
    • Polls Amazon EC2 to verify whether the instance is running or stopped.

Step Function

The Step Function organizes each of the Lambda functions into a workflow. The steps are laid out in a specific order, which reliably coordinates each component and steps through the workflow of the application. Also, in the Step Functions console, you can watch the Step Function operate and see each step as it moves through the workflow.

Maintenance Window

To set up a Maintenance Window using CloudFormation, you need three resources, and they must be created in the following order: a Maintenance Window, a Maintenance Window target, and a Maintenance Window task. The DependsOn attribute of CloudFormation allows us to order the creation of our resources. In the provided template, you can see that each of the Maintenance Window resources uses DependsOn to ensure this ordering. See the following excerpt:

MaintenanceWindow:
Type: 'AWS::SSM::MaintenanceWindow'
Properties:
Name: 'LambdaMW'
AllowUnassociatedTargets: 'true'
Schedule: 'rate(30 minutes)'
Duration: '2'
Cutoff: '1'
DependsOn: TestInstance

AMIInfo/AMIInfoFunction

These two resources are used only for convenience, they perform a lookup that generates the AMI-ID for the EC2 instance that we create in the stack. For more information see the documentation.

Activity walkthrough

 

Create the CloudFormation stack

Follow these steps to create the CloudFormation stack for this walkthrough.

  1. Open the AWS CloudFormation console at https://console.aws.amazon.com/cloudformation.
  2. Choose Create Stack.
  3. On the Select Template page, choose Specify an Amazon S3 template URL and give the following S3 URL: https://s3-us-west-2.amazonaws.com/management-tools-blog-cf-template-storage/InstancePatchingWithSSMTemplate.yaml.
  4. For Stack Name, give an appropriate name, choose Next.
  5. On the Options page, choose Next.
  6. On the Review page, select I acknowledge that AWS CloudFormation might create IAM resources with custom names and choose Create.

You can view the progress of CloudFormation progress by selecting your stack and choosing the Events tab. Once CloudFormation has built each resource, it will show CREATE_COMPLETE as the Status for the stack.

Follow these steps to view the Maintenance Window created in this walkthrough.

  1. Open the AWS Systems Manager console at https://console.aws.amazon.com/systems-manager.
  2. Navigate to Explore Maintenance Windows.
  3. Choose the Maintenance Window named SSMStepFunctionDemo.
  4. Choose the View Details button.
  5. Choose the History tab.

It is possible that on the first Maintenance Window execution you might see a status message stating that there were no tasks to execute. This happens because the Maintenance Window is set to run every 30 minutes. When it gets created, it attempts to run immediately, but since there is no task associated with it, the execution returns Success with the above status. The schedule of a Maintenance Window can be in the form of a cron or rate expression and since our Maintenance Window uses a rate expression, it will not run again for another 30 minutes. This issue also occurs for cron expressions that run in explicit intervals, for example cron(0/15 * * * ? *). We can restart it by disabling and re-enabling the Maintenance Window.

Here are the steps for restarting a Maintenance Window created in this walkthrough.

  1. Choose the Maintenance Window you want to restart.
  2. Choose Actions then choose Disable maintenance window.
  3. Choose Actions again then choose Enable maintenance window.

Here are the steps for viewing the Step Function created in this walkthrough.

  1. Open the AWS CloudFormation console at https://console.aws.amazon.com/cloudformation.
  2. Choose the CloudFormation stack created previously.
  3. Select the Resources tab, search for the StateMachine created by CloudFormation. Take note of the name of the StateMachine, it will look something like this SSMStepFunctionDemo-YCIADKH4IGYU.
  4. Open the AWS State Machine console at https://console.aws.amazon.com/states.
  5. Choose the StateMachine with the name from step #3.
  6. Under the list of executions for the StateMachine find the currently running execution, then click on the link for that execution in the Name column.

You can now watch as the Step Function executes each of the Lambda functions created through CloudFormation. Note that the actual updating of the patches can take some time, at least 10 to 15 minutes.

Viewing patching results

Now that the execution of the Step Function is complete, we can view the results of the Patching activity.

Here are the steps to view the Maintenance Window patching results:

  1. Open the StateMachine execution from the previous step.
  2. Under the Visual Workflow tab, Choose the step labeled Patch Instance.
  3. Under Execution Details on the right, Choose the Output tab.
  4. Copy the value of commandId.
  5. Open the AWS Systems Manager console at https://console.aws.amazon.com/systems-manager.
  6. Choose Explore Run Command.
  7. Choose the search bar at the top of the table to add a filter.
  8. Select Command Id as the filter and paste in the Command Id you copied from step #4.
  9. Select the Command.
  10. In the Output tab at the bottom, click View Output.

The output will display various pieces of information including which patches were installed, the snapshot ID for the patching snapshot, and the patch baseline used.

Verifying that the instance has stopped

After the execution of the Step Function is complete, it will stop the instance. We can see this by viewing the instance in the EC2 console.

  1. Open the EC2 console at https://console.aws.amazon.com/ec2.
  2. Choose the Instances link under the Instances tab on the left.
  3. Find the EC2 instance with the Name ssm-stepfunction-demo.

Conclusion

In this post, I have shown you how to use CloudFormation to create a Maintenance Window that executes a Step Function, which uses Patch Manager to patch long running instances that we don’t want to keep on at all times. You can use CloudFormation to easily deploy Maintenance Windows across multiple Regions to ensure that resources are properly patched and that the states of your resources are as you expect. This also greatly reduces the burden of keeping Maintenance Windows in sync across Regions, guaranteeing that the Maintenance Windows are all operating in the same manner.

For further information regarding using some of the AWS services in this blog post see the following user guides:

Maintenance Window User Guide
Patch Manager User Guide
Step Functions User Guide
Lambda User Guide
CloudFormation User Guide

About the Author

 

Spencer Glazier is a Software Development Engineer on the AWS Systems Manager team. He’s passionate about delivering products and services that enables customers to do more, faster and cheaper. In his free time he enjoys getting up early to run along the narrow streets of Seattle and spending time with his family.