AWS Cloud Operations Blog

Automate Running Tasks Using Amazon EC2 Systems Manager Maintenance Windows

In Amazon EC2 Systems Manager, a maintenance window defines a specific set of tasks, along with a set of managed instances where those tasks should be run and the schedule for when the tasks should run. Each task also has a velocity and error threshold defined (for example, run the task on at most four instances at a time and stop if there are one or more errors). You can use this to automate running many common systems administration tasks to ensure they run when needed and that you get notified about any problems running the tasks.

In this post, I discuss how maintenance windows work and provide a walkthrough for setting one up.

Maintenance window overview

You can consider the Maintenance Window capability of Systems Manager to be a replacement for tools like cron or Windows Task Scheduler. Instead of scheduling tasks on individual instances, you can use a feature that’s fully integrated with AWS and the rest of Systems Manager that also provides a central access point for task history and notification support.

Here are some of the benefits:

  • Schedule and duration
  • Targets
  • Tasks
  • History

Schedule and duration

The schedule controls when tasks run. The schedule can be defined as either a cron or a rate expression. There is also a duration that defines how long the maintenance window runs (1–23 hours).

It’s important to note that a maintenance window never stops any running tasks when it reaches the end of the duration; any started task runs to completion. However, when the duration closes, the maintenance window does stop starting any remaining registered tasks.

A cutoff period parameter controls the time after which new tasks aren’t started. For instance, if a maintenance window has a duration of 4 hours and a cutoff period of 1 hour, tasks are started up until 3 hours in.

Targets

A target defines one or more instances for which tasks can be registered. That way, you control the instances on which tasks can operate. This is an optional feature.

Define a target by either specifying one or more instance IDs or using tags. Targeting using instance IDs is frequently useful when you’re working with long-running instances, such as a database server. Using tags allows you to target instances dynamically. Every time the tasks run, instances that match the specified tag are identified and used as the targets. This can be helpful in scenarios where you have many instances to target or want to make sure to always run the task for any existing instances. For example, you may have instances in Auto Scaling groups.

Tasks

Tasks define the actions to be performed when a schedule triggers. If the maintenance window is configured to require registered targets, the task can run for one or more targets. If a task that uses specific instance IDs and the maintenance window requires registered targets, then the task’s target instance IDs must be registered with the maintenance window as well. If registered targets are not required, you can pick any managed instance for the task, by specifying an ID or using tags.

Every task is a Run Command document where you define the parameters and how it should be invoked. When registering the task, specify the name of the document —or ARN in the case of shared documents, along with any parameters required for the document. Parameters are specified as a JSON string and the exact format depends on the document you are using. For the AWS CLI, you can also use the CLI shorthand syntax.

Task priority determines the order in which tasks are run, when multiple tasks are registered. The lower the number, the higher the priority (for example, a task with a priority of 3 runs before one with a priority of 5). If two tasks have the same priority, they run in parallel and to completion before tasks at the next priority level start. This helps to ensure that the most important tasks are run first, in case there are more tasks than can be run in the allotted time.

Finally, you also control how many instances the task should run on at the same time and how many errors to allow before stopping the task. For example, you can run the task for at most two instances at a time, and stop if a single task invocation fails. In this context, an error is anything Run Command considers a failed command invocation (for example, if the application or script invoked returns a non-zero exit code).

History

Every time registered tasks run, Systems Manager captures the results so you can review the history. Amazon CloudWatch Events is also supported, so you can get notifications about the maintenance window and task execution.

A maintenance window also contains the history of all the times it has run. This allows you to see the results of every task that ran, the corresponding task invocations, and the corresponding Run Command invocation. The history provides a central place to see the status of every task run for the last 30 days. You don’t have to log in to individual instances to see task status of scheduled tasks, as with cron or Windows Task Scheduler.

Getting started with maintenance windows

The sections below show you how to create a maintenance window, register targets and tasks, and view the history of the maintenance window using the AWS CLI and the AWS Tools for PowerShell. All of these operations can also be done using the AWS Management Console.

Prerequisites

Before you can start using maintenance windows, there are some prerequisites that must be configured:

  1. Your instances must be configured with the SSM agent and an appropriate instance profile so that they can call the Systems Manager API. For more information, see Installing SSM Agent.
  2. An IAM role must be created for Maintenance Windows to assume when running tasks on your behalf. This role must be specified when registering a task. For more information, see Configuring Roles and Permissions for Maintenance Windows.
  3. Users registering tasks with maintenance windows must have the IAM PassRole permission. For more information, see Configuring Roles and Permissions for Maintenance Windows.

Create a maintenance window

The Systems Manager documentation describes how to create a maintenance window using the console or AWS CLI. The following PowerShell example shows how to create a maintenance window that runs every Tuesday at 4PM with a cut-off period of 1 hour:

PS > New-SSMMaintenanceWindow -Name "Every-Tuesday" `
    -Schedule "cron(0 16 ? * TUE *)" -Duration 4 -Cutoff 1 `
    -AllowUnassociatedTarget $false

Register targets

When registering a target, you can specify the optional owner-information field, which is passed along in all CloudWatch events raised for this target. This can be helpful when you’re processing events.

Registering a target using tags

AWS CLI

> aws ssm register-target-with-maintenance-window
   --window-ID mw-0123456789abcdef
   --targets "Key=tag:Project,Values=Website"
   --owner-information "Website instances"
   --resource-type "INSTANCE"

AWS Tools for PowerShell

PS> $tagTargets = `
@{Key="tag:Project";Values=@("Website")}
PS> Register-SSMTargetWithMaintenanceWindow `
-WindowId "mw-0123456789abcdef" `
-Target $tagTargets `
-OwnerInformation "Website instances" `
-ResourceType "INSTANCE"

Registering a target using instance IDs

AWS CLI

> aws ssm register-target-with-maintenance-window
   --window-ID mw-0123456789abcdef
   --targets "Key=InstanceIds,Values=i-0123456,i-789abcd"
   --owner-information "Database servers"
   --resource-type "INSTANCE"

AWS Tools for PowerShell

PS> $instanceIdTargets = `
@{Key="InstanceIds";Values=@("i-0123456","i-789abcd")}
PS> Register-SSMTargetWithMaintenanceWindow `
    -WindowId "mw-0123456789abcdef" `
    -Target $instanceIdTargets `
    -OwnerInformation "Database servers" `
    -ResourceType "INSTANCE"

Register tasks

Each task requires an associated role that the maintenance window can assume. In most cases, you can create a single role to use for all tasks. There may also be scenarios where you need to create specific roles for specific tasks, to control more closely the instances and documents that the role is allowed to use. For more information about how to set up a general role to use for tasks in maintenance windows, see Configuring Roles and Permissions for Maintenance Windows.

Registering a task for a target

AWS CLI

> aws ssm register-task-with-maintenance-window 
    --window-ID mw-063b7d9dcd2fac2f0 
    --targets "Key=WindowTargetIds,Values=abcdef012-3456-789a-bcde-f0123456789a" 
    --task-arn "AWS-RunPowerShellScript" 
    --service-role-arn "arn:aws:iam::123456789012:role/mw-task-role" 
    --task-type "RUN_COMMAND" 
    --task-parameters '{\"commands\":{\"Values\":[\"ipconfig.exe\"]}}' 
    --max-concurrency 1 
    --max-errors 1 
    --priority 1

AWS Tools for PowerShell

PS> $parameters = @{}
PS> $parameterValues = New-Object Amazon.SimpleSystemsManagement.Model.MaintenanceWindowTaskParameterValueExpression
PS> $parameterValues.Values = @("ipconfig.exe")
PS> $parameters.Add("commands", $parameterValues)
PS> Register-SSMTaskWithMaintenanceWindow `
    -WindowId mw-063b7d9dcd2fac2f0 `
    -Target @{ Key="WindowTargetIds";Values="abcdef012-3456-789a-bcde-f0123456789a" } `
    -TaskArn "AWS-RunPowerShellScript" `
    -ServiceRoleArn "arn:aws:iam::123456789012:role/mw-task-role" `
    -TaskType "RUN_COMMAND" `
    -TaskParameter $parameters `
    -MaxConcurrency 1 `
    -MaxError 1 `
    -Priority 1

Registering a task for instance IDs

AWS CLI

> aws ssm register-task-with-maintenance-window 
    --window-ID mw-063b7d9dcd2fac2f0 
    --targets "Key= InstanceIds,Values=i-0123456,i-789abcd" 
    --task-arn "AWS-RunPowerShellScript" 
    --service-role-arn "arn:aws:iam::123456789012:role/mw-task-role" 
    --task-type "RUN_COMMAND" 
    --task-parameters '{\"commands\":{\"Values\":[\"ipconfig.exe\"]}}' 
    --max-concurrency 1 
    --max-errors 1 
    --priority 1

AWS Tools for PowerShell

PS> $parameters = @{}
PS> $parameterValues = New-Object Amazon.SimpleSystemsManagement.Model.MaintenanceWindowTaskParameterValueExpression
PS> $parameterValues.Values = @("ipconfig.exe")
PS> $parameters.Add("commands", $parameterValues)
PS> Register-SSMTaskWithMaintenanceWindow `
    -WindowId mw-063b7d9dcd2fac2f0 `
    -Target @{ Key="InstanceIds";Values="i-0a00def7faa94f1dc" } `
    -TaskArn "AWS-RunPowerShellScript" `
    -ServiceRoleArn "arn:aws:iam::123456789012:role/mw-task-role" `
    -TaskType "RUN_COMMAND" `
    -TaskParameter $parameters `
    -MaxConcurrency 1 `
    -MaxError 1 `
    -Priority 1

View history

The history for a maintenance window consists of the following levels of detail:

  • Each execution occurrence (one entry each time it runs)
  • Each task that was run in each occurrence (one record for each registered task)
  • Each task invocation (one record for each defined target)
    Note: A task with a list of instance IDs as the target turns into a single invocation.
  • The actual Run Command invocation details

Look at the maintenance window execution history

The WindowExecutionId field uniquely identifies a particular execution of a maintenance window and the StartTime field represents the time that the maintenance window was run. The execution ID can be used to get more detail on the tasks run in the maintenance window.

AWS CLI

> aws ssm describe-maintenance-window-executions
    --window-ID mw-063b7d9dcd2fac2f0

The command returns something similar to the following:

{
  "WindowExecutions": [
    {
      "Status": "SUCCESS",
      "WindowId": "mw-063b7d9dcd2fac2f0",
      "WindowExecutionId": "75d4e9fb-20db-4c30-bf73-1a2514f59265",
      "StartTime": 1479046329.861
    },
    {
      "Status": "SUCCESS",
      "WindowId": "mw-063b7d9dcd2fac2f0",
      "WindowExecutionId": "c8a2916b-e1de-46fb-800c-c41a9fd4f63f",
      "StartTime": 1479047229.861
    },
    {
      "Status": "SUCCESS",
      "WindowId": "mw-063b7d9dcd2fac2f0",
      "WindowExecutionId": "16551bd0-df3d-4f9d-827c-15a643492ddd",
      "StartTime": 1479047588.543
    }
  ]
}

AWS Tools for PowerShell

PS > Get-SSMMaintenanceWindowExecutionList `
     -WindowId mw-063b7d9dcd2fac2f0

The command returns something similar to the following:

EndTime           : 3/14/2017 17:13:07
StartTime         : 3/14/2017 16:01:40
Status            : SUCCESS
StatusDetails     :
WindowExecutionId : 75d4e9fb-20db-4c30-bf73-1a2514f59265
WindowId          : mw-063b7d9dcd2fac2f0

EndTime           : 3/21/2017 17:02:02
StartTime         : 3/21/2017 16:00:27
Status            : SUCCESS
StatusDetails     :
WindowExecutionId : c8a2916b-e1de-46fb-800c-c41a9fd4f63f
WindowId          : mw-063b7d9dcd2fac2f0

EndTime           : 3/28/2017 17:22:16
StartTime         : 3/28/2017 16:02:33
Status            : SUCCESS
StatusDetails     :
WindowExecutionId : 16551bd0-df3d-4f9d-827c-15a643492ddd
WindowId          : mw-063b7d9dcd2fac2f0

Look at the tasks performed by a particular maintenance window execution

In the examples above, a single task was run in the maintenance window. To get more detailed information about the task, use the TaskExecutionId field.

AWS CLI

> aws ssm describe-maintenance-window-execution-tasks 
    --window-execution-ID 75d4e9fb-20db-4c30-bf73-1a2514f59265

The command returns something similar to the following:

{
    "WindowExecutionTaskIdentities": [
        {
            "Status": "SUCCESS",
            "TaskArn": "AWS-RunPowerShellScript",
            "StartTime": 1479046330.269,
            "TaskType": "RUN_COMMAND",
            "EndTime": 1479046351.331,
            "WindowExecutionId": "75d4e9fb-20db-4c30-bf73-1a2514f59265",
            "TaskExecutionId": "219d5326-20e5-4014-b7be-1dc13aa9e85f"
        }
    ]
}

AWS Tools for PowerShell

PS > Get-SSMMaintenanceWindowExecutionTaskList `
    -WindowExecutionId 75d4e9fb-20db-4c30-bf73-1a2514f59265

The command returns something similar to the following:

EndTime           : 3/14/2017 17:13:07
StartTime         : 3/14/2017 16:01:40
Status            : SUCCESS
StatusDetails     :
TaskArn           : AWS-RunPowerShellScript
TaskExecutionId   : 219d5326-20e5-4014-b7be-1dc13aa9e85f
TaskType          : RUN_COMMAND
WindowExecutionId : 75d4e9fb-20db-4c30-bf73-1a2514f59265

Look at the invocations for a particular task

In addition to the detailed information about the parameters used for the invocation, the ExecutionId field also contains the command ID that can be used to get detailed information from Run Command.

AWS CLI

> aws ssm describe-maintenance-window-execution-task-invocations 
    --window-execution-ID 75d4e9fb-20db-4c30-bf73-1a2514f59265 
    --task-ID 219d5326-20e5-4014-b7be-1dc13aa9e85f

This command returns specific information about the Run Command invocation:

{
    "WindowExecutionTaskInvocationIdentities": [
        {
            "Status": "SUCCESS",
            "Parameters": "{\"documentName\":\"AWS-RunPowerShellScript\",\"instanceIds\":[\"i-09a618aec652973a9\",\"i-0a00def7faa94f1dc\",\"i-0fff3aab684d01b23\"],\"parameters\":{\"commands\":[\"ipconfig.exe\"]},\"maxConcurrency\":\"2\",\"maxErrors\":\"
3\"}",
            "ExecutionId": "654eb74d-d1bc-4c32-8d5f-98d1dc602e94",
            "InvocationId": "824d7b43-cd1c-4f9d-927b-97f147acbfc9",
            "StartTime": 1479046330.393,
            "EndTime": 1479046351.071,
            "WindowExecutionId": "75d4e9fb-20db-4c30-bf73-1a2514f59265",
            "TaskExecutionId": "219d5326-20e5-4014-b7be-1dc13aa9e85f"
        }
    ]
}

AWS Tools for PowerShell

PS > Get-SSMMaintenanceWindowExecutionTaskInvocationList ` 
    -WindowExecutionId 75d4e9fb-20db-4c30-bf73-1a2514f59265 `
    -TaskId 219d5326-20e5-4014-b7be-1dc13aa9e85f

This command returns specific information about the Run Command invocation:

EndTime           : 3/14/2017 17:13:07
ExecutionId       : 654eb74d-d1bc-4c32-8d5f-98d1dc602e94
InvocationId      : 824d7b43-cd1c-4f9d-927b-97f147acbfc9
OwnerInformation  :
Parameters        : {"documentName":"AWS-RunPowerShellScript","commands":["ipconfig.exe"],"maxConcurrency":"1","maxErrors":"1"}
StartTime         : 3/14/2017 16:01:40
Status            : SUCCESS
StatusDetails     : Incomplete
TaskExecutionId   : 219d5326-20e5-4014-b7be-1dc13aa9e85f
WindowExecutionId : 75d4e9fb-20db-4c30-bf73-1a2514f59265
WindowTargetId    :

Get the details of a Run Command invocation

To get the specific results from the Run Command invocation (for instance the output of the command), you use the ExecutionId field from the task invocation above with the Run Command ListCommands API.

AWS CLI

> aws ssm list-commands --command-ID 654eb74d-d1bc-4c32-8d5f-98d1dc602e94

The command returns specific details from Run Command.

{
    "Commands": [
        {
            "Comment": "",
            "Status": "Success",
            "MaxErrors": "3",
            "Parameters": {
                "commands": [
                    "ipconfig.exe"
                ]
            },
            "ExpiresAfter": 1479049930.641,
            "ServiceRole": "",
            "DocumentName": "AWS-RunPowerShellScript",
            "TargetCount": 3,
            "OutputS3BucketName": "",
            "NotificationConfig": {
                "NotificationArn": "",
                "NotificationEvents": [],
                "NotificationType": ""
            },
            "CompletedCount": 3,
            "Targets": [],
            "StatusDetails": "Success",
            "ErrorCount": 0,
            "OutputS3KeyPrefix": "",
            "RequestedDateTime": 1479046330.641,
            "CommandId": "654eb74d-d1bc-4c32-8d5f-98d1dc602e94",
            "InstanceIds": [
                "i-09a618aec652973a9",
                "i-0a00def7faa94f1dc",
                "i-0fff3aab684d01b23"
            ],
            "MaxConcurrency": "2"
        }
    ]
}

AWS Tools for PowerShell

PS > Get-SSMCommand –CommandId 654eb74d-d1bc-4c32-8d5f-98d1dc602e94
The command returns specific details from Run Command.
CommandId          : 654eb74d-d1bc-4c32-8d5f-98d1dc602e94
Comment            :
CompletedCount     : 3
DocumentName       : AWS-RunPowerShellScript
ErrorCount         : 0
ExpiresAfter       : 3/14/2017 16:01:40
InstanceIds        : {"i-09a618aec652973a9","i-0a00def7faa94f1dc","i-0fff3aab684d01b23"}
MaxConcurrency     : 2
MaxErrors          : 3
NotificationConfig : Amazon.SimpleSystemsManagement.Model.NotificationConfig
OutputS3BucketName :
OutputS3KeyPrefix  :
OutputS3Region     :
Parameters         : {[Operation, Amazon.Runtime.Internal.Util.AlwaysSendList`1[System.String]], [commands,                    Amazon.Runtime.Internal.Util.AlwaysSendList`1[System.String]]}
RequestedDateTime  : 3/24/2017 05:37:03
ServiceRole        :
Status             : Success
StatusDetails      : Success
TargetCount        : 3
Targets            : {}

Summary

In this post, I described some of the benefits and helped you get started creating maintenance windows with AWS CLI and PowerShell. Look at the Systems Manager documentation for more details and keep an eye out for more blog posts about how you can use maintenance windows to automate systems management tasks.
If you have questions or suggestions, please comment below.

About the Author

Mats Lannér is a Software Development Manager on the Amazon EC2 Systems Manager team, responsible for the Maintenance Window, Patch Manager and Document capabilities. When not working on Systems Manager he enjoys working and hiking with his two German Shepherds.