AWS Cloud Operations Blog
Automate suspension of an AWS CodePipeline release during critical events using AWS Systems Manager Change Calendar and Amazon EventBridge
In this blog post, I show you how to set up public holidays calendars using AWS Systems Manager Change Calendar and suspend your AWS CodePipeline pipelines during the critical holidays in these calendar events. For example, let’s say an application release pipeline in your AWS account builds and deploys a new version of the application after the changes are pushed to the code repository. On New Year’s Day, even if changes are pushed to the code repository, you don’t want a new build to be deployed as the team members might not be available to handle any issues or the application is business critical for New Year’s holiday.
Prerequisites
To complete the steps in this post, you need a release pipeline created using AWS CodePipeline. You can deploy this sample pipeline manually or by using this AWS CloudFormation template.
The template takes the following input parameters:
CodePipelineName — Name of the pipeline.
PublicSubnetId — Subnet ID with internet access.
SecurityGroupId — Security group ID that allows ingress and egress and HTTP and HTTPS connections.
EC2KeypairName — Name of EC2 key pair.
AmiId — AMI ID to launch the instance. The Amazon Linux 2 AMI is used by default.
To deploy the pipeline using CloudFormation, save the template’s contents to the deploycodepipeline.yaml file and execute the following command to create a CloudFormation stack. Replace the values in red with your own values.
Note: The code in this post is intended for the Linux command line interface (CLI). For Windows, replace any occurrences of \ with ^
Note: AWS CloudFormation uses a temporary session generated from your user credentials. Make sure the user who creates the stack has permissions to do so. You can also explicitly specify the role while executing the command. For information, see create-stack documentation in the AWS CLI Command Reference.
Solution overview
Systems Manager Change Calendar lets you set up date and time ranges (referred to as events) to specify when actions can be performed in your AWS account. When you create a Change Calendar entry, you are creating a Systems Manager document of the type Change Calendar that contains iCalendar 2.0 data in plaintext format. A Change Calendar entry can be one of two types:
- Open by default
(DEFAULT_OPEN): Actions that are tracking Change Calendar can run by default but are blocked from running during associated events. During events, the state of aDEFAULT_OPENcalendar isCLOSED. - Closed by default
(DEFAULT_CLOSE): Actions that are tracking Change Calendar do not run by default but can run during events associated with the calendar entry. During events, the state of aDEFAULT_CLOSEDcalendar isOPEN.
Let’s use the software release example. On New Year’s Day, you don’t want a new build of your application to be released even if new changes are pushed to the code repository. Deployment on the other days of the week is fine. In other words, there is only one day when deployment must be blocked.
You create a Change Calendar entry of DEFAULT_OPEN that allows deployments and then you create an event for New Year’s Day, which changes the calendar state to CLOSED during the event. You can monitor this change in the default state of the calendar with Amazon EventBridge rules that trigger the configured workflows.
In this blog post, you’ll create a public calendar of DEFAULT_OPEN with public holidays as calendar events. Using Amazon EventBridge, you’ll track the CLOSED events, causing a change in the calendar’s state and triggering an automation workflow to suspend the pipeline deployment.
The solution described in this blog post includes the following steps:
- Create a Systems Manager Change Calendar entry using the
DEFAULT_OPENtype. - Create a Systems Manager Automation document to suspend the pipeline release.
- Create an AWS Identity and Access Management (IAM) policy and a role to delegate permissions to the Systems Manager Automation document.
- Create an IAM role to delegate permissions to EventBridge events, which invokes the Systems Manager Automation document.
- Create an EventBridge rule to monitor the calendar’s state and trigger the Automation document when the calendar state changes.
Step 1: Create a Systems Manager Change Calendar entry using the DEFAULT_OPEN type
When you create a Change Calendar entry, you are creating a Systems Manager document of the type ChangeCalendar. These documents can be created using either the AWS Systems Manager console or the CreateDocument API call. You can copy the example public calendar documents in this GitHub repository to your local machine and create Change Calendar entries in your AWS account.
In this blog post, I use this public holidays calendar. First, copy the contents of any calendar file (.ics) and save the contents locally as SampleHolidays2021_AWS.ics. Then use the create-document AWS CLI command to create the calendar in your AWS account. For example:
After you create the calendar, you can review it in the Systems Manager console.
Step 2: Create a Systems Manager Automation document to suspend the pipeline release
In AWS CodePipeline, a pipeline has multiple stages. Each stage includes actions to be performed on the application artifacts. When the pipeline execution moves from one stage to another, it’s called a transition. You can disable a stage’s inbound transition to prevent executions from entering that stage.
To enable or disable this transition, create a Systems Manager Automation document, using the source to change your pipeline’s desired stage after taking input from the user. This document enables or disables the pipeline transition using EnableStageTransition and DisableStageTransition API calls.
The document takes the following input parameters:
AutomationAssumeRole — IAM role to execute the Automation document.
PipelineName — Name of the pipeline for which you want to enable or disable transitions.
DesiredState — Whether to enable or disable transition of the artifacts.
StageName — Name of the stage where you want to modify the inbound or outbound transition of artifacts.
Transition — Specifies whether artifacts are prevented from transitioning into the stage and being processed by the actions in that stage (inbound) or prevented from transitioning from the stage after they have been processed by the actions in that stage (outbound).
Reason — Reason for modifying the transition state.
description: >
# Suspend CodePipeline
This document suspends the transition of the pipelines stages created in AWS
CodePipeline
## Parameters
1. **_AutomationAssumeRole_**: (Optional) Provide the AWS IAM role that
automation will use for the execution of the document. For more information,
visit the
[link](https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-setup.html)
2. **_PipelineName_**: (Required) Provide the name of the pipeline in which
you want to disable the flow of artifacts from one stage to another.
3. **_StageName_**: (Required) The name of the stage where you want to disable
the inbound or outbound transition of artifacts.
4. **_TransitionType_**: (Required) Specifies whether artifacts are prevented
from transitioning into the stage and being processed by the actions in that
stage (inbound), or prevented from transitioning from the stage after they
have been processed by the actions in that stage (outbound). Allowed values
are
* *Inbound*
* *Outbound*
5. **_Reason_**: (Required) The reason given to the user that a stage is
disabled, such as waiting for manual approval or manual tests. This message is
displayed in the pipeline console UI.
6. **_DesiredState_**: (Required) Whether to enable or disable transition of the artifacts. Allowed values
are
* *Enable*
* *Disable*
---
assumeRole: '{{AutomationAssumeRole}}'
schemaVersion: '0.3'
parameters:
AutomationAssumeRole:
default: ''
description: >-
(Optional) IAM role which Automation will assume to execute this automation. For more
information, visit -
https://docs.aws.amazon.com/systems-manager/latest/userguide/automation-setup.html
type: String
PipelineName:
description: >-
(Required) Provide the name of the pipeline in which you want to disable
the flow of artifacts from one stage to another.
type: String
StageName:
description: >-
(Required) The name of the stage where you want to disable the inbound or
outbound transition of artifacts.
type: String
TransitionType:
allowedValues:
- Inbound
- Outbound
description: >-
(Required) Specifies whether artifacts are prevented from transitioning
into the stage and being processed by the actions in that stage (inbound),
or prevented from transitioning from the stage after they have been
processed by the actions in that stage (outbound)
type: String
DesiredState:
allowedValues:
- Enable
- Disable
description: Desired state of the pipeline
type: String
Reason:
description: >-
(Required) The reason given to the user that a stage is disabled, such as
waiting for manual approval or manual tests. This message is displayed in
the pipeline console UI.
type: String
default: Disabled due to Calendar state change
mainSteps:
- name: BranchingOnDesiredState
action: 'aws:branch'
inputs:
Choices:
- StringEquals: Enable
Variable: '{{DesiredState}}'
NextStep: EnablingPipelineTransition
Default: SuspendingPipelineTransition
onFailure: Abort
isCritical: true
isEnd: true
- name: SuspendingPipelineTransition
action: 'aws:executeAwsApi'
maxAttempts: 3
inputs:
pipelineName: '{{PipelineName}}'
reason: '{{Reason}}'
stageName: '{{StageName}}'
Service: codepipeline
Api: DisableStageTransition
transitionType: '{{TransitionType}}'
isCritical: true
isEnd: true
- name: EnablingPipelineTransition
action: 'aws:executeAwsApi'
maxAttempts: 3
inputs:
pipelineName: '{{PipelineName}}'
stageName: '{{StageName}}'
Service: codepipeline
Api: EnableStageTransition
transitionType: '{{TransitionType}}'
isCritical: true
isEnd: true
Note: The Reason input parameter in the Automation document cannot be blank while suspending the pipeline. See DisableStageTransition in the AWS CodePipeline API Reference.
To create the Automation document, save the contents of this Automation document to a file named suspendPipelineAutomation.yaml. Save the file locally and then execute the following command:
Step 3: Create IAM policies and a role to delegate permissions to the Systems Manager Automation document
Next, create an IAM policy that provides the permissions required to enable and disable the pipeline stage transition. Copy and save the following contents to a file named codepipelinePolicy.json. Replace the ARN in red with the ARN of your pipeline.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"codepipeline:EnableStageTransition",
"codepipeline:DisableStageTransition"
],
"Resource": "arn:aws:codepipeline:eu-west-1:0123456789012:myPipeline/*"
}
]
}
To use this policy, create an IAM role, AutomationExecutionRoleForCodePipeline, and then attach the policy to the role. This role has a trust relationship with AWS Systems Manager. Save the contents of the policy to a file named assumeRolePolicySSM.json.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ssm.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
Execute the following command to create the IAM role. Make a note of the ARN.
Use the following command to attach CodepipelineTransitionPolicy to the role.
Step 4: Create an IAM role to delegate permissions to EventBridge
Create a second IAM role, EventBridgeInvocationRole. EventBridge uses this role to start the automation workflow. This role has two custom IAM policies:
- CodePipelineStartAutomation, which allows EventBridge to start the automation when a Change Calendar state change event occurs.
- CodePipelineAutomationPassRole, which allows EventBridge to pass the AutomationExecutionRoleForCodePipeline role to Systems Manager Automation.
Here is the content of the CodePipelineStartAutomation policy. Replace the values in red with the ARN of the Automation document you created in step 2 and save the contents locally in a file named codePipelineStartAutomation.json. For more information about Systems Manager resource ARNs, see Resources in the AWS Systems Manager User Guide.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "ssm:StartAutomationExecution",
"Resource": "arn:aws:ssm:eu-west-1:0123456789012:automation-definition/SuspendCodePipeline:$DEFAULT"
}
]
}
Here is the content of the CodePipelineAutomationPassRole policy. Replace the values in red with the ARN of the IAM role you created in step 3 and save the contents of the policy in a file named codePipelineAutomationPassRole.json.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "iam:PassRole",
"Resource": "arn:aws:iam::0123456789012:role/AutomationExecutionRoleForCodePipeline"
}
]
}
Copy and save the contents of the trust relationship with EventBridge in a file named eventbridgeAssumeRolepolicy.json.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "events.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
Use the following command to create the IAM role. Make a note of the ARN.
Attach the policies to the role.
Step 5: Create an EventBridge rule to monitor the state of the calendar and use the Automation document as a target for this rule
Next, create an EventBridge rule that watches for the event where the PublicHolidays calendar changes to the CLOSED state. Replace the ARN in red with the ARN of the PublicHolidays calendar created in step 1.
Now add the SuspendCodePipeline Automation document you created in step 3 as a target. To pass parameters to this document, you can use the Input transformer feature in EventBridge to define multiple JSON paths from the event pattern text and store those values as a variable. These variables are then passed as an input to the event target, which is the SuspendCodePipeline Automation document.
The following is the resulting JSON output of a sample event pattern when a Change Calendar entry switches to the CLOSED state.
{
"version": "0",
"id": "a36ee795-9844-7e8b-02c8-a737651f54f1",
"detail-type": "Calendar State Change",
"source": "aws.ssm",
"account": "0123456789012",
"time": "2020-09-17T18:03:01Z",
"region": "eu-west-1",
"resources": ["arn:aws:ssm:eu-west-1:0123456789012:document/PublicHolidays"],
"detail": {
"state": "CLOSED",
"atTime": "2020-12-25T00:00:00Z",
"nextTransitionTime": "2020-12-26T00:00:00Z ",
"event": "New Year’s Day"
}
}
From this event JSON, the value of event is stored as the variable reason. In the Input transformer JSON, InputPath is used to define variables and InputTemplate is used to pass these variables to the target Automation document:
"InputTransformer": {
"InputPathsMap": {
"reason": "$.detail.event"
},
"InputTemplate": "{\"AutomationAssumeRole\":[\"arn:aws:iam::0123456789012:role/AutomationExecutionRoleForCodePipeline\"],\"PipelineName\":[\"myPipeline\"],\"StageName\":[\"Deploy\"],\"TransitionType\":[\"Inbound\"],\"DesiredState\":[\"Disable\"],\"Reason\":[\"<reason>\"]}"
To add the target to the rule, use the put-targets AWS CLI command. To pass in the configuration for the target, copy and save the following content in a file named targetdetails.json. Replace the values in red with values for your environment.
[
{
"Id": "Target",
"Arn": "arn:aws:ssm:eu-west-1:0123456789012:automation-definition/SuspendCodePipeline:$DEFAULT",
"RoleArn": "arn:aws:iam::0123456789012:role/EventBridgeInvocationRole",
"InputTransformer": {
"InputPathsMap": {
"reason": "$.detail.event"
},
"InputTemplate": "{\"AutomationAssumeRole\":[\"arn:aws:iam::0123456789012:role\/AutomationExecutionRoleForCodePipeline\"],\"PipelineName\":[\"myPipeline\"],\"StageName\":[\"Deploy\"],\"TransitionType\":[\"Inbound\"],\"DesiredState\":[\"Disable\"],\"Reason\":[<reason>]}"
}
}
]
Run the following command to create a target for the MonitoringClosedCalendarState event.
Step 6: (Optional) Create an EventBridge rule to monitor the calendar state and trigger the Automation document to start the pipeline when the calendar state reverts to default
The Automation document you created in step 2 can also be used to start the pipeline after the state of the Change Calendar returns to the default.
Create an EventBridge rule that monitors the OPEN state and triggers the Automation document.
This Automation document enables or disables the pipeline based on the value of the DesiredState input parameter. To suspend the pipeline transition, the value is set to Disable. Now the input value to the parameter should be set to Enable. Here are the contents of the targetdetails.json file. Replace the values in red with values for your environment.
[
{
"Id": "Target",
"Arn": "arn:aws:ssm:eu-west-1:0123456789012:automation-definition/SuspendCodePipeline:$DEFAULT",
"RoleArn": "arn:aws:iam::0133456789012:role/EventBridgeInvocationRole",
"InputTransformer": {
"InputPathsMap": {
"reason": "$.detail.event"
},
"InputTemplate": "{\"AutomationAssumeRole\":[\"arn:aws:iam::0123456789012:role\/AutomationExecutionRoleForCodePipeline\"],\"PipelineName\":[\"myPipeline\"],\"StageName\":[\"Deploy\"],\"TransitionType\":[\"Inbound\"],\"DesiredState\":[\"Enable\"]}"
}
}
]
Execute the following command to create a target for the MonitoringOpenCalendarState event.
Test the solution
- In the left navigation pane of the AWS Systems Manager console, choose Change Calendar.
- Navigate to the PublicHolidays calendar entry.
- On the calendar entry’s details page, choose Create event.
- On the Create scheduled event page, in Event details, enter a display name for your event (for example,
TestingCodePipelineBlog).
- In Event start date and Event end date, enter or choose a day in the format MM/DD/YYYY to start the event, and then enter a time on the specified day in the format
hh:mm:ss(hours, minutes, and seconds) to start the event.
- In Schedule time zone, choose a time zone that applies to the start and end times of the event.
- Choose Create scheduled event and wait for the event to start.
- In the left navigation pane, choose Automation and wait for the automation execution to complete.
- Open the AWS CodePipeline console and navigate to the pipeline. You’ll see that the stage transition has been disabled, as shown in Figure 2.
Note: Pipeline execution IDs may vary.
Cleanup
After you test this solution, delete the pipeline and other resources to avoid continuing charges to your account.
To delete the pipeline
If you used the CloudFormation stack to create the pipeline, follow the instructions in . Deleting the stack will delete the resources. You must delete all of the objects in the S3 buckets first.
If you created the pipeline manually, follow the steps in Clean up resources.
To delete the other resources
To delete the EventBridge rules, follow the instructions in Deleting or Disabling an EventBridge Rule. To delete the IAM role, see Deleting roles or instance profiles. To delete the Change Calendar entry, follow the instructions in Delete a Change Calendar entry.
To delete the Automation document, use the delete-document CLI command.
Conclusion
In this blog post, I showed you how to create public calendars using Systems Manager Change Calendar, suspend the pipeline during any business-critical public events, and re-enable after the calendar state reverts to the default. You might want to set up a calendar entry to allow changes only on weekdays or during work hours to ensure administrator support is available to troubleshoot failed actions or deployments. You also might want to deploy event-specific marketing or promotional campaigns in an automated way through a Change Calendar entry and an Automation runbook.
For more information, see the Using AWS Systems Manager Change Calendar to prevent changes during critical events blog post, Automation walkthroughs, and Working with runbooks.
About the Authors
Saud ul Khalid is a Cloud Support Engineer in AWS Premium Support. He specializes in AWS Systems Manager, EC2 Image Builder and Amazon EC2 Linux. In his role, he enjoys helping customers solve complex issues and building automated workflows. Outside work, he loves to cook food for his family and friends, watch movies and go on hiking adventures with his friends.
Erik Weber is a World-wide Specialist Solutions Architect for AWS Management & Governance services. He specializes in AWS Systems Manager and AWS Config. Outside of work, Erik has a passion for hiking, cooking, and biking.