Using AWS Systems Manager Change Calendar to prevent changes during critical events

When applications and their development teams grow or evolve around service-specific independent teams, coordination and visibility among projects, deployments and operational events become a critical issue. AWS customers use multiple AWS accounts for many different reasons, adding guardrails around resources and increasing the need for collaboration between different AWS account owners and teams.

Previously, these teams had to use general purpose calendar and email tools to inform their organization of upcoming events. These central teams did not have control to prevent inadvertent changes during critical events and had to rely on the rest of the organization to follow change-control guidance.

With AWS Systems Manager Change Calendar, a fully managed service, customers can create calendars and define important business events.

Overview

Change Calendar can be one of two types: open calendar which allows actions by default or closed calendar which denies actions by default. Customers can define their events on the calendar to prevent or accept potentially disruptive actions during these critical events based on the state returned by the calendar.

For customers that use multiple AWS accounts, Change Calendar provides the flexibility to create a single master change control calendar that can be enforced across multiple AWS accounts, or lets each department maintain its own calendar for AWS account-level events.

This blog demonstrates how Change Calendar can be used to prevent a maintenance window from being triggered because it might impact a business event. The maintenance window regularly stops Amazon EC2 instances in the demo environment to save costs. This example sets up a calendar event for a customer demo and shows how an upcoming customer demo meeting represented by a calendar event in Change Calendar can be used to stop the execution of the regular maintenance operation.

AWS Systems Manager Change Calendar

Change Calendar supports multiple calendars per AWS account and calendars can be shared across multiple AWS accounts. Changes to the calendar itself are audited and users can get notifications and reminders for upcoming calendar events. With Change Calendar, AWS Systems Manager enhances operational safety during business events.

For example, a customer would like to limit certain disruptive Automation document executions to certain time periods. If the automation is being executed during a time conflicting with the desired calendar(s)’ state the automation action does not proceed with the remaining steps. This is achieved by leveraging Change Calendar service’s GetCalendarState within automation.

Calendars are created as iCalendar documents in the AWS Systems Manager Documents service, as shown in the following screenshot.

There are two different Change Calendar types:

• Open by default: Returns OPEN state by default and CLOSED state if it has an event falling on the requested time period.
• Closed by default: Returns CLOSED state by default and OPEN state if it has an event during a selected time period.

Change Calendar State – Change Calendar state can either be OPEN or CLOSED. Its state depends on the default state, and whether or not there are any events currently active. An active event will flip the default state (OPEN to CLOSED or CLOSED to OPEN). Open by default calendar works as shown in the following diagram.

In a closed by default calendar actions are blocked from running except during a scheduled event, as shown in the following diagram.

Calendar Events

Calendar events are items in Change Calendar that represent critical organizational events or launches. You create them in the same way as you would create any other (single or recurring) calendar item. Customers can create new events via the AWS Systems Manager console or the AWS CLI.

Change Calendar in Action

Customers would like to limit certain automation document executions to certain time periods. If the automation is being executed during a time conflicting with the desired calendar(s) state the automation action does not proceed with the remaining steps. This can be achieved by leveraging Change Calendar service’s GetCalendarState within automation.

This example, uses a maintenance window with an automation task to run regularly and stop demo EC2 instances, as shown in the following screenshot.

The example uses the following custom automation workflow to get the state information from the calendar with aws:assertAwsResourceProperty action:

Automation Document

schemaVersion: "0.3"
parameters:
  SNSTopicArn:
    type: String
    description: "(Required) The SNS topic ARN used to send pending approval notification for Admins. The SNS topic name must start with Automation."
  Approvers:
    type: StringList
    description: "(Required) IAM user or user arn of approvers for the automation action"
  InstanceId:
    type: StringList
    description: "(Required) The instance to stop using AWS-StopEC2Instance"
mainSteps:
- name: checkChangeCalendarOpen
  action: aws:assertAwsResourceProperty # Asserts an event state for Change Calendar
  onFailure: step:notifyAdmin  # If Change Calendar state is CLOSED branch to "notifyAdmin" step
  timeoutSeconds: 600
  inputs:
    Service: ssm
    Api: GetCalendarState
    CalendarNames:   ## List of calendars to check the status. 
        - arn:aws:ssm:us-east-1:123456789012:document/CustomerDemosEMEA
    PropertySelector: "$.State" # Returns OPEN / CLOSED as state
    DesiredValues:
    - OPEN   
  nextStep: stopInstances # if Change Calendar state is OPEN “stopInstances” step is executed.
- name: notifyAdmin 
  action: aws:approve
  onFailure: Abort # If approval is denied "Abort" the execution
  inputs:
    NotificationArn: "{{ SNSTopicArn }}" # SNS topic for notifications.
    Message: Admin approval required to execute Automation on critical business event 
    MinRequiredApprovals: 1
    Approvers: "{{ Approvers }}"
  nextStep: stopInstances # If this “Approved” the Automation proceeds to "stopInstances" step.
- name: stopInstances
  action: aws:executeAutomation # AWS-StopEC2Instance document is run for provided instances
  onFailure: Abort
  timeoutSeconds: 120
  inputs:
    DocumentName: AWS-StopEC2Instance 
    RuntimeParameters:
      InstanceId: "{{ InstanceId }}"

The automation document branches to the notifyAdmin step in case the checkChangeCalendarOpen step does not return OPEN as its state. On failure of the step (in which case, the calendar failure returns CLOSED) the automation flow jumps to the notifyAdmin step.  The notifyAdmin step sends out an SNS notification to the IT administrators about the change to get their approval before aborting the maintenance operation entirely.

Summary

In the world of modern software development, adoption of microservices and devops practices transforms deployment and maintenance of applications. It’s critical that teams collaborate to ensure features are delivered in a sustainable and secure manner. In Change Calendar, before making a change or performing a maintenance operation, AWS Systems Manager capabilities now check with Change Calendar for applicable events and honor the response from the calendar.

With launch of Change Calendar, customers define important business events with specific states (OPEN or CLOSED) on the calendar to prevent potentially disruptive actions during those time frames. This provides an easy and reliable way to fully automate the process of preventing undesired changes during critical time periods.

Today, all AWS Systems Manager capabilities are integrated with Change Calendar and more AWS services will integrate later this year.

About the Author

Baris Furtinalar is a Senior Solutions Architect at AWS in Luxembourg. He is passionate about cloud computing and he believes this shift to the cloud helps companies transform their businesses, gain agility, and improve operational responsiveness. In his spare time, he loves spending time with his daughter, speeding on German Autobahns, and enjoying healthy food.