Networking & Content Delivery
Configuring Amazon Application Recovery Controller zonal autoshift observer notifications
Configuring Amazon Application Recovery Controller (ARC) zonal autoshift observer notifications can be an enabler on your journey to architecting and operating resilient workloads on AWS, a key focus of the reliability pillar of AWS Well-Architected. AWS provides an array of design principles and services to guide users in building highly available and fault-tolerant architectures. A crucial component of a sound resilience strategy is the ability for workloads to withstand potential Availability Zone (AZ) impairment within an AWS Region.
In this post, discover how you can use the ARC zonal autoshift observer notifications as an additional signal for potential AZ issues. Implementing zonal autoshift observer notifications allows you to proactively monitor the health of your applications and take informed actions to make sure of their resilience and continuity of service. Let’s learn how you can enable these notifications in the Regions where you operate workloads and receive the notification in your preferred channel.
Zonal autoshift observer notifications
ARC allows you to rapidly recover workloads. There are two functionalities, multi-AZ recovery, which includes zonal shift and zonal autoshift, and multi-Region recovery, which includes routing control and readiness checks.
Zonal autoshift allows AWS to automatically shift traffic away from an AZ when AWS detects a potential failure there. It operates on the principle of static stability, where your application is pre-scaled across multiple AZs to handle the complete loss of capacity in any single zone. For more information on static stability, including a discussion on how Amazon architects their services for static stability, see The Amazon Builders’ Library post, Static stability using Availability Zones.
If you want to know when a zonal autoshift occurs but haven’t configured it for your resources yet, ARC now provides observer notifications. Observer notifications allow you to receive notifications for autoshifts using Amazon EventBridge events. These notifications sent to your EventBridge bus, inform you that AWS is shifting other users away from an AZ (see example zonal autoshift events). Therefore, observer notifications can be used by users as a signal to validate that their critical workloads are still operating normally or that they need to start taking recovery actions.
Operationally working with zonal autoshift observer notifications
Receiving timely notifications about potential AZ impairment is crucial for users to maintain application availability and resilience. These notifications serve as an early warning system, allowing you to proactively monitor the health of your applications and take informed actions to mitigate the impact of an AZ issue. Being alerted when AWS initiates an autoshift allows you to quickly assess the situation, coordinate incident response efforts, and enact necessary recovery procedures to make sure of service continuity.
For all workloads, we recommend integrating with the ARC zonal autoshift functionality so that AWS can automatically route traffic away from impaired AZs. For a deeper dive into zonal autoshift, see the post, Zonal autoshift – Automatically shift your traffic away from Availability Zones when we detect potential issues. We understand adopting this feature can be a longer-term journey for some teams. Therefore, we recommend that you at least enable zonal autoshift observer notifications so that you can receive signals directly from AWS when AWS has detected a potential issue in an AZ.
We recommend integrating these notifications into your standard operational incident response procedures. If you don’t have existing processes in place for handling operational incidents, then review the Responding to events section of the Operational Excellence Pillar – AWS Well-Architected. For example, you can take observer notifications and trigger a page to the relevant on-call teams for your AWS workloads. Then, operators can review their applications and determine their health and which operational runbooks they should enact as a result.
Ultimately, the goal is to get to a point where you are automating the responses to events. This would entail enabling zonal autoshift at the workload level to automatically shift traffic. Alternatively, you could have the observer notification trigger custom runbooks that initiate AZ evacuation processes.
Walkthrough: Enabling delivery of zonal autoshift observer notifications
Now that we understand the benefits and use-cases for zonal autoshift observer notifications, we can walkthrough enabling this feature and receive notifications.
Create an Amazon SNS topic for delivering notifications
We use Amazon Simple Notification Service (Amazon SNS) to send an email notification to subscribers when an observer notification event occurs. To create an Amazon SNS topic, follow the steps in the documentation on creating an Amazon SNS topic, and choose the Standard topic type.
Subscribe your email address to your new topic. For more details see Subscribing an email address to an Amazon SNS topic using the AWS Management Console. Make sure to confirm the email subscription by choosing Confirm subscription in the email from Amazon SNS.
Enable ARC zonal autoshift observer notifications
Enable the zonal autoshift observer notification in the Console.
- Open the ARC console.
- In the navigation pane, under Multi-AZ, choose Zonal autoshift.
- Choose Set up zonal autoshift observer notification.
- Choose Enable autoshift observer notification.
- For EventBridge rule name, enter a name.
- Choose the EventBridge default bus in your AWS Region, for example arn:aws:events:eu-west-1:123456789012:event-bus/default
- For the Amazon SNS topic, choose the Amazon SNS topic you created.
- Choose Complete
![Zonal autoshift observer configuration in the ARC console with an optional EventBridge rule configuration to send Autoshift In Progress and Autoshift Complete messages using Amazon SNS.](https://d2908q01vomqb2.cloudfront.net/5b384ce32d8cdef02bc3a139d4cac0a22bb029e8/2024/12/19/zonal-autoshift-observer-notification-console.png)
Figure 1: Enabling Zonal autoshift observer notification with an optional EventBridge rule.
You now have an EventBridge rule configured to match observer notification events and deliver them through Amazon SNS to your email address.
Solution overview
Observer notifications are applicable to all users in the Region, thus you only need to configure delivery in a single AWS account. However, you must enable zonal autoshift observer notifications in each AWS Region for which you want to receive notifications. This can be done manually, repeating the previous steps in each AWS Region, or using AWS CloudFormation StackSets to automate the process. In the following section we show how you can apply this configuration to all AWS Regions you want to monitor and consolidate the events to your hub Region for delivery to your Amazon SNS topic when autoshift events occur.
![An architectural diagram with zonal autoshift observer notifications configured in three Regions, each with a rule to send the notification to a hub region bus. The hub Region bus uses a second rule to send the notification using Amazon SNS.](https://d2908q01vomqb2.cloudfront.net/5b384ce32d8cdef02bc3a139d4cac0a22bb029e8/2024/12/19/ZASON.png)
Figure 2: Consolidating zonal autoshift observer notifications and sending with Amazon SNS.
This solution uses two CloudFormation templates. Using the first template, you create a CloudFormation stack with AWS Identity and Access Management (IAM) resources, roles, and policies for EventBridge and CloudFormation StackSets used in the second template. Using the second template, you create a CloudFormation StackSet where you specify the Regions you want to monitor. The StackSet in each AWS Region enables zonal autoshift notifications and creates an EventBridge rule to forward those events to your hub Region. This solution uses the EventBridge cross-Region event routing feature.
Part 1: Create the IAM Stack
Use this template to create the IAM roles and policies needed by EventBridge and CloudFormation StackSets to consolidate observer notifications to your hub Region event bus.
Figure 3: CloudFormation IAM Stack template.
To get started, follow these steps to create a new stack using the template:
- Save the template in Figure 3 to a file named ZASON-IAMStack.yaml.
- Open the CloudFormation console.
- On the navigation bar at the top of the screen, choose the Region you want to use as your hub and create the stack within.
- On the Stacks page, choose Create stack at top right, and then choose With new resources (standard).
- On the Create stack page, do one of the following:
- For Prepare template, choose Choose an existing template. Then, under Specify template, choose Upload a template file.
- Choose Choose File and choose the ZASON-IAMStack.yaml template file from your local computer.
- Choose Next to continue.
- On the Specify stack details page, type a stack name in the Stack name box.
- Leave the parameters for AdministrationRoleName and ExecutionRoleName as the defaults unless you have a requirement to change them.
- Choose Next.
- This template contains IAM resources. On the Configure stack options page for Capabilities, choose I acknowledge that AWS CloudFormation might create IAM resources with custom names to specify that you want to deploy the IAM resources in the template.
- Choose Next
- On the Review and create choose Submit to launch your stack.
Part 2: Create a StackSet for configuring zonal autoshift observer notifications in the Regions you choose
Use this template to configure zonal autoshift observer notifications and an EventBridge rule in your desired AWS Regions. This is deployed via CloudFormation StackSet.
Figure 4: CloudFormation StackSet template.
- Save the template in Figure 4 to a file named ZASON-StackSet.yaml.
- Open the CloudFormation console.
- In the navigation pane, choose StackSets.
- On the StackSets page, choose Create StackSet
- On the Choose a template page, do one of the following:
- For Permissions, for Permission model (if present), choose Self-service permissions.
- For IAM admin role ARN, choose the Administration role created with the IAM stack (default: AWSCloudFormationStackSetAdministrationRole-ZASON).
- For IAM execution role name, enter the Execution role created with the IAM stack (default: AWSCloudFormationStackSetExecutionRole-ZASON).
- For Prerequisite – Prepare template, choose Template is ready.
- For Template source, choose Upload a template file.
- Choose Choose file to select the ZASON-StackSet.yaml template file from your local computer.
- Choose Next to continue.
- On the Specify StackSet details page, type a stack name in the Stack name box.
- In the Parameters section, specify values for:
- EventBusArn: The Amazon Resource Name (ARN) of your EventBridge default bus in your hub Region. Leaving this set to None prevents the EventBridge rule from being created.
- EventBusInvokeRole: The ARN of your IAM Role with permission to invoke the default event bus in your hub Region. This role is created by the first stack.
- Choose Next to continue creating the stack.
- On the Configure StackSet options page, add any desired tags (optional) and choose Next to continue.
- On the Set deployment options page, for Account numbers enter your account number.
- For Specify regions, add the Regions you want to monitor. For opt in Region, you must enable that Region before choosing to deploy to it here.
- Choose Next to continue creating the stack.
- On the Review page, review the details of your stack.
- Choose Submit to launch your stack.
You’ve enabled zonal autoshift observer notifications in your preferred AWS Regions and configured an EventBridge rule to forward those events to your hub Region where they are sent through Amazon SNS to your preferred email address.
Routing the notifications to your preferred medium
You can choose from a variety of targets for the received zonal autoshift observer notification event. Although this post provided an option for receiving an email notification through Amazon SNS, other options could be targeting an AWS Lambda function that triggers a mechanism to the page through a third-party provider, or creating an incident ticket in your ticketing platform. Alternatively, this trigger could kick off a series of automations through AWS Step Functions. You should determine which targets are necessary for your operation’s needs. See Event bus targets in Amazon EventBridge for the full set of available targets.
Cleaning up: Remove zonal autoshift observer notification configuration
If you decide that you no longer want to keep this configuration, then remove the resources you created:
CloudFormation StackSet and stack:
- Open the CloudFormation console.
- In the navigation pane, choose StackSets.
- On the StackSets page, choose the StackSet name you created in Part 2.
- Choose Actions at top right, then choose Delete stacks from StackSet.
- In Accounts, enter your account number in Account numbers.
- In Specify regions, choose Add all regions.
- Choose Next to continue.
- On the Review page, choose Submit.
- When the deletion completes, confirm that the Stack instances tab lists no remaining instances.
- Choose Actions at top right, then choose Delete StackSet.
- In the navigation pane, choose Stacks.
- For each stack you created in Part 1, choose the stack name you specified.
- Choose Delete in the top right.
Furthermore, delete the Amazon SNS topic, subscription, EventBridge rule, and disable autoshift observer notification in your hub Region.
Conclusion
In this post we explored the operational benefits of enabling zonal autoshift observer notifications. We also walked through enabling zonal autoshift observer notifications in multiple AWS Regions and configuring Amazon SNS as a target for receiving the notifications events through EventBridge. Configuring this solution allows you to have a powerful tool for proactively monitoring the health and resilience of your AWS applications. With these event notifications your teams can receive alerts when AWS initiates an autoshift due to a potential AZ impairment, which your teams can use to quickly assess the situation and take informed actions to maintain service continuity.
We recommend integrating these notifications into your standard operational incident response procedures, such as triggering a page to the relevant on-call team, reviewing application health, or enacting specific runbooks. Eventually, your teams should enable their workloads for zonal autoshift integration so that the zonal autoshift sends all traffic away from one AZ automatically, without manual intervention.
If you’re looking for details into how Amazon builds and operates software, please check out The Amazon Builders’ Library.
About the authors