How to centralize CloudWatch Alarms with Amazon EventBridge and AWS CloudFormation

Amazon CloudWatch lets customers collect monitoring and operational data in the form of logs, metrics, and events, providing an easy way to monitor and receive notifications regarding their workload health and often integrate directly with other systems, such as JIRA Service Desk and ServiceNow.

The CloudWatch alarms feature lets you monitor CloudWatch metrics and receive notifications when the metrics fall outside a certain threshold. This threshold can be a predefined static threshold, an anomaly detection model, metric math expression, and composite alarm. When you create an alarm, you can specify what actions an alarm takes when it changes the state between OK, ALARM, and INSUFFICIENT_DATA states. The most common type of alarm action is to notify one or more people by sending a message to an Amazon Simple Notification Service (Amazon SNS) topic.

An alarm can monitor various metrics, such as CPU utilization from multiple Amazon Elastic Compute Cloud (Amazon EC2) instances or network traffic statistics from the same Amazon EC2 instances and provide notifications. Although creating alarms is straightforward, it can require a management effort to respond and research these alarms in a multi-account environment. This post will show how to centralize your CloudWatch alarms in a multi-account environment with Amazon EventBridge and AWS CloudFormation.

As an example, we will provide an overview of how to create CloudWatch Alarm in Sender Accounts(A&B) and centralize it in the Receiver Account(C) by using CloudFormation to deploy AWS EventBridge Event Buses, Rules and Targets.

Solution overview

This solution will use CloudWatch alarms to trigger a notification in the Sender accounts, which, EventBridge captures and sends to the Receiver account. When you deploy your CloudFormation StackSets, you will have two deployment options as detailed below.

• You can choose to route all of the CloudWatch alarm notifications to the Receiver account, but to the same regions as that of the Alarms in the Sender accounts as indicated in Figure 1.
• You can choose to centralize all of the CloudWatch alarm notifications to the Receiver account within a single region of your choice as indicated in Figure 2.

Figure 1: Solution overview (Figure 1) shows how the default EventBuses in Sender accounts A&B captures Cloudwatch Alarm notifications from us-east-1 and us-west-1 regions and routes the notifications to the respective regions’ custom EventBuses in the Receiver account C.

Figure 2: Solution overview shows how the default EventBuses in Sender accounts A&B captures CloudWatch Alarm notifications from us-east-1 and us-west-1 regions and routes the notifications to the centralized/single region-us-east-1 custom EventBus in the Receiver account C.

Prerequisites

To complete the steps in this walkthrough, you will need the following:

• Organizations with access to the management or delegated administration account.
• An AWS account within the organizations enabled to receive the EventBridge events.
• One or more AWS accounts within the organization with CloudWatch alarms(Sender accounts).

Services
We will be using the following services in this solution:

CloudWatch collects monitoring and operational data in the form of logs, metrics, and events. Then, it provides a unified view of operational health and complete visibility of your AWS resources, applications, and services running on AWS and on-premise.

EventBridge is a serverless event bus service, that makes it easier to build event-driven applications at scale using events generated from your applications, integrated Software-as-a-Service (SaaS) applications, and AWS services.

CloudFormation is an infrastructure as code (IaC) service that lets you easily model, provision, and manage AWS and third-party resources.

AWS CloudFormation StackSets extends the functionality of stacks by enabling you to create, update, or delete stacks across multiple accounts and AWS Regions with a single operation. Using an administrator account, you define and manage a CloudFormation template, and then use the template as the basis for provisioning stacks into selected target accounts across specified AWS Regions.

AWS Organizations is an account management service that lets you consolidate multiple AWS accounts into an organization that you create and centrally manage. Organizations includes account management and consolidated billing capabilities that let you better meet the budgetary, security, and compliance needs of your business.

Getting Started

Creating CloudWatch Alarm (Optional)

We’ll first prepare an environment by creating an a CloudWatch alarm (Optional). You may skip these steps if you already have alarms configured within the Sender accounts A and (Or) B.

1. Open the CloudWatch console.
2. Select All Alarms under the Alarms section on the left navigation pane.
3. Select Create alarm on the right side of the screen.
4. Choose the metric that you would like to create an alarm on by selecting the Select metric and choosing relevant namespace.
5. Select the checkbox on the left of the metric that you would like to create an alarm for, and choose Select metric on the bottom right of your screen.
6. Enter the conditions for the alarm to be triggered, and select Next on the bottom right of your screen.
7. Click Remove in the Notification section, and select Next on the bottom right of the screen.
   Enter a name for the alarm, and select Next on the bottom right of the screen.
   Validate and confirm the Alarm details and select Create alarm on the bottom right of the screen.

CloudFormation StackSet deployment from the Management account
In this section, we provide you with the CloudFormation StackSets that should be executed from the organization management or delegated administration account, which will deploy the EventBridge rules within the Sender accounts A&B, along with the custom bus in the Receiver account C. The template also provides an (optional) CloudWatch Log group as the Receiver account target.

<Cloud Formation Template>

---
AWSTemplateFormatVersion: '2010-09-09' Description: |

Centralize Amazon CloudWatch alarms in a multi-account environment with Amazon EventBridge.

**WARNING** This template creates an Amazon CloudWatch Log group and uses Cross-account/Cross-region events with Amazon EventBridge.

You will be billed for the AWS resources used if you create a stack from this template.

Metadata: AWS::CloudFormation::Interface:

ParameterGroups:
- Label:

default: Account Details Parameters:

- ReceiverAccountIDParam - BusParam
- OrganizationIDParam

- Label:
default: Centralize Regions

Parameters:
- CentralizationParam
- CentralizationRegionParam

- Label:
default: (Optional) Add an Amazon CloudWatch Log group Target

Parameters:

-  CreateTargetParam
-  CloudWatchLogGroupParam

Parameters:
ReceiverAccountIDParam:
Description: The accountID for the Monitoring Account. Type: String
AllowedPattern: ^[0-9]{12}$
MinLength: 1
MaxLength: 12

BusParam:
Description: The name of the custom Bus in the Monitoring Account. Type: String
MinLength: 1
MaxLength: 256

OrganizationIDParam:

Type: String

AllowedPattern: ^o-[a-z0-9]+

Description: The OrgID to allow all members of your AWS Organizations to use the PutEvents.

CentralizationParam: Type: String Default: 'No' AllowedValues:

- 'Yes'

- 'No'

Description: Select 'Yes' if you would like to centralize the events on the receiver account.

CentralizationRegionParam: Type: String
Default: ''

Description: Enter the region to centralize the events on the receiver account. For example: US East (Ohio), type us-east-2. Leave this as blank if the above parameter is selected as 'NO'

CreateTargetParam: Type: String Default: 'No' AllowedValues:

- 'Yes'
- 'No'
Description: Select 'Yes' if you would like to create target .

CloudWatchLogGroupParam:
Type: String
Default: /aws/events/EB-Test
AllowedPattern: ^\/aws\/events\/[\.\-_/#A-Za-z0-9]+ MinLength: 13

MaxLength: 512

Description: Enter the CloudWatch log group name, if you would like to have a target in the receiving account. Use the prefix '/aws/events'.

Conditions: IsReceiver: !Equals

- !Ref 'ReceiverAccountIDParam'

- !Ref 'AWS::AccountId' IsSender: !Not

- !Condition 'IsReceiver' Centralize: !Equals

- !Ref 'CentralizationParam'

- 'Yes' CentralizeRegion: !Equals

- !Ref 'CentralizationRegionParam'

- !Ref 'AWS::Region' ReceiverRegion: !And

- !Condition 'IsReceiver' - !Or

- !Condition 'CentralizeRegion' - !And

- !Not
- !Condition 'Centralize'

- !Not
- !Condition 'CentralizeRegion'

CreateTarget: !And
- !Equals

- !Ref 'CreateTargetParam'

- 'Yes'
- !Condition 'ReceiverRegion'

Resources: ReceiverEventBus:

Type: AWS::Events::EventBus Condition: ReceiverRegion Properties:

Name: !Ref 'BusParam' ReceiverBusPolicy:

Type: AWS::Events::EventBusPolicy Condition: ReceiverRegion DependsOn: ReceiverEventBus Properties:

EventBusName: !Ref 'BusParam' StatementId: MyStatement Statement:

Effect: Allow Principal: '*'
Action: events:PutEvents

Resource: !Sub 'arn:${AWS::Partition}:events:${AWS::Region}:${ReceiverAccountIDParam}:event- bus/${BusParam}'

Condition:
StringEquals:

aws:PrincipalOrgID: !Sub '${OrganizationIDParam}' SenderEventRule:

Type: AWS::Events::Rule Condition: IsSender Properties:

Description: Captures CloudWatch Alarm State Changes. Name: CloudWatchAlarmMonitor
EventBusName: default
State: ENABLED
EventPattern:
source:

- aws.cloudwatch
detail-type:

- CloudWatch Alarm State Change Targets: !If

- Centralize

- - Arn: !Sub 'arn:${AWS::Partition}:events:${CentralizationRegionParam}:${ReceiverAccountI DParam}:event-bus/${BusParam}'

Id: CrossAccountDestinationBus

RoleArn: !GetAtt 'EventBridgeIAMrole.Arn'

- - Arn: !Sub 'arn:${AWS::Partition}:events:${AWS::Region}:${ReceiverAccountIDParam}:event- bus/${BusParam}'

Id: CrossRegionDestinationBus

RoleArn: !GetAtt 'EventBridgeIAMrole.Arn' ReceiverEventRule:

Type: AWS::Events::Rule Condition: ReceiverRegion DependsOn: ReceiverEventBus Properties:

Description: Captures CloudWatch Alarm State Changes. Name: CloudWatchAlarmMonitor
EventBusName: !Ref 'BusParam'
State: ENABLED

EventPattern:
source:

- aws.cloudwatch
detail-type:

- CloudWatch Alarm State Change Targets: !If

- CreateTarget

- - Arn: !Sub 'arn:${AWS::Partition}:logs:${AWS::Region}:${ReceiverAccountIDParam}:log- group:${CloudWatchLogGroupParam}'
Id: CloudWatchLogGroup - !Ref 'AWS::NoValue'

CloudWatchLogGroup:
Type: AWS::Logs::LogGroup Condition: CreateTarget Properties:

LogGroupName: !Ref 'CloudWatchLogGroupParam' CloudWatchLogResourcePolicy:

Type: AWS::Logs::ResourcePolicy Condition: CreateTarget Properties:

PolicyName: TrustEventsToSendLogEvents

PolicyDocument: !Sub '{"Version":"2012-10- 17","Statement":[{"Sid":"TrustEventsToStoreLogEvent","Effect":"Allow","Princi pal":{"Service":["events.amazonaws.com","delivery.logs.amazonaws.com"]},"Acti on":["logs:CreateLogStream","logs:PutLogEvents"],"Resource":"arn:${AWS::Parti tion}:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/events/*:*"}]}'

EventBridgeIAMrole: Type: AWS::IAM::Role Condition: IsSender Properties:

AssumeRolePolicyDocument: Version: 2012-10-17 Statement:

- Effect: Allow
Principal:

Service:
- events.amazonaws.com

Action: sts:AssumeRole Path: /

Policies:
- PolicyName: PutEventsDestinationBus

PolicyDocument: Version: 2012-10-17 Statement:
- Effect: Allow
Action:

- events:PutEvents
Resource: !If

- Centralize

- - !Sub 'arn:${AWS::Partition}:events:${CentralizationRegionParam}:${ReceiverAccountI DParam}:event-bus/${BusParam}'

- - !Sub 'arn:${AWS::Partition}:events:${AWS::Region}:${ReceiverAccountIDParam}:event- bus/${BusParam}'

<CloudFormation Template>

To deploy the above template, follow these steps:

1. Save the above template as a .yaml file.
2. Go to the CloudFormation StackSets on the left navigation pane of the CloudFormation console within the management or delegated administration account.
3. Choose Create StackSet on the top right of the screen.
4. Choose the Upload a template file option, select Choose file to select the CloudFormation template saved as mentioned above, and select Next on the bottom right of the screen.
5. On the Specify StackSet details page, provide a stack name. Ex: “Centralize-Amazon-CloudWatch-Alarms”.
6. Under Parameters, review the default parameters, and enter the required ReceiverAccountIDParam, BusParam, and OrganizationIDParam.

There are several parameters for the CloudFormation StackSet:

o   ReceiverAccountIDParam: The Receiver account within your Organization where the CloudWatch alarms will be centralized.

o   BusParam: The name of the EventBridge custom bus which will be created in the receiving Receiver account.

o   OrganizationIDParam: The Organization ID in which the CloudFormation StackSet will be run.

o   CentralizationParam: Select Yes if you would like a single region within the receiving Receiver account to accept all of the CloudWatch alarms.

o   CentralizationRegionParam: If you selected Yes above, then enter in the region on which you would like to centralize ex: us-east-1, as depicted in Figure 2 .

o   CreateTargetParam(Optional): Select Yes if you would like to have all of the events sent as logs to a CloudWatch Log Group  in the receiving Receiveraccount. Select No if you don’t want to setup a CloudWatch Log group as the target.

Note: The Receiver account can have any available targets in the EventBridge console and upto 5 targets per Rule.

o   CloudWatchLogGroupParam: If you select Yes above, then enter in the CloudWatch Log group name that you would like to use.

Figure 3: CloudFormation Create StackSet details page

7. On the Configure stack options page, you can choose to add tags, choose additional options, or just choose Next.
8. On the Set deployment options page, you can choose the deployment targets, regions, and options, then choose Next.
9. On the Review page, validate your parameters and acknowledge that AWS Identity and Access Management (IAM) resources will be created. Finally, select Submit.

Once you select Submit, you can follow the process and view the deployment status via the Events tab of the CloudFormation stack. When it finishes deploying, your environment has been setup. If you selected Yes for the CreateTargetParam parameter, then you can follow the next steps to see the CloudWatch Alarm State Change notifications within your accounts.

Note: If you chose No for the CreateTargetParam parameter, you can verify the working of the setup by verifying respective target invocation.

View Alarms notifications on CloudWatch Log Groups (Optional) in the Receiver account

If there’s no data present within the CloudWatch Log Group, then no alarms have been triggered since deploying the CloudFormation StackSets. You can follow the optional steps at the beginning of this post to create an CloudWatch Alarm. Follow the next steps to view CloudWatch Alarms which have been captured.

1. Go to the CloudWatch Log groups on the CloudWatch console.
2. Choose the log group name, which you entered into the CloudWatchLogGroupParam CloudFormation parameter.
3. Choose one of the Log Streams available to see the notification.
4. Select the arrow to the left of Timestamp to extend the log event and view the CloudWatch alarm information.

Figure 4: CloudWatch Log group events page

Cost

The CloudFormation template will provision EventBridge buses, rules, and a CloudWatch Log Group target if selected. The pricing is $1.00/million cross-account EventBus events, $0.50 per GB of
CloudWatch Log data ingested, and $0.03 per GB of log data stored. Your final cost will vary depending on the amount of CloudWatch alarms processed through the architecture. CloudFormation StackSets and Organizations do not cost extra.

Cleanup

1. Delete the CloudFormation StackSets created earlier.
2. Delete the optional CloudWatch Alarm created earlier.

Next Steps

This post only scratches the surface of what’s possible with EventBridge. For example, You can create EvenBridge rules to send only specific alarms that are pertinent to your environment, or specific alternative EventBridge targets. You can even perform more advanced automation, such as monitoring your CloudWatch Alarms within your Slack Channels, by utilizing Amazon SNS as an EventBridge target along with AWS Chatbot, and much more.