Centralized incident management with AWS Control Tower and PagerDuty

Managing and operating large-scale applications and workload in the cloud requires central governance, control, and visibility. Today, AWS Control Tower customers are configuring a secure baseline and using guardrails to provide ongoing governance for their multi-account environment. Customers also want to extend AWS Control Tower guardrail notification to their existing incident management and compliance alerting process. Having a centralized incident management process is critical to keeping systems running smoothly.

In this blog post, I show you how you can use PagerDuty, an AWS Advanced Technology Partner, to integrate the PagerDuty solution in AWS Marketplace with AWS Control Tower. PagerDuty helps organizations manage incidents across their AWS multi-account environment, helps teams identify and mitigate business-impacting compliance issues, and helps reduce risk and time to resolution.

AWS Control Tower uses Amazon Simple Notifiation Service (Amazon SNS) to send compliance notification to the audit account’s email address. These alerts help you inspect the compliance events timeline and details about the non-compliant resources. During the AWS Control Tower initial launch, you acknowledge a manual SNS subscription for the aws-controltower-AggregateSecurityNotifications topic, one per supported Region.

As shown in the following architecture diagram, AWS Config monitors member accounts and aggregates compliance status to the audit account. You deploy an AWS CloudFormation StackSet in the AWS Control Tower management account. This StackSet configures PagerDuty’s Amazon SNS subscription to the aws-controltower-AggregateSecurityNotifications topic as an HTTPS endpoint in the audit account. Then, you configure PagerDuty rulesets to trigger AWS Control Tower guardrail notification as incident. Refer to the following diagram.

Solution overview

To integrate AWS Control Tower with PagerDuty, follow these steps.

1. Subscribe to the PagerDuty solution in AWS Marketplace
2. Get the PagerDuty integration key
3. Configure the Amazon SNS subscription
4. Configure PagerDuty service and rules
5. Verify the integration

Prerequisites

Before getting started, you need the following:

• A working version of the AWS Control Tower environment. To configure AWS Control Tower, see Getting Started with AWS Control Tower.
• To try out the solution without subscription, you can register for a PagerDuty developer account.

Solution walkthrough

Step 1: Subscribe to the PagerDuty solution in AWS Marketplace

See PagerDuty’s implementation guide on how to subscribe to this solution in AWS Marketplace.

Step 2: Get the PagerDuty integration key

You need a PagerDuty integration key to receive AWS Control Tower guardrail notification over the PagerDuty HTTPS endpoint.

1. Log in to the PagerDuty console.
2. In the top menu, choose Services and then Event Rules.
3. Select Default Global Ruleset.
4. Expand Incoming Event Source. You should see your Integration Key. Take note of this information.

Step 3: Configure the Amazon SNS subscription

A: Deploy a CloudFormation template to automate the PagerDuty SNS subscription.

1. Download this AWS CloudFormation Template.
2. Log in to your AWS Control Tower management account with administrator access. Open the CloudFormation console to launch a StackSet. Choose Upload a template file and select the CloudFormation template that you downloaded. Choose Next.
3. For StackSet name, enter a name. For PagerDuty HTTPS endpoint, replace [YOUR_PAGERDUTY_INTEGRATION_KEY_HERE] with your Integration Key. Choose Next.
4. Choose Self-service permissions. For IAM role name, choose AWSControlTowerStackSetRole. Replace the default value of IAM execution role name with AWSControlTowerExecution. Choose Next.
5. For Account Numbers, enter the AWS account ID for your AWS Control Tower audit account.
6. For Specify regions, Select the AWS Control Tower supported Regions by referring to How AWS Regions Work With AWS Control Tower. Choose Next. Select the I acknowledge that AWS CloudFormation might create IAM resources checkbox. Choose Submit. You can verify the StackSet deployment completion on the Stack Instances tab in the CloudFormation console. Wait for the status to change from OUTDATED to CURRENT for all stack instances.

B: Verify Amazon SNS subscription

1. After the StackSet completes, log in to your AWS Control Tower audit account and open the Amazon SNS console.
2. In the left sidebar, choose Topics and select aws-controltower-AggregateSecurityNotifications.
3. In the Subscriptions tab, verify a new subscription for aws-controltower-AggregateSecurityNotifications topic that points to your PagerDuty HTTPS endpoint in this format, https://events.pagerduty.com/x-ere/[YOUR_PAGERDUTY_INTEGRATION_KEY]. Select this subscription and verify that Raw message delivery is enabled.

Step 4: Configure PagerDuty service and rules

Create a PagerDuty service to group the AWS Control Tower guardrail notification. Then create two PagerDuty rules, one to trigger when a non-compliant resource is detected and another to resolve when the resource is back to compliant status.

A: Create a PagerDuty service to group the AWS Control Tower guardrail notification

To create a new service, follow these steps.

1. Log in to your PagerDuty console.
2. In the top menu, choose Services and then Service Directory.
3. On the top right, select New Service. For Name, enter AWS Control Tower and provide a description for this service. Select Don’t use an integration. Keep the default values for other setting. Choose Add Service.

B: Create a rule to trigger an incident when a noncompliant resource is detected

To create a rule to trigger an incident, follow these steps.

1. Log in to your PagerDuty console. In the top menu, choose Services and then Event Rules.
2. In the Rulesets table, select Default Global Ruleset. Choose New Event Rule.
3. For When events match these conditions:
   • Enter source contains aws.config. Choose Add Condition.
   • Enter detail.newEvaluationResult.complianceType equals NON_COMPLIANT. Choose Add Condition.
4. For Customize Event Fields, define two custom variables by entering the following information in Define Custom Variable. Choose Add Variable for each variable.
   

   Name	Value	Regex
   resourceId	(.*)	detail.newEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.resourceId
   ruleName	(.*)	detail.newEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.configRuleName

5. For Customize Event Fields, choose Add Event Field and define two events by entering the following information in Replace Event Field. Keep the drop-down as Template.
   

   Event Field (CEF)	Value
   dedup_key	{{resourceId}}-{{ruleName}}
   summary	Rule {{ruleName}} on {{resourceId}} is now NON_COMPLIANT

6. For Do these things, select Create an incident on a Service.
   • Choose Basic tab, select AWS Control Tower from the Route to a Service drop-down menu. For Incident Creation, select Immediately.
   • Choose Advanced tab, select Set a custom trigger/resolve action and choose Always trigger an alert.
7. Choose Save Rule.

C: Create a rule to resolve when the resource returns to compliance

To create a rule to resolve incident, follow these steps.

1. Log in to your PagerDuty console. In the top menu, choose Services and then Event Rules.
2. In the Rulesets table, select Default Global Ruleset. Choose New Event Rule.
3. For When events match these conditions:
   • Enter source contains aws.config. Choose Add Condition.
   • Enter detail.newEvaluationResult.complianceType equals COMPLIANT. Choose Add Condition.
4. For Customize Event Fields, define two custom variables by entering the following information in Define Custom Variable. Choose Add Variable for each variable.
   

   Name	Value	Regex
   resourceId	(.*)	detail.newEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.resourceId
   ruleName	(.*)	detail.newEvaluationResult.evaluationResultIdentifier.evaluationResultQualifier.configRuleName

5. For Customize Event Fields, choose Add Event Field and define two events by entering the following information in Replace Event Field. Keep the drop-down as Template.
   

   Event Field (CEF)	Value
   dedup_key	{{resourceId}}-{{ruleName}}
   summary	Rule {{ruleName}} on {{resourceId}} is now COMPLIANT

6. For Do these things, select Suppress Alert.
   • Choose Basic tab, and then from the Route to a Service drop-down menu, select AWS Control Tower.
   • Choose Advanced tab, select Set a custom trigger/resolve action and Always resolve an alert.
7. Choose Save Rule.

You now have PagerDuty ready to trigger incident for AWS Control Tower guardrail notification.

Step 5: Verify the integration

Follow these steps to verify the integration.

1. This blog post, How to Detect and Mitigate Guardrail Violation with AWS Control Tower, shows how to simulate a guardrail violation.
2. To simulate a guardrail violation, log in to your AWS Control Tower management account. Then in that blog post, follow Step 3: Enable a guardrail at the child OU and Step 4: Detect guardrail violation.
3. Log in to the PagerDuty console. In the top menu, choose Incidents and then Alerts. You should see that the alert has been triggered.
4. Choose SHOW DETAILS.
5. Select View Message for more details about the alert.

By successfully triggering a guardrail violation, you have verified the integration between AWS Control Tower and PagerDuty.

Cleaning up

To avoid incurring future charges, you can remove example resources by first deleting the AWS CloudFormation stack instances and then deleting the StackSet.

Conclusion

In this blog post, I showed how to centralize incident management across the AWS Control Tower multi-account environment using PagerDuty. I provided a sample AWS CloudFormation template to automate PagerDuty’s subscription to the Amazon SNS topic for AWS Control Tower guardrail notification. I also showed how to configure PagerDuty to trigger incident for AWS Control Tower guardrail notification. This solution helps you to drive faster problem resolution and a better customer experience. For more information, see Solutions for AWS Control Tower in AWS Marketplace. To get the PagerDuty solution, visit the product page in AWS Marketplace.

About the authors

Cher Simon is a Senior Solutions Architect at AWS. Cher enjoys working with AWS customers in solving architectural and operational challenges by leveraging cloud native services and best practices at scale.

Joe Pusateri is a Solutions Architect at PagerDuty. He loves to solve application integration and architecture challenges for his customers to smooth their processes and reduce their risk.