AWS Cloud Operations & Migrations Blog

Send Organizational AWS Health Events to Amazon Chime or Slack

There’s now a much easier solution! Please see our blog post on AWS Health Aware – Customize AWS Health Alerts for Organizational and Personal AWS Accounts. 

Receiving notifications for AWS Health events can be done in multiple ways depending on your desired platform, from email notifications with Amazon SNS to account-specific chat notifications with AWS Chatbot. Recently, the team behind AWS Health API (available to AWS Business/Enterprise Support customers) released AWS Health Organizational View, which allows you to aggregate all AWS Health events across your AWS Organizations.

Solution overview

AWS Health Organizational View Alerts (AHOVA) is an automated notification tool for sending well-formatted Alerts to your Amazon Chime room or Slack channel.

The following terms are used in this post:

  • Webhook URL is a public URL that can be used to send HTTP Post requests. Both Amazon Chime and Slack can create a unique webhook URL to use specifically for your chat room or channel.
  • Health Event ARN is the Amazon Resource Name of an AWS Health Event. It is presented in the following format: arn:aws:health:region::event/SERVICE_ISSUE_ID.
  • LastUpdatedTime is the last time a Health event ARN was updated by AWS. If there is an ongoing AWS Health issue, the issue updates its status—such as open, investigating, or closed—throughout the event.

The described solution uses an Amazon CloudWatch Scheduled Rule which triggers an AWS Lambda function to check the AWS Health Organizational View API every 60 seconds. The Lambda function verifies if the AWS Health Event is new or has been updated by comparing the event’s last updated time with entries in an Amazon DynamoDB table. If there is a new event or an existing event has been updated, the Lambda function posts the event to an Amazon Chime room or a Slack channel using the webhook URL. The webhook URL is encrypted via a second Lambda function using AWS Key Management Service, and is only decrypted when posting to Amazon Chime or Slack.

The following diagram illustrates the workflow used to query the AWS Health API and posts to Amazon Chime or Slack.

ahova architecture diagram

  1. A webhook URL is entered into an AWS CloudFormation template that is then encrypted using AWS KMS.
  2. A CloudWatch Scheduled Rule triggers a Lambda function every 60 seconds to query the AWS Health Organizational View API.
  3. The AWS Health Organizational View API goes out to every account in the organization and returns a JSON response.
  4. The Lambda function looks for an existing entry in the DynamoDB table by comparing the JSON response Health Event ARNs and the LastUpdatedTime.
  5. The Lambda function formats a message, decrypts the webhook URL, and posts a well-formatted message to Amazon Chime or Slack.

Prerequisites

The following prerequisites are necessary for this solution:

  1. AWS Organizations with all features
  2. AWS Enterprise or Business Support on all accounts in your AWS Organizations.
  3. The AWS CLI is installed on your machine.
  4. AWS CLI access, the ability to launch AWS CloudFormation Stacks with resources, and the ability to create IAM roles in the AWS Organizations master account.

Enabling AWS Health Organizational View

The following steps are all performed within the AWS Organizations master account.

  1. Upgrade the AWS CLI and boto3 (this example uses Python).
  • Verify that the AWS CLI and boto3 are not outdated, as shown in the following screenshot, by running the following command:

pip list -o

Console window showing output of pip list -o which shows python installed package version and the latest version

  • If neither awscli or boto3 appear in the list, go to Step 2.
  • If awscli and/or boto3 appear in the list you can upgrade them both by running the following command, as shown in the following screenshot:

pip install –-upgrade awscli boto3

Console window showing output of pip install --upgrade awscli and boto3

  1. Enable AWS Health Organizational View via the AWS CLI from the AWS Organizations master account:
  • From the AWS CLI command, run:

aws health enable-health-service-access-for-organization

  • Enabling this feature is an asynchronous process and takes time to complete. There is no charge for enabling this. To see the status of the process, as shown in the following screenshot, run:

aws health describe-health-service-status-for-organization

Console window showing output of aws health describe-health-service-status-for-organization

Once you see ENABLED, the service is active. Any new AWS Health events from this point forward can now be queried from the AWS Health Organizational View API. Any AWS Health events occurring before the service was enabled are not returned.

Deploying AWS Health Organizational View Alerts for Amazon Chime

Files for this deployment are found in the chime-version folder.

  1. Create a new Amazon Chime chat room (if you want to use an existing one, go to Step 2).
  2. Create an Amazon Chime webhook in the chat room (you need a webhook URL for Step 5).
  3. Download the following two files in the chime-version folder: CFT_chime-version.yml, and healthapi-chime-v0.0.0.zip.
  4. Log in to the AWS Management Console as an Administrator and navigate to Amazon S3.
  • Create a new Amazon S3 bucket or use an existing bucket. Upload the healthapi-chime-v0.0.0.zip file to the bucket.
  1. Navigate to AWS CloudFormation and choose Create Stack (if you’ve already created a Stack before, select With new resources (standard)).
  • Under Specify template, select Upload a template file and browse for CFT_chime-version.yml. Choose Next.
  • Enter a Stack name (for example, AHOVAChime)
  • For the Lambda bucket, enter only the name of the Amazon S3 bucket from Step 4 (for example, my-bucket-name).
  • For the Lambda Key, enter only the name/location of healthapi-chime-v0.0.0.zip (if it’s in the root bucket, enter healthapi-chime-v0.0.0.zip; if it’s in a folder, enter <folder name>/healthapi-chime-v0.0.0.zip).
  • For the Search Back, you can leave it at the default of 24, which will return events and updates to events going back 24 hours. Feel free to increase or decrease as necessary.
  • For the Regions, you can leave it blank for alerts on all Regions or you can enter in comma-separated Regions if you only want to alerts for those Regions (for example, us-east-1, us-west-1).
  • For the ChimeURL, put in the webhook URL you got from Step 2 without the https:// in front and choose Next.
  1. Choose Next and review your parameters. If everything looks good, select the check box to acknowledge that this AWS CloudFormation template creates IAM roles and choose Create stack. In roughly 5–10 minutes the solution is deployed. You’ll only see AWS Health Events that occurred after you enabled AWS Health Organizational View.

Deploying AWS Health organizational view alerts for Slack

Files for this deployment are in the slack-version folder.

  1. Create a new Slack channel (if you want to use an existing one, skip to Step 2).
  2. In your browser, go to workspace-name.slack.com/apps where workspace-name is the name of your Slack Workspace.
  • In the search bar, search for Incoming Webhooks and choose
  • Choose Add to Slack.
  • Select the channel you created in Step 1 and choose Add Incoming Webhooks integration.
  • From this page, you can change the name of the webhook (for example, AWS Bot), the icon/emoji to use, etc.
  • For this deployment, you need the Webhook URL for Step 5.
  1. Download the following three files from the slack-version folder: CFT_slack-version.yml, and healthapi-slack-v0.0.0.zip.
  2. Log in to the AWS Management Console as an Administrator and navigate to Amazon S3.
  • Create a new S3 bucket or use an existing one and upload healthapi-slack-v0.0.0.zip to the bucket.
  1. Navigate to AWS CloudFormation and choose Create Stack (if you’ve already created a Stack before, select With new resources (standard)).
  • Under Specify template, select Upload a template file and browse for CFT_slack-version.yml. Choose Next.
  • Enter a Stack name (for example, AHOVASlack)
  • For the Lambda bucket, enter only the name of the Amazon S3 bucket from Step 4 (for example, my-bucket-name).
  • For the Lambda Key, enter only the name/location of healthapi-slack-v0.0.0.zip (if it’s in the root bucket, enter healthapi-slack-v0.0.0.zip; if it’s in a folder, enter <folder name>/healthapi-slack-v0.0.0.zip).
  • For the Search Back, you can leave it at the default of 24, which will return events and updates to events going back 24 hours. Feel free to increase or decrease as necessary.
  • For the Regions, leave it blank for alerts on all Regions or you can enter in comma-separated Regions (for example, us-east-1, us-west-1).
  • For the SlackURL, put in the webhook URL you got from Step 2 without the https:// in front and choose Next.
  1. Choose Next and review your parameters. If everything looks good, select the check box to acknowledge this AWS CloudFormation template creates IAM roles and choose Create stack. In roughly 5–10 minutes the solution is deployed. You’ll only see AWS Health Events that occurred after you enabled AWS Health Organizational View.

Conclusion

In this post, you learned how AWS Organizational View can aggregate all AWS Health alerts in your AWS Organizations. You then deployed a serverless infrastructure via AWS CloudFormation that sends those alerts to Amazon Chime or Slack. You now should be able to proactively monitor and react to AWS Health events for your entire AWS Organizations. To get started head over to our aws-samples Github repository and download AWS Health Organizational View Alerts (AHOVA).

About the Author

Jordan Roth is a Senior Technical Account Manager and Enterprise Support Lead for AWS Enterprise Support. He is also a VMware Cloud on AWS Subject Matter Expert within AWS and holds three AWS Certifications, as well as 12 VMware Certifications. Jordan works with Enterprise customers to design, deploy, and manage their AWS and/or VMware Cloud on AWS architectures and provides them white-glove support through the entire lifecycle. In his spare time, he enjoys traveling the globe with his wife, cooking, completing escape rooms, and running around with his two dogs.