How to Use Amazon CloudWatch Events to Monitor Application Health

Amazon CloudWatch Events enables you to react selectively to events in the cloud as well as in your applications. Specifically, you can create CloudWatch Events rules that match event patterns, and take actions in response to those patterns. CloudWatch Events lets you process both AWS-provided events and custom events (those that you create and inject yourself). The AWS-provided events that CloudWatch Events supports include:

• Amazon EC2 instance state-change events.
• Auto Scaling lifecycle events, and instance launch and terminate notifications.
• Scheduled events.
• AWS API call and console sign-in events reported by AWS CloudTrail.

See the full list of supported events.

In this post, I will show how to inject your own events into CloudWatch Events, and define event patterns and their corresponding responses.

Scenario—application availability

In our scenario, an organization deploys an application to a large number of computers. The application deploys a health-checking component along with the application to each computer, which in turn sends application health reports to a central location. The organization wants to monitor all instances of the deployed application and its availability in near real time. As a result, the health-checking component reports one of three color-coded health statuses about the application:

• Green: The application is healthy, so no action is required.
• Yellow: The application has failed health checks. Process the report with AWS Lambda.
• Red: The application might have failed on at least one computer because it passed the threshold of failed health checks. Notify the operations team.

Using this color-coding, you can monitor an application’s availability and pursue follow-up actions, if required.

Set up the solution

This solution requires you to have a central location that will collect all the health-check data, analyze it, and then take action based on the color-coding. To do this, you will use CloudWatch Events and have the health-checking component send reports as events.

You will use the put-events AWS CLI command, which expects an event entry to be defined by at least two fields: the Source and the DetailType (the latter defines the type of the detail, such as the payload of the event). I use com.mycompany.myapp as my Source. I use myCustomHealthStatus as the DetailType. For the optional payload of an event entry, I use a JSON object with one key, HealthStatus, with possible values of Green, Yellow, and Red.

The following diagram depicts the HealthStatus application’s reporting architecture.

A CloudWatch Events rule is the conditional statement that maps an incoming event and routes the event to its targets. A target is a resource, such as a Lambda function or an Amazon SNS topic that is invoked when a rule is triggered. As shown in the preceding diagram, I set up the rules such that:

1. If HealthStatus is Yellow, invoke a Lambda function named SampleAppDebugger.
2. If HealthStatus is Red:
   1. Publish to an SNS topic named RedHealthNotifier, which notifies the operations team and indicates that the situation might require human intervention.
   2. Publish to an Amazon SQS queue named ReportInspectionQueue for the operations team to inspect the individual health-check status by using custom scripts.

Now, let’s deploy the solution. Be sure to deploy it in the US West (Oregon) Region, or choose a region where all the services used in this post are supported.

Create three targets

To create the rules noted above, you first need to create three targets. The first target is invoked when HealthStatus is Yellow. The second and the third targets are invoked when HealthStatus is Red. You can learn more using the getting started links provided in this section.

To create the three targets:

1. Create a Lambda function called SampleAppDebugger with the run-time version Python 2.7 by using the following code. Creation of the Lambda function will in turn create CloudWatch Logs groups for its logging.

#This Lambda function can process the Yellow HealthStatus report sent by an AppInstance.
#For the purposes of this blog post, the function only prints the received report.

from __future__ import print_function
def lambda_handler(event, context):
      print("nProcessing the following Yellow reportnn")
      print(event)
      print("nnProcessing completed!n")

Create two rules

You will now create two rules. The first rule matches events with a HealthStatus of Yellow. The second rule matches events with a HealthStatus of Red.

To create the first rule with the AWS CLI:

1. Save the following event pattern as a file named YellowPattern.json.

{
 "detail-type": [
   "myCustomHealthStatus"
 ],
 "detail": {
   "HealthStatus": [
     "Yellow"
   ]
 }
}

$ aws events put-rule 
--name myCustomHealthStatusYellow 
--description myCustomHealthStatusYellow 
--event-pattern file://YellowPattern.json

3. Make the Lambda function, SampleAppDebugger, that you created earlier the target of the rule by running the following command.

$ aws events put-targets 
--rule myCustomHealthStatusYellow 
--target Id=1,Arn=arn:aws:lambda:<Region>:<AccountNumber>:function:SampleAppDebugger

4. Use the following command to add permission to the Lambda function so that CloudWatch Events can invoke the function when the rule is triggered.

$ aws lambda add-permission 
--function-name SampleAppDebugger 
--statement-id AllowCloudWatchEventsToInvoke 
--action 'lambda:InvokeFunction' 
--principal events.amazonaws.com 
--source-arn arn:aws:events:<Region>:<AccountNumber>:rule/myCustomHealthStatusYellow

If you prefer to use the AWS Management Console, you can create the first rule with the following steps:

1. Go to the Amazon CloudWatch console.
2. Click Events in the left pane
3. Click Create Rule, and then click Show advanced options.
4. Click Edit the JSON version of the pattern.
5. Paste the contents of the YellowPattern.json file in the text box.
6. Click Add target under the Targets section. Select the Lambda function SampleAppDebugger from the list.
7. Click Configure details.
8. Name the rule as myCustomHealthStatusYellow.

Now, create the second rule, myCustomHealthStatusRed. Follow the same preceding steps, but replace Yellow with Red in the JSON file name, and save the file as RedPattern.json. Add the SNS topic, RedHealthNotifier, and the SQS queue, ReportInspectionQueue, which you created earlier as two targets for your rule. Make sure to use a different ID for the second target if you are using the AWS CLI, or click Add Target one more time in the AWS Management Console. If you use the AWS CLI, add resource-based policies for CloudWatch Events to invoke each of the targets.

Test the setup

To test the setup, you must add a test case for the custom events in which you inject your event and monitor the rest of the system for the desired results. You again will use the CloudWatch Events put-events CLI command to inject your custom events. To test the setup, you will simulate the component’s input via CLI.

Start with a test event by saving the following to a file named TestYellowEvent.json.

[
 {
   "Source": "com.mycompany.myapp",
   "DetailType": "myCustomHealthStatus",
   "Detail": "{"HealthStatus": "Yellow", "OtherField": "OtherValue"}"
 }                            
]

Run the following command to inject a custom event with Yellow status.

$ aws events put-events --entries file://TestYellowEvent.json

Run the following command to inject a custom event with Green status.

$ aws events put-events --entries file://TestGreenEvent.json

Run the following command to inject a custom event with Red status.

$ aws events put-events --entries file://TestRedEvent.json

You can verify the results by using the CloudWatch console:

1. Click Metrics in the left pane.
2. In the CloudWatch Metrics by Category pane, under Event Metrics, select By Rule Name, and select the MatchedEvents metric for the respective rule.

When you inject the Green event, nothing happens because there is no rule in CloudWatch Events that matches this event. However, when the Yellow event is injected, CloudWatch Events matches it with a rule and invokes the Lambda function. The function in turn generates an entry in the corresponding CloudWatch Logs group, and the CloudWatch metrics can be verified from the Lambda console. Similarly, the Red event results in a message sent to the SQS queue that can be received from the Amazon SQS console and published to the SNS topic, which in turn sends an email to the email address you subscribed during the target-creation step.

If you were to inject an event with a HealthStatus of, say, Orange or with a detail-type of, say, myProcessStatus, the event would still be injected. However, the event would not match any rules and therefore would be treated similarly to the Green event.

Summary

In this post, I showed you how to create two CloudWatch Events rules that monitor for color-coded health status reports from multiple instances of an application. I simulated the input from the health-checking component by using the put-events CLI command. Your applications can also can use the AWS SDK to inject the events.

If you have comments about this blog post, please submit them below in the “Comments” section. If you have troubleshooting questions related to the solution in this post, please start a new thread on the CloudWatch forum.

– Saurabh

Want more AWS Security how-to content, news, and feature announcements? Follow us on Twitter.

More from this author can be found here.