AWS Cloud Operations & Migrations Blog

Customize Amazon CloudWatch alarm notifications to your local time zone – Part 2

In part 1 of this series, I cover how to customize Amazon CloudWatch alarm notifications using CloudWatch Events rule. For more information, see the Customize Amazon CloudWatch alarm notifications to your local time zone – Part 1 blog post.

In this part, I walk through the process of using Amazon Simple Notification Service (Amazon SNS) to customize Amazon CloudWatch alarm notifications.

Customize CloudWatch alarms by using a Lambda function subscribed to an SNS topic

You can use AWS Lambda subscribed to an SNS topic, where the Lambda function parses the alarm event and creates a customized notification.

Pros:

  • You can create one Lambda function in any AWS Region and subscribe it to the SNS topic that the alarm will trigger. Note that you will incur additional charges for using two SNS topics, check Amazon SNS pricing.

When the alarm state changes, the alarm triggers an SNS topic to which the Lambda function is subscribed. SNS invokes the Lambda function that creates the customized message. The message is sent to the SNS topic.

Walkthrough

To customize your alarm using an SNS subscription:

  • Create SNS topic for notification and use your email address to subscribe to it.
  • Install the PyTZ library and package with custom functions as AWS Lambda layer. With layers, you can use libraries in your function without needing to include them in your deployment package.
  • Create AWS Lambda execution role. This is the AWS Identity and Access Management (IAM) role that AWS Lambda assumes when it runs your function.
  • Create a Lambda function with code using custom functions in the layer, add environment variables with your time zone, time zone abbreviation, and SNS topic ARN.
  • Create SNS topic for CloudWatch alarm notification action and subscribe the Lambda function to it.
  • Edit CloudWatch alarm actions, and add the SNS topic for CloudWatch alarm notification action.

If you prefer to use a CloudFormation template to create these resources, launch the following stack, from stack Outputs tab add ‘AlarmActionSNSTopicARN‘ to the CloudWatch alarm you want to customize its notification action.

Launch Stack button

Prerequisites

If you want to follow along make sure you have access to the AWS Management Console with the proper IAM permissions required to create AWS Lambda layer, AWS Lambda function, AWS Lambda execution role, Amazon SNS topic and modify CloudWatch Alarm.

Create an SNS topic for notification

Create an SNS topic and use your email address to subscribe to it. Later you’ll add the SNS topic ARN to the Lambda function environment variable.

To create SNS topic

  1. Sign in to the Amazon SNS console, and from the left navigation pane, choose Topics.
  2. On the Topics page, choose Create topic.
  3. By default, the console creates a FIFO topic. Choose Standard.
  4. In Details, enter a name for the topic (for example, NotificationSNSTopic), and then choose Create topic.

To create a subscription to the topic

  1. In the left navigation pane, choose Subscriptions.
  2. On the Subscriptions page, choose Create subscription.
  3. On the Create subscription page, choose the Topic ARN field to see a list of the topics in your AWS account.
  4. Choose the topic that you created in the previous step.
  5. For Protocol, choose Email.
  6. For Endpoint, enter an email address that can receive notifications.
  7. Choose Create subscription.
  8. Check your email inbox and choose Confirm subscription in the email from AWS Notifications. The Sender ID is usually no-reply@sns.amazonaws.com.
  9. Amazon SNS opens your web browser and displays a subscription confirmation with your subscription ID.

Create a Lambda layer package

Install the PyTZ library dependencies for your Lambda function.

  1. Install the libraries in a package directory with the pip’s –target
mkdir python
pip3  install -t /python pytz
  1. Under the python directory, create changeAlarmToLocalTimeZone.py file that contains three functions:
  • getAllAvailableTimezones to print all available time zones.
  • searchAvailableTimezones to print time zones that match the sub string.

For example, TimeZone.SearchAvailableTimezones(‘sy’)
Returns:
Matched zone: Antarctica/Syowa
Matched zone: Australia/Sydney

  • changeAlarmToLocalTimeZone function to change the local time zone for the alarm.
import json
import boto3
import datetime
import pytz
import re
import urllib
import pytz
import re

def searchAvailableTimezones(zone):
    for s in pytz.all_timezones:
        if re.search(zone, s, re.IGNORECASE):
            print('Matched Zone: {}'.format(s))

def getAllAvailableTimezones():
    for tz in pytz.all_timezones:
        print (tz)

def changeAlarmToLocalTimeZone(event,timezoneCode,localTimezoneInitial,platform_endpoint):
    tz = pytz.timezone(timezoneCode)
    #exclude the Alarm event from the SNS records
    AlarmEvent = json.loads(event['Records'][0]['Sns']['Message'])

    #extract event data like alarm name, region, state, timestamp
    alarmName=AlarmEvent['AlarmName']
    descriptionexist=0
    if "AlarmDescription" in AlarmEvent:
        description= AlarmEvent['AlarmDescription']
        descriptionexist=1
    reason=AlarmEvent['NewStateReason']
    region=AlarmEvent['Region']
    state=AlarmEvent['NewStateValue']
    previousState=AlarmEvent['OldStateValue']
    timestamp=AlarmEvent['StateChangeTime']
    Subject= event['Records'][0]['Sns']['Subject']
    alarmARN=AlarmEvent['AlarmArn']
    RegionID=alarmARN.split(":")[3]
    AccountID=AlarmEvent['AWSAccountId']

    #get the datapoints substring
    pattern = re.compile('\[(.*?)\]')
    
    #test if pattern match and there is datapoints
    if pattern.search(reason):
        Tempstr = pattern.findall(reason)[0]

        #get in the message all datapoints timestamps and convert to localTimezone using same format
        pattern = re.compile('\(.*?\)')
        m = pattern.finditer(Tempstr)
        for match in m:
            Tempstr=match.group()
            tempStamp = datetime.datetime.strptime(Tempstr, "(%d/%m/%y %H:%M:%S)")
            tempStamp = tempStamp.astimezone(tz)
            tempStamp = tempStamp.strftime('%d/%m/%y %H:%M:%S')
            reason=reason.replace(Tempstr, '('+tempStamp+')')
    

    #convert timestamp to localTimezone time
    timestamp = timestamp.split(".")[0]
    timestamp = datetime.datetime.strptime(timestamp, "%Y-%m-%dT%H:%M:%S")
    localTimeStamp = timestamp.astimezone(tz)
    localTimeStamp = localTimeStamp.strftime("%A %B, %Y %H:%M:%S")

    #create Custom message and change timestamps

    customMessage='You are receiving this email because your Amazon CloudWatch Alarm "'+alarmName+'" in the '+region+' region has entered the '+state+' state, because "'+reason+'" at "'+localTimeStamp+' '+localTimezoneInitial +'.'
    
    # Add Console link
    customMessage=customMessage+'\n\n View this alarm in the AWS Management Console: \n'+ 'https://'+RegionID+'.console.aws.amazon.com/cloudwatch/home?region='+RegionID+'#s=Alarms&alarm='+urllib.parse.quote(alarmName)
    
    #Add Alarm Name
    customMessage=customMessage+'\n\n Alarm Details:\n- Name:\t\t\t\t\t\t'+alarmName
    
    # Add alarm description if exist
    if (descriptionexist == 1) : customMessage=customMessage+'\n- Description:\t\t\t\t\t'+description
    customMessage=customMessage+'\n- State Change:\t\t\t\t'+previousState+' -> '+state

    # Add alarm reason for changes
    customMessage=customMessage+'\n- Reason for State Change:\t\t'+reason
 
    # Add alarm evaluation timeStamp   
    customMessage=customMessage+'\n- Timestamp:\t\t\t\t\t'+localTimeStamp+' '+localTimezoneInitial

    # Add AccountID    
    customMessage=customMessage+'\n- AWS Account: \t\t\t\t'+AccountID
    
    # Add Alarm ARN
    customMessage=customMessage+'\n- Alarm Arn:\t\t\t\t\t'+alarmARN

    #push message to SNS topic
    response = platform_endpoint.publish(
        Message=customMessage,
        Subject=Subject,
        MessageStructure='string'
    )

Create a deployment package from the installed libraries and .lib file under the python directory.

zip -r SNSSubscribtion-pytzLayer.zip ./python/*

To create a Lambda layer

  1. In the AWS Lambda console, open the Layers page, and choose Create layer.
  2. Enter a name and optional description for your layer.
  3. To upload your layer code, do one of the following, and then choose Create.
    • To upload a .zip file from your computer, choose Upload a .zip file, choose your .zip file, and then choose Open.
    • To upload a file from Amazon Simple Storage Service (Amazon S3), choose Upload a file from Amazon S3. For Amazon S3 link URL, enter a link to the file.
    • (Optional) For Compatible runtimes, choose up to five runtimes.
    • (Optional) For License, enter any required license information.

Create AWS Lambda execution role

By default, when you create a function in the Lambda console, AWS Lambda creates an execution role. You can also create an execution role in the IAM console.

To create an execution role in the IAM console

  1. Open the Roles page and choose Create role.
  2. Under Common use cases, choose Lambda.
  3. Choose Next: Permissions.
  4. Choose Next: Tags.
  5. Choose Next: Review.
  6. Enter a name and description for the role and then choose Create role.
  7. Open the role and add an inline policy.
  8. On the JSON tab, add the following permission. Enter your values for Region ID, Account ID, Lambda function Name, and SNS Topic ARN. 
    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:[Region ID]:[Account ID]:log-group:/aws/lambda/[Lambda function Name]"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": [
                "arn:aws:logs:[Region ID]:[Account ID]:log-group:/aws/lambda/[Lambda function Name]:*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "sns:Publish",
            "Resource": "[SNS Topic Arn]"
        }
    ]
}
  1. Review the policy.
  2. Enter a name for the policy and then choose Create policy.

Create a Lambda function

  1. Open the AWS Lambda console and choose Create a function.
  2. Select Author from scratch.
  3. Under Basic information, enter a name for the function. For Runtime, confirm that Python 3.8 is selected.
  4. Under Change default execution role, select Use an existing role, and then choose the Lambda execution role.
  5. Choose Create function.
  6. On the Configuration tab, in Designer, choose Layers.
  7. Choose Add a layer, select Custom layers, choose the created layer, and then choose Add.
  8. On the Configuration tab, in Designer, choose the Lambda function name.
  9. In Environment variables, choose Edit and then add three variables:
    • In Key, enter NotificationSNSTopic. In Value, enter the SNS topic ARN.
    • In Key, enter TimeZoneCode. In Value, enter your time zone code (for example, Australia/Sydney).
    • In Key, enter TimezoneInitial. In Value , enter the abbreviation for your time zone (for example, AEST). Time zones are often named by how many hours they are different from UTC time, so for example, you can also enter UTC+11.
  1. Select Save.
  2. Overwrite your function code in the embedded editor with the following code.
import boto3
import os
from changeAlarmToLocalTimeZone import *

#Get SNS Topic ARN from Environment variables
NotificationSNSTopic = os.environ['NotificationSNSTopic']

#Get timezone corresponding to your localTimezone from Environment variables 
timezoneCode = os.environ['TimeZoneCode']

#Get Your local timezone Initials, E.g UTC+2, IST, AEST...etc from Environment variables 
localTimezoneInitial=os.environ['TimezoneInitial']

#Get SNS resource using boto3
SNS = boto3.resource('sns')

#Specify the SNS topic to push message to by ARN
platform_endpoint = SNS.PlatformEndpoint(NotificationSNSTopic)

def lambda_handler(event, context):
    
    #Call Main function
    changeAlarmToLocalTimeZone(event,timezoneCode,localTimezoneInitial,platform_endpoint)
    
    #Print All Available timezones
    #getAllAvailableTimezones()
   
    #search if Timezone/Country exist
    #searchAvailableTimezones('sy')
  1. Choose Deploy.

The Lambda function will receive the following event JSON from the SNS topic:

{
    "Records": [
        {
            "EventSource": "aws:sns",
            "EventVersion": "1.0",
            "EventSubscriptionArn": "arn:aws:sns:[region Id]:[Account ID]:TriggerLambda:1d1c009f-c81b-40a5-b7ad-614287867179",
            "Sns": {
                "Type": "Notification",
                "MessageId": "f9f5ed56-3d38-57c8-b4ea-b51588f5f871",
                "TopicArn": "arn:aws:sns:[region Id]:[Account ID]:TriggerLambda",
                "Subject": "ALARM: \"Test LocalTime\" in US East (N. Virginia)",
                "Message": "{\"AlarmName\":\"Test LocalTime\",\"AlarmDescription\":\"Alarm Notification in my local timezone\",\"AWSAccountId\":\"[Account ID]\",\"NewStateValue\":\"ALARM\",\"NewStateReason\":\"Threshold Crossed: 1 out of the last 1 datapoints [0.0 (04/12/20 03:56:00)] was greater than or equal to the threshold (0.0) (minimum 1 datapoint for OK -> ALARM transition).\",\"StateChangeTime\":\"2020-12-04T03:57:01.659+0000\",\"Region\":\"US East (N. Virginia)\",\"AlarmArn\":\"arn:aws:cloudwatch:[region Id]:[Account ID]:alarm:Test LocalTime\",\"OldStateValue\":\"OK\",\"Trigger\":{\"Period\":60,\"EvaluationPeriods\":1,\"ComparisonOperator\":\"GreaterThanOrEqualToThreshold\",\"Threshold\":0.0,\"TreatMissingData\":\"- TreatMissingData:                    missing\",\"EvaluateLowSampleCountPercentile\":\"\",\"Metrics\":[{\"Expression\":\"FILL(m1, 0)\",\"Id\":\"e1\",\"Label\":\"Expression1\",\"ReturnData\":true},{\"Id\":\"m1\",\"MetricStat\":{\"Metric\":{\"Dimensions\":[{\"value\":\"API\",\"name\":\"Type\"},{\"value\":\"DescribeAlarms\",\"name\":\"Resource\"},{\"value\":\"CloudWatch\",\"name\":\"Service\"},{\"value\":\"None\",\"name\":\"Class\"}],\"MetricName\":\"CallCount\",\"Namespace\":\"AWS/Usage\"},\"Period\":60,\"Stat\":\"Average\"},\"ReturnData\":false}]}}",
                "Timestamp": "2020-12-04T03:57:01.702Z",
                "SignatureVersion": "1",
                "Signature": "WcgVMPrlQsJY3yqbds968tqKPC6KKDWHSjIwEmzKVHZYg6foN9F5sm2Tp5IWPgaM9wMmYg8dpQjkxSm4q9V9iP1PbLp81RgJS2NghdeHNVnyxyzywXFMDztYZpgB2pjzfT101RVGpUwVPntOpBeBq2KAs/NrFX1nS2aTK/OX+gyOxwYZxRftzd+ttHA+PCh0kKlym7nnxaWuO9hgSrnupH2YttuvsdTSAOZ4MGhBON/sMmmlcxzfiFD+jJaqlHFmQ0DncjSe1NNwceOpwNsue6//sMYU1QzV6bO34I343KmQdXYw/KISDz7qH70Odm7nRLN3ExSOhtC/FS0/dXGl4Q==",
                "SigningCertUrl": "https://sns.[region Id].amazonaws.com/SimpleNotificationService-010a507c1833636cd94bdb98bd93083a.pem",
                "UnsubscribeUrl": "https://sns.[region Id].amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:[region Id]:[Account ID]:TriggerLambda:1d1c009f-c81b-40a5-b7ad-614287867179",
                "MessageAttributes": {}
            }
        }
    ]
}

Create SNS topic for alarm notification actions

Create an SNS topic for alarm notification actions. Subscribe your Lambda function to the SNS topic. You must add the SNS topic ARN in your alarm actions (for example, OK or In alarm or Insufficient data).

  1. Sign in to the Amazon SNS console, and in the left navigation pane, choose Topics.
  2. On the Topics page, choose Create topic.
  3. By default, the console creates a FIFO topic. Choose Standard.
  4. In the Details section, enter a name for the topic (for example, AlarmActionSNSTopic).
  5. Scroll to the end of the form and choose Create topic.

To create a subscription to the topic for your Lambda function

  1. In the left navigation pane, choose Subscriptions.
  2. On the Subscriptions page, choose Create subscription.
  3. On the Create subscription page, choose the Topic ARN field to see a list of the topics in your AWS account.
  4. Choose the topic that you created in the previous step.
  5. For Protocol, choose AWS Lambda.
  6. For Endpoint, enter the ARN for your Lambda function.
  7. Choose Create subscription.

Add CloudWatch alarm action

Edit your alarm action and add the SNS topic ARN for alarm notification actions.

To edit your alarm action

  1. Open the CloudWatch console and from the left navigation pane, choose Alarms.
  2. Choose the name of the alarm, choose Edit, and choose Next.
  3. Under Notification, choose In alarm. From Send a notification to, choose your SNS topic for alarm notification, and then choose Next.
  4. Under Preview and create, review the information and conditions, and then choose Update alarm.

Output result

When you use the Australia/Sydney time zone (AEST):

Original message:

Original CloudWatch alarm notification message using UTC time zone
Customized CloudWatch alarm notification:

Cutomized CloudWatch alarm notification message using local time zone (AEST)

Cleanup

To avoid ongoing charges to your account, delete the resources you created in this walkthrough.

If you are using AWS CloudFormation template, then delete the stack

Conclusion

In this blog post, I’ve shown you how to use Lambda function subscribed to an SNS topic to customize CloudWatch alarm notification to the local time zone of your resources, logs, and metrics. For more information, check Amazon CloudWatch Alarms documentation.

 

About the Author

Ahmed Magdy Wahdan

Magdy is a Cloud Support Engineer and CloudWatch SME for Amazon Web Services. He helps global customers design, deploy, and troubleshoot large-scale networks built on AWS. He specializes in CloudWatch, Elastic Load Balancing, Auto Scaling, and Amazon VPC. In his spare time, he loves to free-dive and make desserts.