How to get notified on specific Lambda function error patterns using CloudWatch

This post demonstrates how to automate alert notifications for specific AWS Lambda function errors using an Amazon CloudWatch log subscription. CloudWatch Logs let you invoke a Lambda function when a log entry matches a pattern. Amazon CloudWatch alarms are used to notify when an error occurs with a Lambda function; this notification does not give any specifics about the error. For scenarios where you need specificity on the error in the notification, you can use a CloudWatch Logs subscription. CloudWatch Logs subscriptions let you match entries with a particular error pattern in your log and be notified with those error details. This saves you an extra step to parse through the logs and take necessary action. You can also use this as a blueprint to build automated reactive measures when that error pattern is detected in your Lambda function.

This post walks you through how to configure a CloudWatch log subscription that triggers an AWS Lambda function to process the log. The Lambda function uses Amazon SNS to send an email with the specific error details and the log location.

Solution overview

Architecture Diagram showing how cloudwatch log subscription triggers error processing lambda function to send email notification with specific log error

The architecture for this solution is fairly straightforward. You have a bunch of Lambda functions whose specific critical errors you want to be notified about. CloudWatch Logs captures the logs from these Lambda functions. It invokes the “error processing” Lambda function when a log entry matches a filter pattern, for example, ERROR, CRITICAL, or a custom error. This error processing Lambda function in turn publishes a message to an Amazon SNS topic, to which can subscribed to get an email when the error occurs.

For the purposes of this post, we will use the following sample error-generating Lambda function:

import logging
import os

logging.basicConfig(level=logging.DEBUG)
logger=logging.getLogger(__name__)


def lambda_handler(event, context):
    logger.setLevel(logging.DEBUG)
    logger.debug("This is a sample DEBUG message.. !!")
    logger.error("This is a sample ERROR message.... !!")
    logger.info("This is a sample INFO message.. !!")
    logger.critical("This is a sample 5xx error message.. !!")

Deployment tutorial

Prerequisites

To deploy this solution, you need access to the AWS Management Console.
You need permission to create IAM roles and policies, SNS topics, Lambda functions, and CloudWatch event rules.

To implement this solution, you must create:

An SNS topic
An IAM role
A Lambda function
A CloudWatch log trigger

Step 1: Creating an SNS topic

To create an SNS topic, complete the following steps:

Open the Amazon SNS console.
In the left navigation pane, choose Topics.
Select Create topic. For Topic name, enter a name and choose Create topic. You can now see the MySNSTopic page. The Details section displays the topic’s Name, ARN, Display name (optional), and the AWS account ID of the Topic owner.
In the Details section, copy the topic ARN to the clipboard, for example: arn:aws:sns:us-east-1:123456789012:MySNSTopic
In the left navigation pane, choose Subscriptions and Create subscription.
On the Create subscription page, do the following:
1. Enter the topic ARN of the topic you created earlier:arn:aws:sns:us-east-1:123456789012:MySNSTopic
2. For Protocol, select Email.
3. For Endpoint, enter an email address that can receive notifications.
4. Choose Create subscription.
Note that for email subscriptions, you must confirm the subscription by clicking the confirm subscription link you will receive via email. Once the subscription is confirmed, you are ready to receive email notifications.

Step 2: Creating an IAM role

To create an IAM role, complete the following steps. For more information, see Creating an IAM role.

In the IAM console, choose Policies from the left navigation pane, and select Create Policy.

Choose the JSON tab and enter the following IAM policy, replacing the Topic ARN with your SNS topic ARN created in previous step:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "sns:Publish",
            "Resource": "arn:<partition>:sns:<region>:<AWS account number>:<name of the SNS topic from previous step>"
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:<partition>:logs:<region>:<AWS account number>:log-group:/aws/lambda/<name of the lambda function you are going to create in next step>:*"
        }
    ]
}

Choose Review policy.
Enter a name (MyCloudWatchRole) for this policy and choose Create policy. Note the name of this policy for later steps.
In the left navigation pane, choose Roles and select Create role.
On the Select role type page, choose AWS service as your trusted entity and choose Lambda under common use cases.
Choose Next: Permissions.
Filter policies by the policy name that you just created, and select the check box.
Choose Next: Tags, and give it an appropriate tag.
Choose Next: Review. Give this IAM role an appropriate name, and note it for future use.
Choose Create role.

Step 3: Creating a Lambda function

To create a Lambda function, complete the following steps. For more information, see Create a Lambda Function with the console.

In the Lambda console, choose Author from scratch.
For Function Name, enter the name of your function.
For Runtime, choose Python 3.7.
For Execution role, select Use an existing role, then select the IAM role created in the previous step.

Choose Create Function, remove the default function, and copy the following code into the Function Code window:

# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
# Licensed under the Apache License, Version 2.0 (the "License").
# You may not use this file except in compliance with the License.
# A copy of the License is located at## http://aws.amazon.com/apache2.0/
# or in the "license" file accompanying this file.
# This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
# either express or implied. See the License for the specific language governing permissions
# and limitations under the License.
# Description: This Lambda function sends an email notification to a given AWS SNS topic when a particular
#              pattern is matched in the logs of a selected Lambda function. The email subject is
#              Execution error for Lambda-<insert Lambda function name>.
#              The JSON message body of the SNS notification contains the full event details.

# Author: Sudhanshu Malhotra

import base64
import boto3
import gzip
import json
import logging
import os

from botocore.exceptions import ClientError

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)


def logpayload(event):
    logger.setLevel(logging.DEBUG)
    logger.debug(event['awslogs']['data'])
    compressed_payload = base64.b64decode(event['awslogs']['data'])
    uncompressed_payload = gzip.decompress(compressed_payload)
    log_payload = json.loads(uncompressed_payload)
    return log_payload


def error_details(payload):
    error_msg = ""
    log_events = payload['logEvents']
    logger.debug(payload)
    loggroup = payload['logGroup']
    logstream = payload['logStream']
    lambda_func_name = loggroup.split('/')
    logger.debug(f'LogGroup: {loggroup}')
    logger.debug(f'Logstream: {logstream}')
    logger.debug(f'Function name: {lambda_func_name[3]}')
    logger.debug(log_events)
    for log_event in log_events:
        error_msg += log_event['message']
    logger.debug('Message: %s' % error_msg.split("\n"))
    return loggroup, logstream, error_msg, lambda_func_name


def publish_message(loggroup, logstream, error_msg, lambda_func_name):
    sns_arn = os.environ['snsARN']  # Getting the SNS Topic ARN passed in by the environment variables.
    snsclient = boto3.client('sns')
    try:
        message = ""
        message += "\nLambda error  summary" + "\n\n"
        message += "##########################################################\n"
        message += "# LogGroup Name:- " + str(loggroup) + "\n"
        message += "# LogStream:- " + str(logstream) + "\n"
        message += "# Log Message:- " + "\n"
        message += "# \t\t" + str(error_msg.split("\n")) + "\n"
        message += "##########################################################\n"

        # Sending the notification...
        snsclient.publish(
            TargetArn=sns_arn,
            Subject=f'Execution error for Lambda - {lambda_func_name[3]}',
            Message=message
        )
    except ClientError as e:
        logger.error("An error occured: %s" % e)


def lambda_handler(event, context):
    pload = logpayload(event)
    lgroup, lstream, errmessage, lambdaname = error_details(pload)
    publish_message(lgroup, lstream, errmessage, lambdaname)

In the Environment variables section, enter the following key-value pair:
- Key= snsARN
- Value= the ARN of the MySNSTopic created earlier
Choose Save.

Step 4. Creating a CloudWatch log trigger

To add a trigger, choose Add trigger and then choose CloudWatch Logs from the drop-down.
Under Log group, select the CloudWatch log group name for the Lambda function from which you want error notifications. In our case, this would be the log group for the sample error-generating Lambda function discussed above.
Enter an appropriate value for the Filter name, and under Filter pattern, enter the filter value for the log you want to be notified on. For example- “?ERROR ?WARN ?5xx” will filter for terms like ERROR, WARN, or 5xx in the log. Log filter and pattern syntax document contains more examples for other complex patterns.
Enable trigger, and Add.

Solution validation

To validate my solution, I’m going to run the sample error-generating Lambda function and filter on the “?ERROR ?WARN ?5xx” pattern to get an email notification as follows:

email notification screenshot showing specific error messages like [ERROR] and [CRITICAL]

Similarly, I can create a filter pattern for any specific error, for example, error 5xx as shown, and get the following notification on just that error message:

Lambda function trigger configuration screenshot with error filter pattern set to 5xx

email notification screenshot showing specific error messages- 5xx error message

Cleaning up

To avoid ongoing charges, delete the resources created in the previous steps including the CloudWatch Events rule, the Lambda function, and the SNS topic.

Conclusion

This post demonstrates how you can use a CloudWatch Log filter to parse a log for a Lambda function and get an email notification on that specific error. For further reading, see:

About the Author

Sudhanshu Malhotra is a solutions architect at AWS. Sudhanshu enjoys working with customers and helping them deliver complex solutions in AWS in the area of DevOps, infrastructure as code, and AWS Config management. In his spare time, Sudhanshu enjoys spending time with his family, hiking, and tinkering with cars.

Rajat Mathur is an Enterprise Solutions Architect at Amazon Web Services. Rajat is a passionate technologist who enjoys building innovative solutions for AWS customers. His core areas of focus are IoT, Networking and Serverless computing. In his spare time, Rajat enjoys long drives, traveling and cooking.

AWS Cloud Operations Blog