Send alerts on custom AWS DMS errors from Amazon CloudWatch Logs

AWS Database Migration Service (AWS DMS) is a widely used AWS service to migrate data into the AWS Cloud from on-premises databases or within AWS databases in the cloud.

For smooth migrations using AWS DMS, it’s imperative to monitor the AWS DMS resources and tasks, specifically for errors and warnings. Monitoring AWS DMS task via task status, Amazon CloudWatch, and DMS events can help you analyze errors and take corrective actions to fix them. In this post, we discuss how to trigger notifications based on keywords or specific error messages occurring in the AWS DMS task logs using Amazon CloudWatch log subscription and an AWS Lambda function to process the AWS DMS logs. The Lambda function uses Amazon Simple Notification Service (Amazon SNS) to send an email with the specific error details.

Solution overview

The architecture is an event-based solution comprised of AWS DMS tasks that send logs to CloudWatch. When logs are present in CloudWatch, it utilizes a filter pattern to match log events. When a log entry matches a defined filter pattern, for example, “table is suspended” or a custom error, it triggers a Lambda function to publish a message to an SNS topic. The following diagram illustrates the solution workflow.

Understanding the Code in AWS Lambda Function

The CloudWatch Log trigger sends a message event to the Lambda function whenever a defined filter pattern is matched in the DMS error logs. A filter pattern can be a specific AWS DMS task error or keyword you wish to monitor and receive an alert through email. The default filter pattern is suspended, if you want to monitor any other AWS DMS error, then change the value of FilterPattern in template.yaml

The filtered CloudWatch logs are sent as a JSON document which is parsed to identify the DMS task, Replication Instance and associated DMS task error message. These details help us form an email message body in order to notify the user(s) about a specific DMS task with an error.

We then use this information to form text that is published to an SNS Topic ARN using the Python SDK (Boto3).

In case an error arises during the Lambda execution, the Lambda error events are published to CloudWatch for troubleshooting. You can then review Lambda errors and debug the code as required. The Lambda function execution completes once the email message is sent successfully.

You can check out the entire code for the solution in the aws-samples GitHub repository.

Prerequisites

To implement this solution, you must create the following resources:

An AWS Account
AWS DMS task
SNS topic
AWS Identity and Access Management (IAM) role
Shell environment with AWS SAM CLI installed and configured

Create an AWS DMS task

Create an AWS DMS task for required source and target for data migrations. For instructions, refer to Creating a task.

Create an SNS topic

Amazon Simple Notification Service (Amazon SNS) is a managed service that provides message delivery from publishers to subscribers (also known as producers and consumers). Publishers communicate asynchronously with subscribers by sending messages to a topic, which is a logical access point and communication channel. Create an SNS topic for sending email notifications and confirm your email subscription.

Deployment

From an Infrastructure-as-Code (IaC) perspective, this post leverages the AWS Cloud Development Kit (CDK) to deploy the solution. Using the AWS CDK, we create the following resources.

AWS Lambda
Amazon CloudWatch Logs Subscription

To use the SAM CLI, you need the following tools.

SAM CLI – Install the SAM CLI
Python 3 installed

Next, follow these steps to build the solution.

Review the README section of the project available in the aws-samples GitHub repository.
Once you are satisfied with the logic and implementation, download and clone the repository
```
git clone https://github.com/aws-samples/dbs-blog-dms-events-sns-notification-cdk.git
```
Go to the project folder, to build and deploy your solution for the first time, run the following in your shell.
```
sam build
sam deploy –-guided
```
AWS SAM CLI will prompt values for the following required parameters

Parameters	Description
`pSNSTopicName`	Name of the SNS topic you created earlier.
`pLambdaVPCSecurityGroup`	Name of the Amazon VPC Security group for AWS Lambda function.
`pLambdaSubnetId1`	Name of the Amazon VPC subnet group for AWS Lambda function.
`pDMSLogGroupName`	Name of the Amazon CloudWatch Log group which you wish to monitor for AWS DMS errors.

Solution validation

To simulate the setup, we created an AWS DMS task in full load mode with table preparation mode as DO_NOTHING. The target table already had data from the source with a primary key. As a result, upon full load operation, AWS DMS tries to load duplicate data and the table suspends.

With our Lambda function and filter pattern for suspended, we receive the following error suspension notification.

Clean up

When you’re done using this solution, delete the following resources you created to avoid ongoing charges.

AWS DMS task
SNS topic
AWS Identity and Access Management (IAM) role
To delete the SAM solution that you created, use the AWS CLI. Assuming you used your project name for the stack name, you can run the following:
```
aws cloudformation delete-stack --stack-name DMSEventsNotifications
```

Conclusion

In this post, we demonstrated how you can get custom AWS DMS task alerts via email for specific errors in task logs using a Lambda trigger on a CloudWatch Logs group. Also, this setup can be useful in capturing AWS DMS errors or information from CloudWatch Logs that aren’t natively part of AWS DMS event notification.

Use this link to find more posts about monitoring databases with Amazon CloudWatch.

For more information on our service, see our documentation. We encourage you to innovate and use this solution to suit your business needs. Happy migrating!

About the authors

Sushant Deshmukh is a Database Consultant with AWS Professional Services Team. He works with AWS customers and partners to build highly available, scalable and secured database architectures on AWS. He provides technical design and implementation expertise in running database workloads on AWS, also helping customers migrate and modernize their databases to AWS Cloud. Outside of work, he enjoys traveling and exploring new places, playing volleyball and spending time with his family and friends.

Aritra Biswas is a Cloud Support DBA with Amazon Web Services and Subject Matter Expert for AWS Database Migration Service, he has over a decade of experience in working with relational databases. At AWS, He works with Service Teams, Technical Account Managers, Solutions Architects, and assists customers migrate database workloads to AWS. Outside of work he enjoys playing racquetball and spending time with family and friends.

Mihir Rathwa is a Cloud Application Architect at AWS Professional Services based in Dallas, Texas. He works with AWS customers on a broad range of services and technologies to build solutions for new levels of complexity, scale and performance on the cloud. Outside work, he loves visiting new places for food and culture, tinker his house with automation, and visiting his friends and family.

Parth Shah is a Startup Solutions Architect at Amazon Web Services. He enjoys working with startup customers in cloud adoption and business strategy as well as helping them design applications and services on AWS. Outside of work, he enjoys gaming, soccer, traveling, and spending time with his friends and family.

AWS Database Blog