Testing and debugging Amazon CloudWatch Synthetics canary locally

Introduction

Amazon CloudWatch Synthetics canaries are scripts that monitor your endpoints and APIs by simulating the actions of a user. These canaries run on a schedule, check the availability and latency of your applications, and alert you when there are issues. Canary scripts are written in Node.js and Python, and they run inside an AWS Lambda function. The development process for canaries closely mirrors that of applications, involving tasks like coding, testing, debugging and deployment. However, unlike traditional application development, there is no local testing and debugging environment to seamlessly code, test and iterate quickly. Due to the absence of a testing environment, you may rely on less efficient methods, resulting in prolonged canary development time. This in turn increases operational load for you in maintaining canaries because additional time is spent in testing canary code.

In this blog post, you will set up a local testing environment to test and debug Synthetics canaries locally using Visual Studio Code IDE.

Overview of solution

Using AWS Serverless Application Model (SAM) and Visual Studio Code editor you will set up a local development environment for a Synthetics canary. The Lambda function defined in a SAM template will emulate the canary behavior locally. Using debug breakpoints in Visual Studio Code editor you will debug a sample canary.

Walkthrough

This blog post provides a step-by-step process for setting up a local testing environment to test and debug Synthetics canaries. By following a few steps, you will be able to modify code, test, and debug NodeJS Canaries directly within the Visual Studio Code editor. The local testing environment uses a Serverless Application Model (SAM) container, to simulate an AWS Lambda function to emulate the behavior of a Synthetics canary. By the end of this guide, you will learn the following:

- Set up local testing environment in few simple steps
- Debug a Synthetics canary locally and iterate quickly
- Integrate 3rd party dependencies with canary code
- Integrate local debugging into an existing canary source package

Pre-requisites

For this walkthrough, you should have the following prerequisites:

1. Choose or create an Amazon S3 bucket for canary artifacts.
  Note: You can skip this step if you wish. The canary can still be run locally, but it will not be able to upload artifacts to the S3 bucket. The canary logs will contain errors related to this, but you can safely ignore them. If you use an Amazon S3 bucket, we recommend that you set the bucket lifecycle to delete objects after a few days. For more information, see Managing your storage lifecycle.
2. Set up a default AWS profile for your AWS account.
3. Set default region to your preferred AWS Region (ex: us-west-2).
4. Install latest version of AWS Serverless Application Model (SAM) CLI installed.
5. Install Visual Studio Code IDE. You can choose other IDEs but this guide provides instructions to work with Visual Studio Code only.
6. Install Docker and start the daemon.
7. AWS Toolkit for VS Code extension.

Setting up local testing environment for NodeJS canary

1. Clone aws-samples/synthetics-canary-local-debugging-sample repository using this cmd:

git clone https://github.com/aws-samples/synthetics-canary-local-debugging-sample.git

2. Go to NodeJS canary source directory.

cd synthetics-canary-local-debugging-sample/nodejs-canary/src

3. Run npm install to install canary dependencies.

Launch configuration

The launch configuration file is present at .vscode/launch.json. It contains configuration to allow the template file to be discovered by Visual Studio Code. It defines a Lambda payload with the required parameters to invoke the canary successfully. Here is the launch configuration for NodeJS canary:

{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "aws-sam",
            "request": "direct-invoke",
            "name": "SAM:Run NodeJS Canary",
            "invokeTarget": {
                "target": "template",
                "templatePath": "${workspaceFolder}/nodejs-canary/template.yml",
                "logicalId": "NodeJSPuppeteerCanary"
            },
            "lambda": {
                "runtime": "nodejs18.x",
                "payload": {
                    "json": {
                        "canaryName": "LocalSyntheticsCanary",
                        "artifactS3Location": {
                            "s3Bucket": "cw-syn-results-123456789012-us-west-2",
                            "s3Key": "local-run-artifacts"
                        },
                        "customerCanaryHandlerName": "heartbeat-canary.handler"
                    }
                },
                "environmentVariables": {}
            }
        }
    ]
}

Please note:

1. Make sure you modify the payload JSON with your own S3 bucket (s3Bucket) for canary artifacts. You can provide any name for the canary (canaryName).
2. Other optional fields that you can provide in payload JSON:

s3EncryptionMode: valid values: SSE_S3 | SSE_KMS

s3KmsKeyArn: <KMS Key ARN>

activeTracing: valid values: true | false

canaryRunId: <UUID> (required if active tracing is enabled)

At this point your package structure should look like this

└── synthetics-canary-local-debugging-sample
    ├── nodejs-canary
    │ ├── src
    │ │ ├── cw-synthetics.js
    │ │ ├── heartbeat-canary.js
    │ │ ├── node_modules
    │ │ │ └── @faker-js
    │ │ └── package.json
    │ └── template.yml

Debugging canary

Add breakpoints in the canary code where you wish to pause execution by clicking on the editor margin and then go to Run and Debug mode. Execute the canary by clicking on the play button. When the canary executes, the logs will be tailed in the debug console. If you added breakpoints the canary execution will pause at each breakpoint, allowing you step through code and inspect variable values, instance methods, object attributes, function call stack etc.

After the canary execution is completed, the artifacts (logs, screenshots, HAR and reports) will be uploaded to the Amazon S3 bucket specified in the lambda payload in the launch configuration. You can view the artifacts either in the Amazon S3 console or directly in the editor in the AWS Toolkit window. Please note that artifacts, excluding logs, will be overwritten when you run the canary next time because the screenshot file names and report file name do not change, except for the log file name. If you wish to retain the artifacts, ensure to provide a different Amazon S3 key in the payload JSON so that artifacts will be saved to a different Amazon S3 key. Additionally, canary metrics, including SuccessPercent, Duration, Step metrics, and HTTP status code metrics, will also be published for the run under the CloudWatchSynthetics namespace for the canary (ex: LocalSyntheticsCanary), which can be viewed in the CloudWatch Metrics console.

There is no cost incurred for running and debugging canary locally except for the artifacts stored in Amazon S3 bucket and the Synthetics canary metrics generated by each local run.

Synthetics canary running in debug mode

Figure 1: Synthetics canary running in debug mode

Note
Debugging visual monitoring canaries poses a challenge due to the reliance on base screenshots captured during the initial run for subsequent comparisons. In a local development environment, runs are neither stored nor tracked, treating each iteration as an independent, standalone run. Consequently, the absence of a run history makes it impractical to perform debugging for canaries relying on visual monitoring.

Integrating local testing environment into your canary package

You can easily integrate local canary testing environment into your existing canary package by copying three files:

1. Put the template.yml file into package root. Make sure to modify the path for CodeUri to point to the directory where your canary code exists.
2. Put cw-synthetics.js file in canary source directory. Note: Ensure that you retrieve the most recent version of the file from the GitHub repository, as older versions may not be compatible with future runtimes or may not accurately replicate the actual canary behavior.
3. Put launch configuration file (.vscode/launch.json) in package root. Make sure to put it inside .vscode directory; create it if it does not exist already.

Cleaning up

Since the canary is executed locally, the only cleanup required is to remove any canary artifacts that were uploaded to the Amazon S3 bucket. You should delete those objects from the S3 bucket.

Conclusion

Testing Synthetics canaries locally offers developers a powerful and efficient means to identify and address issues quickly and provides the workflow and tools similar to application development that you are already familiar with. It’s important to note that while local debugging is a valuable tool it cannot fully replicate the real-world production environment. Elements such as network latency, caching, transient website issues, database I/O latency and other external factors unique to a live environment may not be accurately reproduced in a local setup. Therefore, it is essential to conduct thorough testing in the actual production environment to account for these external variables and ensure the canaries perform optimally under real-world conditions.

Debugging a Synthetics Python canary follows the same procedure as discussed in the blog post. You can find code samples for Python canaries in the python-canary directory of the sample repository. For additional information on debugging Python canaries, debugging in JetBrains IDEs, and more, refer to the CloudWatch Synthetics documentation.

AWS Cloud Operations & Migrations Blog