Delete Amazon CloudWatch Synthetics dependent resources when you delete a CloudFormation stack

Amazon CloudWatch Synthetics allows you to monitor application endpoints more easily. It runs tests on your endpoints every minute, and alerts you if your application endpoints don’t behave as expected. These tests can be customized to check for availability, latency, transactions, broken or dead links, page load errors, load latencies for UI assets, complex wizard flows, or checkout flows in your applications.

You can use Amazon CloudWatch Synthetics to create canaries, which are configurable scripts that run on a schedule, to monitor your endpoints and APIs. A canary creates dependent resources in your account, such as AWS Lambda functions, CloudWatch logs, CloudWatch alarms, and Amazon Simple Storage Service (Amazon S3) buckets that hold the canary runs artifacts, such as logs and screenshot. However, when you delete a canary, these resources are not automatically deleted.

Similarly, you can use AWS CloudFormation to describe and provision all the infrastructure resources in your cloud environment. You can model your environment in JSON or YAML templates, or in code, by using tools like the AWS Cloud Development Kit (AWS CDK). When canary created from CloudFormation, the canary-dependent resources in your account are not deleted when you delete the stack. The stack deletion will fail if, for example, artifacts S3 bucket isn’t empty.

After you delete a canary that you do not intend to use again, you should also delete the following:

Lambda functions and layers used by the canary. Their prefix is cwsyn-MyCanaryName.
CloudWatch alarms created for the canary. These alarms have a name that starts with Synthetics-Alarm-MyCanaryName. For more information about deleting alarms, see Editing or Deleting a CloudWatch Alarm.
S3 objects and buckets, such as the canary’s results location and artifact location.
AWS Identity and Access Management (IAM) roles created for the canary. These have the name role/service-role/CloudWatchSyntheticsRole-MyCanaryName.
Log groups in CloudWatch Logs created for the canary. These logs groups have the name /aws/lambda/cwsyn-MyCanaryName.

In this post, I’ll show you how to use custom resources (along with AWS Lambda) in a CloudFormation template to delete CloudWatch Synthetics dependent resources.

Overview

I will use Custom resources to trigger a lambda function that will delete all canary dependant resources on CloudFormation stack deletion.

Custom resources enable you to write custom provisioning logic in templates when you create, update, or delete stacks. For example, if resources aren’t available as CloudFormation resource types, you can include them by using custom resources. This enables you to manage all your related resources in a single stack.

When you associate a Lambda function with a custom resource, the function is invoked whenever the custom resource is created, updated, or deleted. AWS CloudFormation calls the Lambda API to invoke the function and to pass the requested data (such as the request type and resource properties) to the function.

On stack deletion, the custom resource triggers the Lambda function passing the CloudWatch canary name and S3 artifact bucket name to delete canary-dependent resources.

On stack deletion, the custom resource triggers the cleaner Lambda function to delete dependent resources. The Lambda function signals state (success/fail) back to the custom resource.

Figure 1: Custom resource flow on stack deletion

Walkthrough

The CloudFormation template creates the following resources:

An S3 bucket for canary artifacts.
A canary execution role with standard permissions.
A canary of the heartbeat monitor type.
A Lambda function to clean up Synthetics resources.
An IAM role with permissions for the Lambda function to perform delete operations.
A CloudFormation custom resource that invokes the Lambda function on stack deletion.

Use the following button to launch the CloudFormation stack that creates an S3 artifact bucket and a heartbeat monitor canary. The canary uses the syn-nodejs-puppeteer-3.1 runtime version to monitor public endpoints.

Prerequisites

If you want to follow along make sure you have access to the AWS Management Console with the proper IAM permissions required to create or delete CloudWatch Synthetics Canaries, AWS Lambda functions, IAM roles, IAM policies and Amazon S3 buckets.

Create an S3 artifact bucket

Create an S3 bucket in the same AWS Region where you create the CloudFormation stack.

  CanaryBucket:
    Type: 'AWS::S3::Bucket'
    Properties:
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      Tags:
        - Key: Name
          Value: Canary Lab

Create a canary execution role

Create a canary execution role and follow the standard security advice of granting least privilege, or granting only the permissions required to perform a task. This role must include lambda.amazonaws.com as a principal in the trust policy. The role must also have the following permissions:

s3:PutObject
s3:GetBucketLocation
s3:ListAllMyBuckets
cloudwatch:PutMetricData
xray:PutTraceSegments
logs:CreateLogGroup
logs:CreateLogStream
logs:PutLogEvents

If you want to create a canary on a VPC, you need to add permissions for ec2:CreateNetworkInterface, ec2:DescribeNetworkInterface, and ec2:DeleteNetworkInterface.

  CanaryExecutionRole:
    Type: 'AWS::IAM::Role'
    Properties:
      Description: Role used to run CW Canaries
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: 'sts:AssumeRole'
            Principal:
              Service: lambda.amazonaws.com
      Policies:
        - PolicyName: CanaryCFNRoleLabPermissions
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 's3:ListAllMyBuckets'
                  - 'xray:PutTraceSegments'
                Resource: '*'
              - Effect: Allow
                Action:
                  - 's3:GetBucketLocation'
                  - 's3:PutObject'
                Resource:
                  - !Sub '${CanaryBucket.Arn}/*'
              - Effect: Allow
                Action:
                  - 'logs:CreateLogGroup'
                  - 'logs:CreateLogStream'
                  - 'logs:PutLogEvents'
                Resource: !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/cwsyn-${CanaryName}-*'
              - Effect: Allow
                Action:
                  - 'cloudwatch:PutMetricData'
                Resource: '*'
                Condition:
                  StringEquals:
                    cloudwatch:namespace: 'CloudWatchSynthetics'

Create a CloudWatch canary

Use a blueprint or your custom code to create a canary to monitor your endpoint. To create canaries, your IAM role/user must have the CloudWatchSyntheticsFullAccess policy. If you are creating an IAM role for the canary, you also need the iam:CreateRole, iam:CreatePolicy, and iam:AttachRolePolicy permissions. For more information, see Required roles and permissions for CloudWatch canaries in the Amazon CloudWatch User Guide.

  CloudWatchSyntheticsCanary:
    Type: 'AWS::Synthetics::Canary'
    Properties:
      Tags:
        - Key: Name
          Value: Canary Lab
      Name: !Ref CanaryName
      ExecutionRoleArn: !GetAtt CanaryExecutionRole.Arn 
      Code:
        Handler: pageLoadBlueprint.handler
        Script:  |
            const URL = require('url');
            const synthetics = require('Synthetics');
            const log = require('SyntheticsLogger');
            const syntheticsConfiguration = synthetics.getConfiguration();
            
            const loadBlueprint = async function () {
            
                const urls = [process.env.EndpointURL];
                
                // Set screenshot option
                const takeScreenshot = true;
            
                /* Disabling default step screen shots taken during Synthetics.executeStep() calls
                * Step will be used to publish metrics on time taken to load dom content but
                * Screenshots will be taken outside the executeStep to allow for page to completely load with domcontentloaded
                * You can change it to load, networkidle0, networkidle2 depending on what works best for you.
                */
                syntheticsConfiguration.disableStepScreenshots();
                syntheticsConfiguration.setConfig({
                  continueOnStepFailure: true
                });
            
                let page = await synthetics.getPage();
                
                for (const url of urls) {
                    await loadUrl(page, url, takeScreenshot);
                }
            };

            // Reset the page in-between
            const resetPage = async function(page) {
                try {
                    await page.goto('about:blank',{waitUntil: ['load', 'networkidle0'], timeout: 30000} );
                } catch(ex) {
                    synthetics.addExecutionError('Unable to open a blank page ', ex);
                }
            }
            
            const loadUrl = async function (page, url, takeScreenshot) {
                let stepName = null;
                let domcontentloaded = false;
            
                try {
                    stepName = URL.parse(url).hostname;
                } catch (error) {
                    const errorString = `Error parsing url: ${url}.  ${error}`;
                    log.error(errorString);
                    /* If we fail to parse the URL, don't emit a metric with a stepName based on it.
                      It may not be a legal CloudWatch metric dimension name and we may not have an alarms
                      setup on the malformed URL stepName.  Instead, fail this step which will
                      show up in the logs and will fail the overall canary and alarm on the overall canary
                      success rate.
                    */
                    throw error;
                }
                
                await synthetics.executeStep(stepName, async function () {
                    
                    /* You can customize the wait condition here. For instance, using 'networkidle2' or 'networkidle0' to load page completely.
                      networkidle0: Navigation is successful when the page has had no network requests for half a second. This might never happen if page is constantly loading multiple resources.
                      networkidle2: Navigation is successful when the page has no more then 2 network requests for half a second.
                      domcontentloaded: It's fired as soon as the page DOM has been loaded, without waiting for resources to finish loading. Can be used and then add explicit await page.waitFor(timeInMs) 
                    */
                    const response = await page.goto(url, { waitUntil: ['domcontentloaded'], timeout: 30000});
                    if (response) {
                        domcontentloaded = true;
                        const status = response.status();
                        const statusText = response.statusText();
            
                        const logResponseString = `Response from url: ${url}  Status: ${status}  Status Text: ${statusText}`;
                        
                        //If the response status code is not a 2xx success code
                        if (response.status() < 200 || response.status() > 299) {
                            throw `Failed to load url: ${url} ${response.status()} ${response.statusText()}`; 
                        }
                    } else {
                        const logNoResponseString = `No response returned for url: ${url}`;
                        log.error(logNoResponseString);
                        throw new Error(logNoResponseString);
                    }
                });
                
                // Wait for 15 seconds to let page load fully before taking screenshot.
                if (domcontentloaded && takeScreenshot) {
                    await page.waitFor(15000);
                    await synthetics.takeScreenshot(stepName, 'loaded');
                    await resetPage(page);
                }
            };
            
            const urls = [];
            
            exports.handler = async () => {
                return await loadBlueprint();
            };
      RuntimeVersion: syn-nodejs-puppeteer-3.1
      ArtifactS3Location: !Sub 's3://${CanaryBucket}'
      Schedule:
        Expression: rate(5 minutes)
      RunConfig:
        TimeoutInSeconds: 60
        EnvironmentVariables: 
          EndpointURL: !Ref MonitoredEndpointURL
      FailureRetentionPeriod: 30
      SuccessRetentionPeriod: 30
      StartCanaryAfterCreation: true

Create a Lambda execution IAM role

Create a Lambda execution role and follow the standard security advice of granting least privilege. This role must have permission to delete the Lambda function, CloudWatch logs, alarms if they exist, the S3 bucket, and S3 artifact objects for CloudFormation stack resources CloudWatchSyntheticsCanary and CanaryBucket.

  CleanerLambdaExecutionRole:
    Type: 'AWS::IAM::Role'
    Properties:
      Description: Role used to run Cleaner lambda function on stack deletion
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: 'sts:AssumeRole'
            Principal:
              Service: lambda.amazonaws.com
      Policies:
        - PolicyName: CleanerLambdaCFNRoleLabPermissions
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - 'lambda:DeleteFunction'
                  - 's3:DeleteObject'
                  - 'logs:DeleteLogGroup'
                  - 'cloudwatch:DeleteAlarms'
                Resource:
                  - !Sub 'arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:cwsyn-${CloudWatchSyntheticsCanary}-*'
                  - !Sub '${CanaryBucket.Arn}/*'
                  - !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/cwsyn-${CloudWatchSyntheticsCanary}-*'
                  - !Sub 'arn:aws:cloudwatch:${AWS::Region}:${AWS::AccountId}:alarm:Synthetics-Alarm-${CloudWatchSyntheticsCanary}-*'                    
              - Effect: Allow
                Action:
                  - 's3:ListBucket'
                Resource:
                  - !GetAtt 
                    - CanaryBucket
                    - Arn                                       
              - Effect: Allow
                Action:
                  - 'synthetics:Get*'
                  - 'cloudwatch:DescribeAlarms'
                Resource: 
                  - !Sub 'arn:aws:synthetics:${AWS::Region}:${AWS::AccountId}:canary:${CloudWatchSyntheticsCanary}'
                  - !Sub 'arn:aws:cloudwatch:${AWS::Region}:${AWS::AccountId}:alarm:*'

Create the cleaner Lambda function

The CloudFormation custom resource sends two events to the Lambda function:

On Create, the Lambda function will receive the following event:

{
  "RequestType": "Create",
  "ServiceToken": "arn:aws:lambda:[[Region id]]:[[Account ID]]:function:[[custom Lambda Name]]",
  "ResponseURL": "https://cloudformation-custom-resource-response-useast2.s3.[[Region id]].amazonaws.com/arn%3Aaws%3Acloudformation%3A[[Region id]]%3A[[Account ID]]%3Astack/Test123412/f4abac20-382c-11eb-b555-0296ed261c1a%7CwaitStackOnCreation%7C078a0d24-f0bf-4181-a831-0369b5fa2d58?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20201207T014015Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=AKIAVRFIPK6PLHKRTTNP%2F20201207%2F[[Region id]]%2Fs3%2Faws4_request&X-Amz-Signature=681b6b9c35e530d03348d965ef0d397082abac4e34515ac5fa840e8fce1e9f03",
  "StackId": "arn: aws: cloudformation: [[Region id]]: [[Account ID]]: stack/Test123412/f4abac20-382c-11eb-b555-0296ed261c1a",
  "RequestId": "078a0d24-f0bf-4181-a831-0369b5fa2d58",
  "LogicalResourceId": "CustomResourceLogicalID",
  "ResourceType": "Custom::cleanCanaryStack",
  "ResourceProperties": {
    "ServiceToken": "arn:aws:lambda:[[Region id]]:[[Account ID]]:function:[[custom Lambda Name]]",
    "BucketName": "[Bucket Resource Name]",
    "CanaryName": "[Canary Resource Name]"
  }
}

On Delete, the Lambda function will receive the following event:

{
  "RequestType": "Delete",
  "ServiceToken": "arn:aws:lambda:[[Region id]]:[[Account ID]]:function:[[custom Lambda Name]]",
  "ResponseURL": "https://cloudformation-custom-resource-response-useast2.s3.[[Region id]].amazonaws.com/arn%3Aaws%3Acloudformation%3A[[Region id]]%3A[[Account ID]]%3Astack/Lab2/dc67bf00-385e-11eb-8657-065e0005c80e%7CwaitStackOnCreation%7C8531a79f-0d4b-48fa-b851-2bfeccd1dbc0?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20201207T115755Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=AKIAVRFIPK6PLHKRTTNP%2F20201207%2F[[Region id]]%2Fs3%2Faws4_request&X-Amz-Signature=f8f8313da98b411d3d739d512a9844930f5c50c610cfce43bc10f2caf89ee76f",
  "StackId": "arn:aws:cloudformation:[[Region id]]:[[Account ID]]:stack/Lab2/dc67bf00-385e-11eb-8657-065e0005c80e",
  "RequestId": "8531a79f-0d4b-48fa-b851-2bfeccd1dbc0",
  "LogicalResourceId": "CustomResourceLogicalID",
  "PhysicalResourceId": "2020/12/07/[$LATEST]d9343e2ce5bc443e86ff6695befa87dd",
  "ResourceType": "Custom::cleanCanaryStack",
  "ResourceProperties": {
    "ServiceToken": "arn:aws:lambda:[[Region id]]:[[Account ID]]:function:[[custom Lambda Name]]",
    "BucketName": "[Bucket Resource Name]",
    "CanaryName": "[Canary Resource Name]"
  }
}

I used inline Python code with the cfn-response Python module to signal responses back to CloudFormation template based on the invocation result. If the invocation event is create the Lambda code will always return a responseStatus of SUCCESS. If the invocation event is Delete, it parse the ResourceProperties field in received JSON event to get the canary resource name and S3 artifact bucket name, then using the AWS SDK for Python (Boto3), the code does the following:

Retrieves canary details to locate the Lambda function.
Deletes the Lambda function.
Deletes the CloudWatch log group for the canary Lambda function.
Searches for and deletes alarms that have the prefix of Synthetics-Alarm-MyCanaryName.
Deletes all objects inside the S3 artifact bucket.

In the case of any exception, Lambda will return a responseStatus of FAILED to the CloudFormation stack along with information about which resource returned the error.

  cleanupCanaryStackOnDelete:
    Type: 'Custom::cleanCanaryStack'   
    Properties:
      ServiceToken: !GetAtt CleanCanaryStackLambda.Arn
      BucketName: !Ref CanaryBucket
      CanaryName: !Ref CloudWatchSyntheticsCanary

  CleanCanaryStackLambda:
    Type: 'AWS::Lambda::Function'
    Properties:
      Code:
        ZipFile: |
          import json
          import boto3
          import cfnresponse

          #Initialize boto3 clients for synthetics,lambda, CloudWatch logs,cloudwatch and s3
          synthetics = boto3.client('synthetics')
          lambdaClient = boto3.client('lambda') 
          logs = boto3.client('logs')
          cloudwatch = boto3.client('cloudwatch')
          s3 = boto3.resource('s3')

          def lambda_handler(event, context):
            try:
                bucketName = event['ResourceProperties']['BucketName']
                CanaryName = event['ResourceProperties']['CanaryName']
                Errors=[]
                if event['RequestType'] == 'Delete':

                  try:
                    # Find Lambada name corresponding to your Canary name
                    Canarydetails = synthetics.get_canary(Name=CanaryName)
                    CanaryLambdaName = Canarydetails['Canary']['EngineArn'].split(':')[-2]
                    CanaryArtifactBucket = Canarydetails['Canary']['ArtifactS3Location']
                  except Exception as e:
                    print('Get-Canary failed: '+str(e))
                    Errors.append('Get-Canary failed: '+str(e))
                    pass
                  try:
                    #delete Canary Lambda
                    lambdaClient.delete_function(FunctionName=CanaryLambdaName)
                  except Exception as e:
                    print('Deleting Canary Lambda failed: '+str(e)) 
                    Errors.append('Deleting Canary Lambda failed: '+str(e))
                    pass
                  try:
                    #delete Canary log group
                    CanaryLogGroup='/aws/lambda/'+CanaryLambdaName
                    logs.delete_log_group(logGroupName=CanaryLogGroup)
                  except Exception as e:
                    print('Deleting Canary logs failed: '+str(e))
                    Errors.append('Deleting Canary logs failed: '+str(e))
                    pass

                  # Delete Alarms
                  try:
                    # retrun all alarms metricAlarm and Composite Alarms with prefix 'Synthetics-Alarm-CanaryName' 
                    Alarms = cloudwatch.describe_alarms(AlarmNamePrefix='Synthetics-Alarm-'+CanaryName,AlarmTypes=['MetricAlarm'])
                    
                    # Create list of all metricAlarms as deleteAlarms API can delete up to 100 alarm in one API
                    metricAlarms=[]
                    for alarm in Alarms['MetricAlarms']:
                      metricAlarms.append(alarm['AlarmName'])
                    if metricAlarms:
                      cloudwatch.delete_alarms(AlarmNames=metricAlarms)
                  except Exception as e:
                    print('Deleting Canary Alarm failed: '+str(e))
                    Errors.append('Deleting Canary Alarm failed: '+str(e))
                  
                  #Find Bucket and delete canary objects under prefix if exist
                  try:                    
                    bucket = s3.Bucket(bucketName)
                    #Check if there is prefix in the Canary artifact bucket
                    if (CanaryArtifactBucket.find(bucketName+'/') != -1):
                      #search for objects under prefix
                      objects = bucket.objects.filter(Prefix=CanaryArtifactBucket.split(bucketName+'/')[1])
                    else:
                      #If no prefix, search for all objects under the bucket
                      objects = bucket.objects.filter()

                    for obj in objects:
                      # delete objects with canary prefix
                      s3.Object(bucket.name, obj.key).delete()
                  except Exception as e:
                    print('Deleting S3 Objects failed: '+str(e))
                    Errors.append('Deleting S3 Objects failed: '+str(e))
                    pass
                #IF any of dependency resources was not deleted raise exception to send custom error message to stack
                if Errors:
                  raise Exception(Errors) 
                cfnresponse.send(event, context, cfnresponse.SUCCESS,json.loads("{}"),None,False,'Reason: Stack cleaned successfully')    

            except Exception as e:
                print(e)
                cfnresponse.send(event, context, cfnresponse.FAILED,json.loads("{}"),None,False,'Reason: ' + str(e))          
                   
          
      Handler: index.lambda_handler
      FunctionName: !Sub 'CleanCanaryResources-${AWS::StackName}'
      Role: !GetAtt CleanerLambdaExecutionRole.Arn
      Runtime: python3.8
      Timeout: 900

Create a custom resource

Give the custom resource a name like Custom::cleanCanaryStack. Under Properties, in the ServiceToken property, we point toe the Lambda function ARN. We define resources to pass to the Lambda function in the invocation event, such as the S3 artifact bucket resource name and the canary resource name.

  cleanupCanaryStackOnDelete:
    Type: 'Custom::cleanCanaryStack'   
    Properties:
      ServiceToken: !GetAtt CleanCanaryStackLambda.Arn
      BucketName: !Ref CanaryBucket
      CanaryName: !Ref CloudWatchSyntheticsCanary

Create a CloudFormation stack

Open the CloudFormation console and choose Create Stack.
Choose with new resources (standard).
Choose Upload a template file, and then choose your CloudFormation template file.
Enter a name for your stack, complete the Parameters section, and then choose Next.
Keep the defaults on the Configure stack options page, and then choose Next.
On the Review page, select the checkbox to acknowledge that AWS CloudFormation might create IAM resources, and then choose Create Stack.

Figure 2 shows the created resources:

The Resources tab displays CanaryBucket, CanaryExecutionRole, CleanCanaryStackLambda, CloudWatchSyntheticsCanary, CleanerLambdaExecutionRole, and cleanupCanaryStackOnDelete. All have a Status of CREATE_COMPLETE.

Figure 2: Created resources

Test stack deletion result

On stack deletion, custom resource invokes the lambda function that will delete all canary dependant resources and empty the S3 artifacts bucket. If Lambda failed to empty the S3 artifacts bucket the stack deletion would have failed to delete the S3 artifacts bucket.

To delete a stack

Open the AWS CloudFormation console.
On the Stacks page in the CloudFormation console, select the stack that you want to delete. The stack must be currently running.
In the stack details pane, choose Delete.
Select Delete stack when prompted.

The Events tab displays CanaryBucket, CanaryExecutionRole, CleanCanaryStackLambda, CloudWatchSyntheticsCanary, CleanerLambdaExecutionRole, and cleanupCanaryStackOnDelete. All have a Status of DELETE_COMPLETE.

Figure 3: Deleted resources events

Cleanup

To avoid charges to your account, delete the CloudFormation stack and resources. For more information, see Deleting a stack on the AWS CloudFormation console in the AWS CloudFormation User Guide.

Conclusion

In this blog post I showed you how to use a Lambda-backed custom resource to delete Amazon CloudWatch Synthetics dependent resources, such as AWS Lambda functions, CloudWatch logs, CloudWatch alarms, and S3 artifact buckets and objects.

Although we used the Lambda-backed custom resource on stack deletion, Lambda functions in combination with AWS CloudFormation enable a range of scenarios, such as dynamically looking up AMI IDs during stack creation or implementing and using utility functions, such as string reversal functions. You can also write your own resources to extend AWS CloudFormation beyond AWS resources and provision any other resource you can think of. For example, you can integrate a third-party software as a service (SaaS) product or you can even provision on-premises resources in hybrid environments.

Select your cookie preferences

AWS Cloud Operations Blog