AWS Cloud Operations Blog
Delete Amazon CloudWatch Synthetics dependent resources when you delete a CloudFormation stack
Amazon CloudWatch Synthetics allows you to monitor application endpoints more easily. It runs tests on your endpoints every minute, and alerts you if your application endpoints don’t behave as expected. These tests can be customized to check for availability, latency, transactions, broken or dead links, page load errors, load latencies for UI assets, complex wizard flows, or checkout flows in your applications.
You can use Amazon CloudWatch Synthetics to create canaries, which are configurable scripts that run on a schedule, to monitor your endpoints and APIs. A canary creates dependent resources in your account, such as AWS Lambda functions, CloudWatch logs, CloudWatch alarms, and Amazon Simple Storage Service (Amazon S3) buckets that hold the canary runs artifacts, such as logs and screenshot. However, when you delete a canary, these resources are not automatically deleted.
Similarly, you can use AWS CloudFormation to describe and provision all the infrastructure resources in your cloud environment. You can model your environment in JSON or YAML templates, or in code, by using tools like the AWS Cloud Development Kit (AWS CDK). When canary created from CloudFormation, the canary-dependent resources in your account are not deleted when you delete the stack. The stack deletion will fail if, for example, artifacts S3 bucket isn’t empty.
After you delete a canary that you do not intend to use again, you should also delete the following:
- Lambda functions and layers used by the canary. Their prefix is
cwsyn-MyCanaryName. - CloudWatch alarms created for the canary. These alarms have a name that starts with
Synthetics-Alarm-MyCanaryName. For more information about deleting alarms, see Editing or Deleting a CloudWatch Alarm. - S3 objects and buckets, such as the canary’s results location and artifact location.
- AWS Identity and Access Management (IAM) roles created for the canary. These have the name
role/service-role/CloudWatchSyntheticsRole-MyCanaryName. - Log groups in CloudWatch Logs created for the canary. These logs groups have the name
/aws/lambda/cwsyn-MyCanaryName.
In this post, I’ll show you how to use custom resources (along with AWS Lambda) in a CloudFormation template to delete CloudWatch Synthetics dependent resources.
Overview
I will use Custom resources to trigger a lambda function that will delete all canary dependant resources on CloudFormation stack deletion.
Custom resources enable you to write custom provisioning logic in templates when you create, update, or delete stacks. For example, if resources aren’t available as CloudFormation resource types, you can include them by using custom resources. This enables you to manage all your related resources in a single stack.
When you associate a Lambda function with a custom resource, the function is invoked whenever the custom resource is created, updated, or deleted. AWS CloudFormation calls the Lambda API to invoke the function and to pass the requested data (such as the request type and resource properties) to the function.
On stack deletion, the custom resource triggers the Lambda function passing the CloudWatch canary name and S3 artifact bucket name to delete canary-dependent resources.
Figure 1: Custom resource flow on stack deletion
Walkthrough
The CloudFormation template creates the following resources:
- An S3 bucket for canary artifacts.
- A canary execution role with standard permissions.
- A canary of the heartbeat monitor type.
- A Lambda function to clean up Synthetics resources.
- An IAM role with permissions for the Lambda function to perform delete operations.
- A CloudFormation custom resource that invokes the Lambda function on stack deletion.
Use the following button to launch the CloudFormation stack that creates an S3 artifact bucket and a heartbeat monitor canary. The canary uses the syn-nodejs-puppeteer-3.1 runtime version to monitor public endpoints.
Prerequisites
If you want to follow along make sure you have access to the AWS Management Console with the proper IAM permissions required to create or delete CloudWatch Synthetics Canaries, AWS Lambda functions, IAM roles, IAM policies and Amazon S3 buckets.
Create an S3 artifact bucket
Create an S3 bucket in the same AWS Region where you create the CloudFormation stack.
CanaryBucket:
Type: 'AWS::S3::Bucket'
Properties:
PublicAccessBlockConfiguration:
BlockPublicAcls: true
BlockPublicPolicy: true
IgnorePublicAcls: true
RestrictPublicBuckets: true
Tags:
- Key: Name
Value: Canary LabCreate a canary execution role
Create a canary execution role and follow the standard security advice of granting least privilege, or granting only the permissions required to perform a task. This role must include lambda.amazonaws.com as a principal in the trust policy. The role must also have the following permissions:
s3:PutObjects3:GetBucketLocations3:ListAllMyBucketscloudwatch:PutMetricDataxray:PutTraceSegmentslogs:CreateLogGrouplogs:CreateLogStreamlogs:PutLogEvents
If you want to create a canary on a VPC, you need to add permissions for ec2:CreateNetworkInterface, ec2:DescribeNetworkInterface, and ec2:DeleteNetworkInterface.
CanaryExecutionRole:
Type: 'AWS::IAM::Role'
Properties:
Description: Role used to run CW Canaries
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: 'sts:AssumeRole'
Principal:
Service: lambda.amazonaws.com
Policies:
- PolicyName: CanaryCFNRoleLabPermissions
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 's3:ListAllMyBuckets'
- 'xray:PutTraceSegments'
Resource: '*'
- Effect: Allow
Action:
- 's3:GetBucketLocation'
- 's3:PutObject'
Resource:
- !Sub '${CanaryBucket.Arn}/*'
- Effect: Allow
Action:
- 'logs:CreateLogGroup'
- 'logs:CreateLogStream'
- 'logs:PutLogEvents'
Resource: !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/cwsyn-${CanaryName}-*'
- Effect: Allow
Action:
- 'cloudwatch:PutMetricData'
Resource: '*'
Condition:
StringEquals:
cloudwatch:namespace: 'CloudWatchSynthetics'
Create a CloudWatch canary
Use a blueprint or your custom code to create a canary to monitor your endpoint. To create canaries, your IAM role/user must have the CloudWatchSyntheticsFullAccess policy. If you are creating an IAM role for the canary, you also need the iam:CreateRole, iam:CreatePolicy, and iam:AttachRolePolicy permissions. For more information, see Required roles and permissions for CloudWatch canaries in the Amazon CloudWatch User Guide.
CloudWatchSyntheticsCanary:
Type: 'AWS::Synthetics::Canary'
Properties:
Tags:
- Key: Name
Value: Canary Lab
Name: !Ref CanaryName
ExecutionRoleArn: !GetAtt CanaryExecutionRole.Arn
Code:
Handler: pageLoadBlueprint.handler
Script: |
const URL = require('url');
const synthetics = require('Synthetics');
const log = require('SyntheticsLogger');
const syntheticsConfiguration = synthetics.getConfiguration();
const loadBlueprint = async function () {
const urls = [process.env.EndpointURL];
// Set screenshot option
const takeScreenshot = true;
/* Disabling default step screen shots taken during Synthetics.executeStep() calls
* Step will be used to publish metrics on time taken to load dom content but
* Screenshots will be taken outside the executeStep to allow for page to completely load with domcontentloaded
* You can change it to load, networkidle0, networkidle2 depending on what works best for you.
*/
syntheticsConfiguration.disableStepScreenshots();
syntheticsConfiguration.setConfig({
continueOnStepFailure: true
});
let page = await synthetics.getPage();
for (const url of urls) {
await loadUrl(page, url, takeScreenshot);
}
};
// Reset the page in-between
const resetPage = async function(page) {
try {
await page.goto('about:blank',{waitUntil: ['load', 'networkidle0'], timeout: 30000} );
} catch(ex) {
synthetics.addExecutionError('Unable to open a blank page ', ex);
}
}
const loadUrl = async function (page, url, takeScreenshot) {
let stepName = null;
let domcontentloaded = false;
try {
stepName = URL.parse(url).hostname;
} catch (error) {
const errorString = `Error parsing url: ${url}. ${error}`;
log.error(errorString);
/* If we fail to parse the URL, don't emit a metric with a stepName based on it.
It may not be a legal CloudWatch metric dimension name and we may not have an alarms
setup on the malformed URL stepName. Instead, fail this step which will
show up in the logs and will fail the overall canary and alarm on the overall canary
success rate.
*/
throw error;
}
await synthetics.executeStep(stepName, async function () {
/* You can customize the wait condition here. For instance, using 'networkidle2' or 'networkidle0' to load page completely.
networkidle0: Navigation is successful when the page has had no network requests for half a second. This might never happen if page is constantly loading multiple resources.
networkidle2: Navigation is successful when the page has no more then 2 network requests for half a second.
domcontentloaded: It's fired as soon as the page DOM has been loaded, without waiting for resources to finish loading. Can be used and then add explicit await page.waitFor(timeInMs)
*/
const response = await page.goto(url, { waitUntil: ['domcontentloaded'], timeout: 30000});
if (response) {
domcontentloaded = true;
const status = response.status();
const statusText = response.statusText();
const logResponseString = `Response from url: ${url} Status: ${status} Status Text: ${statusText}`;
//If the response status code is not a 2xx success code
if (response.status() < 200 || response.status() > 299) {
throw `Failed to load url: ${url} ${response.status()} ${response.statusText()}`;
}
} else {
const logNoResponseString = `No response returned for url: ${url}`;
log.error(logNoResponseString);
throw new Error(logNoResponseString);
}
});
// Wait for 15 seconds to let page load fully before taking screenshot.
if (domcontentloaded && takeScreenshot) {
await page.waitFor(15000);
await synthetics.takeScreenshot(stepName, 'loaded');
await resetPage(page);
}
};
const urls = [];
exports.handler = async () => {
return await loadBlueprint();
};
RuntimeVersion: syn-nodejs-puppeteer-3.1
ArtifactS3Location: !Sub 's3://${CanaryBucket}'
Schedule:
Expression: rate(5 minutes)
RunConfig:
TimeoutInSeconds: 60
EnvironmentVariables:
EndpointURL: !Ref MonitoredEndpointURL
FailureRetentionPeriod: 30
SuccessRetentionPeriod: 30
StartCanaryAfterCreation: true
Create a Lambda execution IAM role
Create a Lambda execution role and follow the standard security advice of granting least privilege. This role must have permission to delete the Lambda function, CloudWatch logs, alarms if they exist, the S3 bucket, and S3 artifact objects for CloudFormation stack resources CloudWatchSyntheticsCanary and CanaryBucket.
CleanerLambdaExecutionRole:
Type: 'AWS::IAM::Role'
Properties:
Description: Role used to run Cleaner lambda function on stack deletion
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: 'sts:AssumeRole'
Principal:
Service: lambda.amazonaws.com
Policies:
- PolicyName: CleanerLambdaCFNRoleLabPermissions
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- 'lambda:DeleteFunction'
- 's3:DeleteObject'
- 'logs:DeleteLogGroup'
- 'cloudwatch:DeleteAlarms'
Resource:
- !Sub 'arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:cwsyn-${CloudWatchSyntheticsCanary}-*'
- !Sub '${CanaryBucket.Arn}/*'
- !Sub 'arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/cwsyn-${CloudWatchSyntheticsCanary}-*'
- !Sub 'arn:aws:cloudwatch:${AWS::Region}:${AWS::AccountId}:alarm:Synthetics-Alarm-${CloudWatchSyntheticsCanary}-*'
- Effect: Allow
Action:
- 's3:ListBucket'
Resource:
- !GetAtt
- CanaryBucket
- Arn
- Effect: Allow
Action:
- 'synthetics:Get*'
- 'cloudwatch:DescribeAlarms'
Resource:
- !Sub 'arn:aws:synthetics:${AWS::Region}:${AWS::AccountId}:canary:${CloudWatchSyntheticsCanary}'
- !Sub 'arn:aws:cloudwatch:${AWS::Region}:${AWS::AccountId}:alarm:*'
Create the cleaner Lambda function
The CloudFormation custom resource sends two events to the Lambda function:
On Create, the Lambda function will receive the following event:
{
"RequestType": "Create",
"ServiceToken": "arn:aws:lambda:[[Region id]]:[[Account ID]]:function:[[custom Lambda Name]]",
"ResponseURL": "https://cloudformation-custom-resource-response-useast2.s3.[[Region id]].amazonaws.com/arn%3Aaws%3Acloudformation%3A[[Region id]]%3A[[Account ID]]%3Astack/Test123412/f4abac20-382c-11eb-b555-0296ed261c1a%7CwaitStackOnCreation%7C078a0d24-f0bf-4181-a831-0369b5fa2d58?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20201207T014015Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=AKIAVRFIPK6PLHKRTTNP%2F20201207%2F[[Region id]]%2Fs3%2Faws4_request&X-Amz-Signature=681b6b9c35e530d03348d965ef0d397082abac4e34515ac5fa840e8fce1e9f03",
"StackId": "arn: aws: cloudformation: [[Region id]]: [[Account ID]]: stack/Test123412/f4abac20-382c-11eb-b555-0296ed261c1a",
"RequestId": "078a0d24-f0bf-4181-a831-0369b5fa2d58",
"LogicalResourceId": "CustomResourceLogicalID",
"ResourceType": "Custom::cleanCanaryStack",
"ResourceProperties": {
"ServiceToken": "arn:aws:lambda:[[Region id]]:[[Account ID]]:function:[[custom Lambda Name]]",
"BucketName": "[Bucket Resource Name]",
"CanaryName": "[Canary Resource Name]"
}
}
On Delete, the Lambda function will receive the following event:
{
"RequestType": "Delete",
"ServiceToken": "arn:aws:lambda:[[Region id]]:[[Account ID]]:function:[[custom Lambda Name]]",
"ResponseURL": "https://cloudformation-custom-resource-response-useast2.s3.[[Region id]].amazonaws.com/arn%3Aaws%3Acloudformation%3A[[Region id]]%3A[[Account ID]]%3Astack/Lab2/dc67bf00-385e-11eb-8657-065e0005c80e%7CwaitStackOnCreation%7C8531a79f-0d4b-48fa-b851-2bfeccd1dbc0?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20201207T115755Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=AKIAVRFIPK6PLHKRTTNP%2F20201207%2F[[Region id]]%2Fs3%2Faws4_request&X-Amz-Signature=f8f8313da98b411d3d739d512a9844930f5c50c610cfce43bc10f2caf89ee76f",
"StackId": "arn:aws:cloudformation:[[Region id]]:[[Account ID]]:stack/Lab2/dc67bf00-385e-11eb-8657-065e0005c80e",
"RequestId": "8531a79f-0d4b-48fa-b851-2bfeccd1dbc0",
"LogicalResourceId": "CustomResourceLogicalID",
"PhysicalResourceId": "2020/12/07/[$LATEST]d9343e2ce5bc443e86ff6695befa87dd",
"ResourceType": "Custom::cleanCanaryStack",
"ResourceProperties": {
"ServiceToken": "arn:aws:lambda:[[Region id]]:[[Account ID]]:function:[[custom Lambda Name]]",
"BucketName": "[Bucket Resource Name]",
"CanaryName": "[Canary Resource Name]"
}
}
I used inline Python code with the cfn-response Python module to signal responses back to CloudFormation template based on the invocation result. If the invocation event is create the Lambda code will always return a responseStatus of SUCCESS. If the invocation event is Delete, it parse the ResourceProperties field in received JSON event to get the canary resource name and S3 artifact bucket name, then using the AWS SDK for Python (Boto3), the code does the following:
- Retrieves canary details to locate the Lambda function.
- Deletes the Lambda function.
- Deletes the CloudWatch log group for the canary Lambda function.
- Searches for and deletes alarms that have the prefix of
Synthetics-Alarm-MyCanaryName. - Deletes all objects inside the S3 artifact bucket.
In the case of any exception, Lambda will return a responseStatus of FAILED to the CloudFormation stack along with information about which resource returned the error.
cleanupCanaryStackOnDelete:
Type: 'Custom::cleanCanaryStack'
Properties:
ServiceToken: !GetAtt CleanCanaryStackLambda.Arn
BucketName: !Ref CanaryBucket
CanaryName: !Ref CloudWatchSyntheticsCanary
CleanCanaryStackLambda:
Type: 'AWS::Lambda::Function'
Properties:
Code:
ZipFile: |
import json
import boto3
import cfnresponse
#Initialize boto3 clients for synthetics,lambda, CloudWatch logs,cloudwatch and s3
synthetics = boto3.client('synthetics')
lambdaClient = boto3.client('lambda')
logs = boto3.client('logs')
cloudwatch = boto3.client('cloudwatch')
s3 = boto3.resource('s3')
def lambda_handler(event, context):
try:
bucketName = event['ResourceProperties']['BucketName']
CanaryName = event['ResourceProperties']['CanaryName']
Errors=[]
if event['RequestType'] == 'Delete':
try:
# Find Lambada name corresponding to your Canary name
Canarydetails = synthetics.get_canary(Name=CanaryName)
CanaryLambdaName = Canarydetails['Canary']['EngineArn'].split(':')[-2]
CanaryArtifactBucket = Canarydetails['Canary']['ArtifactS3Location']
except Exception as e:
print('Get-Canary failed: '+str(e))
Errors.append('Get-Canary failed: '+str(e))
pass
try:
#delete Canary Lambda
lambdaClient.delete_function(FunctionName=CanaryLambdaName)
except Exception as e:
print('Deleting Canary Lambda failed: '+str(e))
Errors.append('Deleting Canary Lambda failed: '+str(e))
pass
try:
#delete Canary log group
CanaryLogGroup='/aws/lambda/'+CanaryLambdaName
logs.delete_log_group(logGroupName=CanaryLogGroup)
except Exception as e:
print('Deleting Canary logs failed: '+str(e))
Errors.append('Deleting Canary logs failed: '+str(e))
pass
# Delete Alarms
try:
# retrun all alarms metricAlarm and Composite Alarms with prefix 'Synthetics-Alarm-CanaryName'
Alarms = cloudwatch.describe_alarms(AlarmNamePrefix='Synthetics-Alarm-'+CanaryName,AlarmTypes=['MetricAlarm'])
# Create list of all metricAlarms as deleteAlarms API can delete up to 100 alarm in one API
metricAlarms=[]
for alarm in Alarms['MetricAlarms']:
metricAlarms.append(alarm['AlarmName'])
if metricAlarms:
cloudwatch.delete_alarms(AlarmNames=metricAlarms)
except Exception as e:
print('Deleting Canary Alarm failed: '+str(e))
Errors.append('Deleting Canary Alarm failed: '+str(e))
#Find Bucket and delete canary objects under prefix if exist
try:
bucket = s3.Bucket(bucketName)
#Check if there is prefix in the Canary artifact bucket
if (CanaryArtifactBucket.find(bucketName+'/') != -1):
#search for objects under prefix
objects = bucket.objects.filter(Prefix=CanaryArtifactBucket.split(bucketName+'/')[1])
else:
#If no prefix, search for all objects under the bucket
objects = bucket.objects.filter()
for obj in objects:
# delete objects with canary prefix
s3.Object(bucket.name, obj.key).delete()
except Exception as e:
print('Deleting S3 Objects failed: '+str(e))
Errors.append('Deleting S3 Objects failed: '+str(e))
pass
#IF any of dependency resources was not deleted raise exception to send custom error message to stack
if Errors:
raise Exception(Errors)
cfnresponse.send(event, context, cfnresponse.SUCCESS,json.loads("{}"),None,False,'Reason: Stack cleaned successfully')
except Exception as e:
print(e)
cfnresponse.send(event, context, cfnresponse.FAILED,json.loads("{}"),None,False,'Reason: ' + str(e))
Handler: index.lambda_handler
FunctionName: !Sub 'CleanCanaryResources-${AWS::StackName}'
Role: !GetAtt CleanerLambdaExecutionRole.Arn
Runtime: python3.8
Timeout: 900
Create a custom resource
Give the custom resource a name like Custom::cleanCanaryStack. Under Properties, in the ServiceToken property, we point toe the Lambda function ARN. We define resources to pass to the Lambda function in the invocation event, such as the S3 artifact bucket resource name and the canary resource name.
cleanupCanaryStackOnDelete:
Type: 'Custom::cleanCanaryStack'
Properties:
ServiceToken: !GetAtt CleanCanaryStackLambda.Arn
BucketName: !Ref CanaryBucket
CanaryName: !Ref CloudWatchSyntheticsCanary
Create a CloudFormation stack
- Open the CloudFormation console and choose Create Stack.
- Choose with new resources (standard).
- Choose Upload a template file, and then choose your CloudFormation template file.
- Enter a name for your stack, complete the Parameters section, and then choose Next.
- Keep the defaults on the Configure stack options page, and then choose Next.
- On the Review page, select the checkbox to acknowledge that AWS CloudFormation might create IAM resources, and then choose Create Stack.
Figure 2 shows the created resources:
Figure 2: Created resources
Test stack deletion result
On stack deletion, custom resource invokes the lambda function that will delete all canary dependant resources and empty the S3 artifacts bucket. If Lambda failed to empty the S3 artifacts bucket the stack deletion would have failed to delete the S3 artifacts bucket.
To delete a stack
- Open the AWS CloudFormation console.
- On the Stacks page in the CloudFormation console, select the stack that you want to delete. The stack must be currently running.
- In the stack details pane, choose Delete.
- Select Delete stack when prompted.
Figure 3: Deleted resources events
Cleanup
To avoid charges to your account, delete the CloudFormation stack and resources. For more information, see Deleting a stack on the AWS CloudFormation console in the AWS CloudFormation User Guide.
Conclusion
In this blog post I showed you how to use a Lambda-backed custom resource to delete Amazon CloudWatch Synthetics dependent resources, such as AWS Lambda functions, CloudWatch logs, CloudWatch alarms, and S3 artifact buckets and objects.
Although we used the Lambda-backed custom resource on stack deletion, Lambda functions in combination with AWS CloudFormation enable a range of scenarios, such as dynamically looking up AMI IDs during stack creation or implementing and using utility functions, such as string reversal functions. You can also write your own resources to extend AWS CloudFormation beyond AWS resources and provision any other resource you can think of. For example, you can integrate a third-party software as a service (SaaS) product or you can even provision on-premises resources in hybrid environments.