How do I troubleshoot an AWS Lambda-backed custom resource that's stuck in DELETE_FAILED status or DELETE_IN_PROGRESS status with AWS CloudFormation?

Last updated: 2019-05-07

My custom resource is stuck in the DELETE_FAILED status or the DELETE_IN_PROGRESS status. How can I delete my custom resource?

Resolution

Delete a custom resource that's stuck in the DELETE_FAILED status

If your AWS Lambda-backed custom resource is missing the logic to handle Delete requests and you try to delete a stack, you'll get the "Custom Resource failed to stabilize in expected time" error message. Also, the status of the stack changes to DELETE_FAILED. To delete your stack, complete the following steps:

1.    Open the AWS CloudFormation console.

2.    Choose the stack that contains your custom resource that's stuck in the DELETE_FAILED status.

3.    Choose Actions, and then choose Delete Stack.

4.    In the pop-up window that provides a list of resources to retain, choose the custom resource that's stuck in the DELETE_FAILED status, and then choose Delete.

5.    Choose Actions, and then choose Delete Stack.

The status of your stack changes to DELETE_COMPLETE.

Note: Your custom resource isn't a physical resource, so you don't have to clean up your custom resource after stack deletion.

Delete a custom resource that's stuck in the DELETE_IN_PROGRESS status

If you can't delete a stack that's in DELETE_IN_PROGRESS status, your stack might be missing the logic to handle Delete requests. To add the correct logic, you must manually send a SUCCESS signal to AWS CloudFormation by completing the following steps:

1.    In your AWS CloudFormation template, identify the name of the Lambda function that your custom resource is sending requests to. You can find the function name from the ServiceToken property of the AWS::CloudFormation::CustomResource or Custom::String resource. See the following example:

MyCustomResource: 
  Type: "Custom::PingTester"
  Properties: 
    ServiceToken:
      !Sub |
        arn:aws:lambda:us-east-1:111122223333:function:awsexamplelambdafunction

2.    Open the Lambda console.

3.    In the navigation pane, choose Functions, and then choose the function that you identified in step 1. For example, awsexamplelambdafunction is the function name in the preceding code sample.

4.    Choose the Monitoring tab, and then choose View logs in CloudWatch.

5.    In the Amazon CloudWatch console, choose the latest log.

Note: You can see the CloudWatch logs only if your function has access to Amazon CloudWatch Logs for log streaming.

6.    In the latest log, identify the event where RequestType is Delete, and then copy the values for RequestId, ResponseURL, StackId, LogicalResourceId, and PhysicalResourceId. See the following example:

Received event: {
  "RequestType": "Delete",
  "ServiceToken": "arn:aws:lambda:us-east-1:111122223333:function:awsexamplelambdafunction",
  "ResponseURL": "https://cloudformation-custom-resource-response-useast1.s3.us-east-1.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-east-1%3A111122223333%3Astack/awsexamplecloudformation/33ad60e0-5f25-11e9-a734-0aa6b80efab2%7CMyCustomResource%7Ce2fc8f5c-0391-4a65-a645-7c695646739?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20170313T0212304Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=QWERTYUIOLASDFGBHNZCV%2F20190415%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=dgvg36bh23mk44nj454bjb54689bg43r8v011uerehiubrjrug5689ghg94hb",
  "StackId": "arn:aws:cloudformation:us-east-1:111122223333:stack/awsexamplecloudformation/33ad60e0-5f25-11e9-a734-0aa6b80efab2",
  "RequestId": "e2fc8f5c-0391-4a65-a645-7c695646739",
  "LogicalResourceId": "MyCustomResource",
  "PhysicalResourceId": "test-MyCustomResource-1URTEVUHSKSKDFF",
  "ResourceType": "Custom::PingTester"

7.    To send a SUCCESS response signal to the Delete request, run a curl command with the values you copied from step 6. See the following example:

$ curl -H 'Content-Type: ''' -X PUT -d '{
    "Status": "SUCCESS",
    "PhysicalResourceId": "test-CloudWatchtrigger-1URTEVUHSKSKDFF",
    "StackId": "arn:aws:cloudformation:us-east-1:111122223333:stack/awsexamplecloudformation/33ad60e0-5f25-11e9-a734-0aa6b80efab2
  ",
    "RequestId": "e2fc8f5c-0391-4a65-a645-7c695646739",
    "LogicalResourceId": "CloudWatchtrigger"
  }' 'https://cloudformation-custom-resource-response-useast1.s3.us-east-1.amazonaws.com/arn%3Aaws%3Acloudformation%3Aus-east-1%3A111122223333%3Astack/awsexamplecloudformation/33ad60e0-5f25-11e9-a734-0aa6b80efab2%7CMyCustomResource%7Ce2fc8f5c-0391-4a65-a645-7c695646739?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Date=20170313T0212304Z&X-Amz-SignedHeaders=host&X-Amz-Expires=7200&X-Amz-Credential=QWERTYUIOLASDFGBHNZCV%2F20190415%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=dgvg36bh23mk44nj454bjb54689bg43r8v011uerehiubrjrug5689ghg94hb
  '

The status of your custom resources changes to DELETE_COMPLETE.