When an AWS Data Pipeline activity fails, I want to receive an Amazon Simple Notification Service (Amazon SNS) notification that includes specific information, such as the error stack trace.

Create and subscribe to an SNS topic, and then configure an SnsAlarm action to run when a Data Pipeline activity fails. You can pass any of the activity's fields, such as errorStackTrace, to SnsAlarm using the expression #{node.activityfield}. For example:

{
  "role": "DataPipelineDefaultRole",
  "subject": "Failed PipelineId: #{node.@pipelineId} Error: #{node.errorMessage}",
  "message": "Error: #{node.errorStackTrace}",
  "topicArn": "arn:aws:sns:us-east-1:917786601234:DPsqlActivityFail",
  "name": "DefaultActionSNS1",
  "id": "SnsAlarmActionOnfail",
  "type": "SnsAlarm"
},
  1. Create and subscribe to an SNS topic. Note the Topic ARN (for example, arn:aws:sns:us-east-1:111122223333:my-topic). You'll use this later.
  2. Open the Data Pipeline console.
  3. On the List Pipelines page, choose your Pipeline ID, and then choose Edit Pipeline to open the Architect page.
  4. In the right pane, choose Activities, and then select the activity that you want to receive notifications for (for example, SqlActivity).
  5. In the Add an optional field drop-down list, choose On Fail.
  6. In the On Fail list, choose Create new: Action. Data Pipeline automatically gives the new action a name, such as DefaultAction1. If you want, you can change the name by editing the action in the Others section.
  7. In the right pane, choose Others, and then choose the action that you created in the previous step (for example, DefaultAction1).
  8. In the Type list, choose SnsAlarm.
  9. In the Message field, enter the message that you want to receive when the activity fails. You can include any of the activity's fields in the message using the expression #{node.activityfield}. For example, to receive the stack trace in the SNS notification, enter a message such as "Error: #{node.errorStackTrace}". For more information about the fields that are available for each Data Pipeline activity, see Activities.
  10. In the Subject field, enter a subject line for the notification. The subject can also include any of the activity's fields (for example, "Failed PipelineId: #{node.@pipelineId} Error: #{node.errorMessage}").
  11. In the Topic Arn field, enter the ARN of your SNS topic (for example, arn:aws:sns:us-east-1:111122223333:my-topic).
  12. Choose Save. When you activate your pipeline, you'll receive a notification if the specified activity fails.

Did this page help you? Yes | No

Back to the AWS Support Knowledge Center

Need help? Visit the AWS Support Center

Published: 2019-02-12