Integrate AWS Cost Anomaly Detection Notifications with IT Service Management Workflow – Part 2 ServiceNow

In part one of this blog series, we shared instructions on how you can integrate AWS Cost Anomaly Detection (CAD) notification with Atlassian Jira Service Management (read blog). In part two of the blog series, we will shed light on the integration of CAD with ServiceNow IT Service Management.

ServiceNow IT Service Management offers a widely used, cloud-based ITSM platform designed to improve IT services’ flexibility and agility. You can consolidate your legacy on-premise systems and IT tools into a single data model to transform the IT services experience, automate workflows, gain near real-time visibility, and improve IT productivity.

By leveraging automation, you can invoke advanced yet flexible, high-velocity ITSM workflows in ServiceNow ITSM to create incident records. You can further customize these workflows to add context, assign ownership and drive issues to timely resolution. This also benefits you by eliminating any potential scope for human errors and reducing administrative overhead. You remain in sync with your organization’s defined mechanisms to document and track the end-to-end flow.

AWS Cost Anomaly Detection uses advanced Machine Learning to identify anomalous spends and generate timely alerts, empowering the customers to act swiftly. Organizations following the ITSM framework need duly assigned incident tickets to assign ownership, and record, track and manage the efforts to address the anomaly.

This post explains how you can integrate the AWS Cost Anomaly notifications with ServiceNow ITSM’s built-in abilities to automate incident management workflow.

Architecture Overview

The following architecture diagram illustrates the use of AWS Lambda to retrieve the cost anomaly alerts and trigger a REST API endpoint using ServiceNow Scripted REST API in ServiceNow ITSM. When a cost anomaly is detected by AWS, the cost monitor generates an alert notification and publishes it to an Amazon Simple Notification Service (SNS) Topic. The SNS Topic has a subscription for Lambda function which retrieves the message and in turn, triggers a Scripted REST API endpoint in ServiceNow, also supplying the notification message. Optionally, you can create an SNS Topic subscription to send email/SMS notifications to cloud admins. The REST API triggers the automation workflow to create a new incident management record in ServiceNow ITSM, parses the JSON notification message and populates the defined fields therein.

Figure 1. architecture diagram overview

Prerequisites

1. AWS Cost Anomaly Detection is a feature within AWS Billing and Cost Management Console. To access AWS Cost Anomaly Detection, enable AWS Cost Explorer. For instructions on how to enable AWS Cost Explorer using the console, see Enabling AWS Cost Explorer.
2. Configure AWS Cost Anomaly Detection so that it detects anomalies at a lower granularity and spend patterns, in context to your monitor type. Creating your cost monitors
3. Create an AWS Simple Notification Service (SNS) Topic with the required permissions for the cost anomaly to publish the alert notifications to. Creating an Amazon SNS topic for anomaly notifications
4. Signup and setup the ServiceNow developer program instance. ServiceNow Developer getting started guide

Implementation Steps

Step 1: Configure the Scripted REST API in ServiceNow ITSM

1. Launch your custom ServiceNow developer instance URL (https://devxxxxx.service-now.com/) and login with the admin user role
2. In the search box on the left-hand side, search for “Scripted REST APIs”. Under System Web Services choose Scripted REST APIs
3. Choose New on the right top corner of the page to create a new REST API
4. Provide a Name and API ID of your choice for the new API and choose Submit
   

   Figure 2. Sample screenshot of creating new REST API in ServiceNow

5. Back on the Scripted REST APIs page, search and choose the API that you created in the previous step
6. Under the Resources tab, choose New
7. Provide a Name for this new Scripted REST Resource and under HTTP method choose POST from the dropdown list
   

   Figure 3. Sample screenshot of REST API endpoint URL in ServiceNow

8. On the same page, under the section for Script, use this JavaScript code to parse the notification data from the AWS Lambda function and trigger the incident management workflow
9. Under the Security tab, clear the checkbox that says Requires authentication and choose Submit. Refer Scripted REST API access controls and CORS rules to implement additional access control mechanisms
10. From the Scripted REST API page, notice the Base API path for your newly created REST API service
11. Go to the REST API Explorer page and choose your API Namespace
12. You can find the REST API endpoint URL against the name of the scripted REST resource (In this example: https://devxxxxxx.service-now.com/api/1230129/aws_cost_anomaly_incidents)
    

    Figure 4. Sample screenshot of REST API URL in ServiceNow

Step 2: Launch the AWS CloudFormation Template

Download and launch this AWS CloudFormation template to deploy an AWS Lambda function, Amazon CloudWatch Logs group, and add Lambda subscription to the SNS Topic. We can use this CloudFormation template for integration with Jira Service Management as well which is explained in part one of this blogpost.

Note: Anomaly notification published to the SNS Topic, followed by the Lambda function trigger can incur costs. To learn more about pricing refer: AWS Lambda pricing; Amazon SNS pricing; AWS Pricing Calculator

1. Sign in to the AWS Management Console and navigate to the AWS CloudFormation console
2. Choose Create Stack > “With new resources” and specify a “Stack name”
3. Under “Parameters” provide the REST API Endpoint URL you had created in Step 1, the SnsTopicArn for your SNS Topic (created as part of pre-requisites) and choose Next
   

   Figure 5. Sample screenshot of adding stack details

4. Leave the “Configure stack options” at default values, add tags as needed and choose Next
5. Review the details on the final screen and under “Capabilities” check the box for “I acknowledge that AWS CloudFormation might create IAM resources with custom names”
6. Choose Submit

Note: You can review the progress of your new stack under AWS CloudFormation > Stacks > [StackName] > Events tab

Once the Stack is created successfully, you will see the following resources deployed:

AWS Lambda function, Amazon CloudWatch Log group, SNS Subscription for the Lambda function under your SNS Topic and the corresponding AWS IAM Roles and Policies.

Step 3: Test your Scripted REST API code using a sample JSON

1. Sign in to the Amazon SNS console
2. In the left navigation pane, choose Topics
3. Choose the SNS topic you had created as the pre-requisite, and then choose Publish message
4. In the Message body section, copy this test JSON and choose Publish message
5. On successfully publishing the message to the SNS topic, the Lambda function triggers the REST API endpoint URL in ServiceNow developer instance, which executes the JavaScript code to initiate the workflow Note: You can verify the Lambda execution using CloudWatch metrics under the “Monitor” tab
6. The code parses the JSON format notification data, retrieves the values for the defined variables and creates the new incident management record in ServiceNow Service Management
   

   Figure 6. Sample screenshot of incident management record

7. When the actual cost anomaly alert notification is generated, the REST API script creates the new incident record with the values and details from the cost anomaly notification fields
   

   Figure 7. sample screenshot of incident record details

Clean up

You will continue to incur cost until you clean up the infrastructure that you created for this post:

1. Delete AWS CloudFormation Stack
2. Delete Amazon SNS Topic
3. Delete the Scripted REST APIs you created for this demo
4. Delete the incident records that were created

Conclusion

In this part of the blog post, we discussed how you can integrate the AWS Cost Anomaly notifications with ITSM platform provided by ServiceNow ITSM to automate the creation of incident records with the help of Scripted REST APIs in ServiceNow. This empowers you with an efficient mechanism to automate the end-to-end flow of work to identify, document and track issues related to anomalous spends in your AWS account and make swift decisions. To get started and learn more, visit Getting started with AWS Cost Anomaly Detection and ServiceNow Developer getting started guide.

As an option, you can also add subscriptions for other protocols like Email, SMS, HTTPS etc. to your SNS Topic to get notifications and/or trigger additional workflows. For example, you can integrate the solution to visualize the cost anomaly data using Amazon Managed Grafana. Alternatively, in order to have an organizational view for the cost anomaly data in Amazon QuickSight, you can deploy the Cost Anomaly Dashboard.

Refer part one of this blog post to know about integrating AWS Cost Anomaly notifications with Jira service management to trigger automation workflows using incoming webhook URLs.