Preview: Anomaly Detection and alerting now available in AWS Cost Management

The democratization of cloud technology brings autonomy and agility to end users, who can access and spin up resources quickly.  Builders get to experiment ideas, develop applications, and deploy products globally to meet local customers’ needs.  Organization leaders want to unleash the team’s creativity and accelerate the time-to-market, while keeping the cloud cost within limits. If cost exceeds a reasonable range, leaders want to be informed at the earliest possible time and understand what caused the cost overrun.  Whether there are inappropriate or inefficient usage of cloud resources, or certain products pick up momentum in the market and consume more technology resources as a result of higher traffic and usage, leaders want to be intentional with their investment and reduce the number of surprise bills they receive.  To help better control your costs and save time in investigating anomalous spend, we are pleased to announce the launch of AWS Cost Anomaly Detection (Preview). In this blog post, our Senior Product Manager, Juan Cabrera, will walk you through how you can enable this function.

Starting today, customers can use AWS Cost Anomaly Detection to detect unexpected or unusual spend.  AWS Cost Anomaly Detection uses a multi-layered state machine learning model that learns your unique spend patterns to adjust spend thresholds – this means you do not need to worry about determining appropriate thresholds (e.g. 10% increase in daily spend) and maintaining them as your usage changes over time. These machine learning layers allow Anomaly Detection to detect various types of anomalies, such as a one-time cost spike or gradual, consistent cost increases. Anomaly Detection is customizable to segment your spend data and provides tailored alerting preferences so that you are informed as soon as an anomaly is detected.

Getting started with AWS Cost Anomaly Detection

Anomaly Detection is powered by a complex machine learning model yet it is simple and easy to use in AWS Console.  It takes two steps with a few clicks to enable Anomaly Detection.  Log in your AWS Console and navigate to AWS Cost Management Dashboard, and you will find “Anomaly Detection” on your left pane.

Step 1: Create a monitor

Customers let us know that they need to segment spend to not only manage budgets, but to also evaluate cost performance over time. Anomaly Detection allows you to segment spend so that anomalies can be detected at a desired granularity with contextualized spend patterns. For example: your spend patterns for EC2 usage may differ significantly from the ones for Lambda or S3 – in this case, segmenting spend by AWS Service allows Anomaly Detection to detect separate spend patterns that help decrease false positive alerts. As of this release, there are 4 different Monitor types that are supported:

• AWS Services – We recommend this monitor for customers that do not need to segment spend by internal organization or environment. This single monitor evaluates AWS services individually for anomalies. As you adopt new AWS services, this monitor will automatically start to evaluate that service for anomalies, without you having to change your configuration.
• Linked/Member Account – This monitor evaluates the total spend of an individual (or group of) linked accounts. This monitor is helpful if your organization needs to segment spend by team (or products, services, environments) which you define as individual or groups of accounts.
• Cost Allocation Tag – Similar to the Linked Accounts monitor type, we offer this monitor should you need to segment spend by teams (or products, services, environments) as defined by Cost Allocation Tags. This monitor type restricts to one tag key, but accepts multiple tag values.  For instance, you may have created Cost Allocation Tags with the Tag key of “Cost Center” and various Tag values such as “Cost Center:engineering, Cost Center:Engineering, and Cost Center:Eng”. Once you select the Tag key “Cost Center” as your monitor, you can select all of these different values to be evaluated together as a single monitor.
• Cost Categories – Since the launch of Cost Categories, many customers have started using this service to logically create custom groups that can allow them to better organize and manage spend according to their organizational structure. For customers who use Cost Categories, you can select Cost Categories as your monitor type for anomaly detection. This monitor type restricts to one Cost Category value.  For instance, you may have created a Cost Category named “Team”, under which you created three values: “Team:Alpha”, “Team:Beta”, and “Team:Gamma”, each with a group of AWS accounts.  You will need to specify which value/team you want to monitor and evaluate for spend anomalies.

Step 2: Configure alerting preferences

As you create your monitors, you will be able to configure your alerting preference. Note that you do not have to define an anomaly (e.g. 10% increase in daily spend) – we leverage machine learning to automatically determine and continuously adjust thresholds as your spend patterns change over time. What you need to define is your alerting preferences so that you only receive alerts that are important to you based on the estimated cost impact. Anomalies that do not meet these thresholds are captured and available to you in the Anomaly Detection dashboard.

Additionally, you can determine the alerting frequency. Individual alerts allow you to be notified as soon as an anomaly is detected (note: this is not alerted immediately after the usage occurs) – this means that if multiple anomalies are detected throughout a day, you might receive multiple alerts. Another option is to receive Daily or Weekly summary reports, which consolidate the alerts into a single notification, either daily or weekly.

Lastly, Anomaly Detection supports SNS and email notifications.

You’ll notice that you can configure these preferences within each individual Monitor you have created. This will allow you to notify the appropriate stakeholders within your organization.

• Example: let’s say your organization has two linked/member accounts -1234 and -4321. You can create a monitor that evaluates account -1234 and configure notifications to alert John Doe and Jane Doe. You can also create a separate monitor that evaluates account -4321 and configure notifications to alert John Doe and Jim Doe. In this example, Jane Doe and Jim Doe are only notified if there is an anomaly in the account that they are associated with, but John Doe gets notified of any anomaly across both accounts.

Step 3: Done!

Now that you have configured your monitor(s) and alerting preferences, Anomaly Detection will start to notify you of any future anomalous spend that meets your alerting threshold. Should you find that you’re being notified too often, you can always increase the alerting threshold so that your attention is only captured by the top anomalies – same can be done in the inverse if you’d like to be notified of more anomalies. If you adjust your preferences to match your needs, you do not need to worry that anomalies may go undetected – you can always come back to the Anomaly Detection dashboard to review all anomalies detected, even those that did not meet your alerting threshold.

• Tips: When creating monitors across multiple Monitor Types that overlap in spend, you may receive multiple alerts on the same anomaly.

Using Anomaly Detection

As you begin to see spend anomalies in the Anomaly Detection dashboard, you can dive deeper into each anomaly by clicking on each detection date. You’ll immediately notice 3 things:

Root cause analysis: Regardless of how you configured your monitors, Anomaly Detection will attempt to pin point the root cause of the anomaly by indicating the Service, Account ID, Region, and Usage type that is driving that unusual spend increase. There may be multiple root causes occurring at once – in these cases, Anomaly Detection will provide the most prevalent two root causes.

View in Cost Explorer: Anomaly Detection comes integrated with Cost Explorer so you can graphically visualize the anomaly and perform further analysis, as needed. You’ll notice that the filters and dates are automatically configured based on the anomaly details.

Submit assessment: As you evaluate your anomalies, you have the ability to submit feedback on each anomaly. This not only helps you keep track of which anomalies have you already evaluated, but also helps Anomaly Detection improve to be more tailored to your assessments and preferences.

Conclusion

AWS Cost Anomaly Detection provides you with an easy-of-use, ML-driven capability to detect unusual spend across your AWS accounts. You can configure Anomaly Detection to better reflect your organization’s business needs and/or how you manage your cost and usage. Anomaly Detection provides flexible alerting capability so that the correct stakeholders are notified of each anomaly that is detected, using their own alerting preferences. With the introduction of Anomaly Detection, AWS Cost Management is providing you yet another way to more effectively control your cost and usage while minimizing unintended spend.