Cost Allocation Blog Series #5: Building an AWS cost allocation strategy with Apptio Cloudability

This blog post is contributed by Scott Kellish, Partner Solutions Architect at AWS, and Andrew Midgley, Senior Technical Product Marketing Manager at Apptio

As you establish and implement your cost allocation model, AWS equips you with tools and resources to organize accounts and resources (e.g. AWS Organizations), improve visibility into detailed cost and usage data (e.g. AWS Cost and Usage Reports), and group cost and usage information into meaningful groups by your own rule (e.g. AWS Cost Categories).  For customers who want to understand what premium 3rd party solutions are available to help you better organize and report your cost in preparation for chargeback or showback in your organizations, AWS Partner Network has verified partners that can offer cloud management solutions to help customers manage and optimize AWS environments.   In this blog post, Scott Kellish, Partner Solutions Architect at AWS, and Andrew Midgley, Senior Technical Product Marketing Manager at Apptio, will share how you can leverage Apptio Cloudability to build your cost allocation strategy.

The impact of a cost allocation strategy on the success of cloud initiatives sometimes can be surprising to customers who come from a traditional IT background. When managing hardware in a data center, customers acquire infrastructure negotiated and approved by procurement resulting in fixed upfront payments. Chargeback of these data center costs to the business units is rarely based on consumption. Instead, either the IT group bares the costs or the business units pay using a ‘peanut-butter spread’ allocation model that often feels like a tax. To delivery teams, it often felt like this infrastructure was free. Moving to the cloud has changed the way developers consume IT. Teams no longer need to consider fixed upfront costs of infrastructure, but only consumption costs which are variable in nature.  Freed from the necessity to involve procurement, developers can make decisions in an agile way and deploy via Infrastructure as Code.

Figure 1: Shift from traditional upfront/gated procurement to variable/accessible procurement in the cloud

This shift from an upfront/gated model of procuring IT infrastructure to a variable/accessible model can be transformational for many businesses, but it also brings risks. With a disciplined approach, you can tightly assign cloud costs to business groups or initiatives based on consumption and be clear on the business value delivered. However, without such discipline, these infrastructure costs may exceed planned budgets, and waste can continue to increase as leaders struggle to make teams accountable.

The primary goal of investing in a cost allocation strategy is to assign ownership to cloud costs reliably. Before diving into specifics, it’s worth reviewing the shape that AWS billing data takes. To offer customers complete visibility into your cloud consumption, AWS provides a detailed billing file – called AWS Cost and Usage Report, or CUR – where every resource for every hour appears as an individual row. This can be something well known like the cost of an Amazon Elastic Compute Cloud (Amazon EC2) instance or something more obscure like the cost of Amazon CloudWatch custom metrics. Due to the granularity and scale of cloud, it’s common that these billing files will extend to tens or hundreds of millions of rows. Each of these rows can have as many as 120 or more attribute columns, and with a strong cost allocation strategy in place, you can use them to allocate your costs accurately and do much more with this data. Delivering a cost allocation strategy is generally a three-tier activity spanning Foundations, Business Reporting, and Cost Sharing.

Foundations

It’s a good idea to think about the various ways you need to group costs. Do you need to group based on application, project, team, business unit or even classify the types of spend as cost of goods sold (COGS) versus operating expenses? All of these will be possible with the right foundations that typically have one or two key attributes: accounts and resource tags.  You can read more about the importance of these two attributes and what AWS resources you can use to strategize your account structure and tagging usage from our previous cost allocation blogs series.

TIP: Focus on your “top level allocation” – make sure you have strong protocols in place so that at a minimum every resource can be assigned ownership at the top business division. If you can establish this ownership, you will be able to work with teams on further classifications and have a path to chargeback or showback. It’s also worth noting that resource tags are not retroactively applied within billing data so it’s important to establish your tagging strategy as early as possible.

How can Cloudability help?

There are a number of ways Cloudability can help you successfully allocate costs based on tagging. The product offers a tag mapping capability that greatly eases the pain related to variation in tag keys. For example, tags with different casings (env vs Env) or different names (Env vs Environment) will appear as separate columns in AWS’ billing file. Cloudability will seamlessly combine these into a normalized reporting dimension that is usable across the platform. Cloudability also provides an interactive tool called Tag Explorer that quickly lets customers understand their overall tagging coverage and helps setup a path for remediation.

Figure 2: Cloudability Tag Explorer

Users can click into any untagged buckets of cost to get a complete report with all the required information to locate each untagged resource. After remediating individual problems, it’s also worth considering adding a Service Control Policy or Tagging Policy within AWS Organizations to help prevent future compliance issues.

Business Reporting

Once you have invested in your foundations, preferably institutionalized the agreed-upon patterns through build pipelines, you will be ready for your next stage of allocation. The next task is to apply business rules to these foundations, such that you establish official reporting dimensions that are less aligned to your underlying infrastructure but more aligned to your business concepts.

Broadly speaking, there are two reasons you need to build this additional reporting layer.

Multiple Attributes Required

There are many use cases where it’s not enough to simply rely on one attribute such as a resource tag or account group. Most chargeback/showback reporting dimensions will depend on multiple attributes. Here are two common examples:

·       Mixed Ownership Model – An organization can have some accounts that are shared across teams and some that are team owned. In this case, you need to establish an allocation rule that combines both tags and account groups.

·       Account Fall Back Model – Resource tags are the primary allocation mechanism. However, it is extremely rare to see every item tagged correctly, and so you place an account ownership mapping as a fallback. Again, your allocation rule is based around tags and account groups.

There are a number of other similar allocation models. Sometimes you’ll need to rely on other billing attributes such as region or resource ID. It’s also likely at times you’ll need to have bespoke mappings that intelligently switch allocation based on date boundaries or other relevant attributes.

Additional Layers

You have the opportunity to greatly enrich this foundational data through mapping completely new business layers on top.  Even though your base data may not explicitly have information such as the project ownership of a particular service or what type of expense an item is, you can create mapping rules to surface this detail. The most powerful examples we’ve seen recently involve connecting cloud financial management processes with enterprise IT Service Management (ITSM) workflows. This normally manifests itself with every cloud resource being tagged with a unique identifier for the service it backs. From there, as shown in the diagram below, you can map out the entire organization as it appears at any point in time.

Figure 3: Cloudability Business Mapping tool to recreate entity relationships without full tagging

Note: Every box and person above represent a reportable dimension. Even though cloud engineering only tagged resources with a ServiceID a capability such as Cloudability’s Business Mapping can build this entire picture.

How can Cloudability help?

Cloudability provides a purpose-built feature called Business Mapping which enables you to define the rules for each business grouping or classification upon which reports will be created.  Much of the power behind this feature is due to the flexible language and rich set of operators it provides. Business mapping can harness any of the foundational attributes you rely on and be used to traverse your entire org structure as discussed above. The resulting dimensions can be applied throughout Cloudability as a part of your cloud financial management processes.

Figure 4: Cloudability Business Mapping UI

This “Team Allocation” reporting dimension is being defined through the Business Mapping UI, where admins apply matching operators against multi-cloud billing attributes, and even previously defined Business Dimensions, resulting in all cost items being accurately categorized. Customers regularly automate these tasks through the public API, which for Business Mapping has a sophisticated DSL allowing you to represent your complicated Boolean Logic and making available advanced operators such as regular expression matchers.

Cost Sharing

By investing in your allocation foundations and enriching this information through business mapping, it’s possible to officially group your cloud spend for each of the business concepts that are important to you. As a part of this process, it’s common to encounter buckets of cost that you can’t naturally assign directly to one entity. These costs need to be split up and allocated before you can make teams accountable for them. A common example of this is the monolithic AWS support charges that appear once per month. A more complicated scenario, which is increasingly common and becoming more critical to solve, is how to share the costs associated with running containerized workloads on AWS. The problem is that containerization relies on shared pools of cloud resources, typically being hundreds to thousands of EC2 instances and Amazon Elastic Block Store (Amazon EBS) volumes. It is not possible for you to use the regular billing attributes to assign these costs out but instead you must dive into your container environments to understand who is responsible for driving these costs.

How Can Cloudability Help?

Apptio recently launched capabilities that make accurately allocating container costs straightforward and bring full cloud financial management practices to this spend. The first aspect of this is the automatic grouping of resource costs to each Kubernetes (k8s) cluster. Then sophisticated algorithms evaluate underlying resource utilization (CPU, memory, disk, network) and pod QoS configuration (burstable, guaranteed, best effort) to derive how each cluster’s costs can be split and allocated by k8s specific constructs like namespaces and labels. Finally, this detailed container cost allocation information is integrated into the core analytics platform so that it can be used alongside regular reporting dimensions and metrics to complete your allocation story. Naturally this information can be used with regular features like budgets & forecasts and dashboards to enable team accountability.

Figure 5: Cloudability Container Dashboard

This configurable dashboard provides an example of how AWS customers can analyze their container footprint in the context of their full cloud spend.

Allocation as the platform for driving financial accountability

Without the ability to reliably group your cloud costs in meaningful ways relevant to the business, your financial governance story can barely begin. However, there are other important aspects that you’ll also need to work on so that this story can be complete. The most crucial is to decide how you measure cost at a line by line level: for example, what cost do you want to associate with instance hours covered by Reserved Instances or Savings Plans (with consideration of upfront components) and how do you want to treat any custom pricing agreements that may be in place. Apptio calls this establishing an organizational True Cost metric and it’s necessary to build systems around it.

Success with these activities provides the platform to drive financial accountability across the organization. The outputs can be used within contextual dashboards so that teams have the visibility of your historical and current cloud spending. Official budgets can be overlaid and stakeholders pre-emptively notified when forecasted to overshoot. Rightsizing initiatives will greatly benefit from the shared understanding of what your deployed applications really cost you and the clear lines of ownership. Finally, by accurately allocating costs across the business it’s possible for each business group to align their cloud costs to relevant business metrics and therefore objectively measure the cloud economics. Understanding a 10% increase in cloud costs in the context of a 50% increase in monthly active users would be a critical insight that drives smart business decisions.

Conclusions

With AWS, you can set up account structure and apply meaningful metadata, to create an ownership model of resources usage and establish the  foundation for chargeback and showback purposes appropriate for your organization. You can run reports at your desired granular level and associate the cost and usage data with your business KPIs by integrating the reports with your own CRM/BI systems. With Cloudability, you can further improve your ownership model by making sense of tag variations and remediating for untagged resources.  You can also customize rules for different stakeholder groupings and the business reports they’ll receive. When it comes to allocating costs of containerized workloads, Cloudability can group and allocate costs per container cluster and visualize detailed cost information.