Crafting a robust metrics strategy to quantify your benefits from the cloud

Voiced by Amazon Polly

“Am I really saving?”

To be honest, it can be difficult to answer that question. During a migration, it can be a struggle to gauge IT impact on balance sheets and income statements, since the previously used variable-cost vs. fixed-cost considerations no longer apply. And monthly, quarterly, or annual cost variance can further complicate things.

So, what can you do to find the answer? A deliberate metrics strategy can help demystify cloud economics and support your organization’s ability to achieve its long-term strategic goals. In this blog post, we’ll cover the elements of a good metrics strategy and how you can create one, and we’ll share some KPI examples to help you get started.

The key: governance in metrics strategy

Metrics and reporting have an uncanny ability to mimic bacteria in the natural world–once a report exists it will self-duplicate, modify slightly, and duplicate again. For instance, if you pull the listing of all standardized reports in a department with weak governance, most of them will be copies of an existing report, where only one metric or filter has changed.

While the growing body count of reports is concerning, the real nightmare appears when you’re considering individual metrics. In a poorly governed organization, one metric (i.e., IT spend to revenue ratio) may have a “single” definition that’s actually calculated with multiple formulas. Who owns or is accountable for a truly singular definition? How many iterative formulas exist and who relies on each of them?

There is only one axiom for creating a strong metrics strategy: there has to be a standardized governance structure for metrics and reports. How do you do it? At a minimum, there are four requirements:

1. A mechanism of control to identify which metrics/reports can be standardized
2. An approval process for metrics definitions and formulas
3. A person or team to control which reports and metrics are in production and which are retired. If a team structure is used, consider different personas to divvy up responsibility:
   • Finance Persona – Validates and makes business data available
   • FinOps Persona – Defines metrics roadmap, including reviewing, approving, and retiring, and is responsible for providing help and support
   • Tech Persona – Validates and makes cloud data available, and implements optimization strategies
4. A monitoring and reporting cadence to ensure that non-compliant reports and formula changes are not in production

The bottom line is that the key to an effective metrics strategy is strong governance.

The paradox: customized and comparable metrics

Managers assembling a cloud metrics strategy for the first time often hold two simultaneous opinions: (1) “I’d love my metrics to be tailored to my unique way of doing business to measure internal performance,” and (2) “I really need my metrics to be comparable to a peer set of companies in my industry.” Both of these make sense. There’s massive business benefit to each approach: “I can optimize internal performance and hold myself accountable to external standards.”

But, there’s a problem. If everyone’s metrics are tailored and localized to their business, how can they possibly be compared to anyone else? Some organizations circumvent this by hiring professional service firms to conduct expensive surveys in a peer set, which can cost hundreds of thousands of dollars. And… they only exist for a point in time.

You have to decide what your organizational priority is. If it’s comparability to a peer set, then you should first consider the available market data at your price point, and then tailor a metrics and reporting strategy that adheres to published standards.

If it’s more important to have customized metrics to achieve benefits, then understand you may not be able to make peer set comparisons down the road.

Which approach is right? There’s no one-size-fits-all strategy. This is the one-way door nature of developing a metrics and reporting strategy. A change in philosophy could take years, so you need to make an informed decision.

Historically, the factors that influence which approach wins out correlates with:

• Age of the organization: older organizations lean towards a comparability strategy, while newer organizations prefer a customized approach.
• Number of competitors: organizations with many competitors focus on comparability, while organizations with less competition appreciate customization since there are fewer peers for comparison.
• Cost vs. growth focus: comparability is a key component of strategic cost reduction strategies; whereas, companies looking to grow often seek optimization through a custom approach.

The hurdles: 5 measures of a “good” cloud metric

Once an organization has (1) established good governance and (2) determined the metrics approach, it’s ready to start defining them. With thousands of metrics to potentially choose from, is there a rule-of-thumb to determine if a metric is comparatively “good”? We propose a series of five hurdles that all metrics should clear before being rolled out to your organization:

1. Material
A good metric should align with organizational and corporate strategy. If the metric is related to a function, it should be significant to the core competencies of that function.
Question to ask: “Does the information this metric tells me move the needle?”
Example: “Total IT spend as a % of net revenue” will tell you if IT is growing inline with your business.

2. Volatile
Good metrics have outputs that vary frequently or have volatility. If a metric has the same output, or only changes slightly from time to time, it’s unlikely to influence an action from management (i.e., it gets ignored).
Question to ask: “Is this metric the same every day? If so, can we develop something more informative?”
Example: “TB data transfer per customer” will identify seasonal patterns or one-off spikes of activity.

3. Understandable
I love a good logarithmic function in my statistical analysis as much as the next person. However, to drive adoption of metrics more broadly, they must be easy to understand. Consider avoiding too many variables, fractions of fractions, logarithms, trigonometry, or exponents. Consider taking a gradual approach to improve your metrics while implementing a longer-term strategy.
Question to ask: “If computing the metric requires anything beyond fourth grade math, is there a simpler way to convey this information?”
Example: “AWS costs per user” is easy to understand, and links IT to those who use it.

4. Actionable
The best metrics have built-in, proactive business actions. For instance, if metric A is above threshold B, then do action C. Many metrics exist because they are novel, or represent something management “wants to keep tabs on”. But if they don’t connect to an action someone can take, consider making changes so it can.
Question to ask: “If a metric is ‘just for my information’ and I can’t define an action because of what I see, is there a better metric that drives actionable information?”
Example: “Compute rightsizing opportunities” could trigger optimization actions and lead to savings opportunities.

5. Rare
The hardest of the five hurdles is to make sure that there aren’t too many metrics. I often tell a joke about Key Performance Indicators (KPIs): “You wouldn’t have 100 keys on your keychain; it wouldn’t fit in your pocket! So let’s separate the ‘PIs’ from the ‘KPIs’ and get closer to 5-10.” The larger the group relying on customized metrics, the fewer true key metrics you should have.
Question to ask: “If I cut this metric from my report, would anyone miss it?”

Conclusion and next steps

All companies and job roles can create and use an effective metrics and reporting strategy. In fact, everyone can and should routinely re-evaluate their cloud metrics to ensure they’re meeting the current needs of the business. It’s also never too early to start.

We suggest starting here:

• Establish a good governance structure and get current cloud reporting under control.
• Decide which philosophy should influence your cloud metrics and reporting strategy (comparable with peer set or customized to your business).
• Make sure that all cloud metrics pass the five metrics hurdles (material, volatile, understandable, actionable, and rare), and never be afraid to replace metrics that don’t measure up.

Here are a few proven metrics that follow our “See, Save, Plan, Run” framework, and can serve as a starting point for any company:

• See (track and allocate): bill trend rate, untagged spend rate, standalone accounts rate
• Save (optimize and save): compute rightsizing opportunity count, commitments coverage rate
• Plan (plan and evaluate): bill-shock frequency, forecast variance
• Run (manage and control): CFO CSAT, optimization evolving score, governance score
• Bonus (use unit metrics): spend-to-revenue ratio, S3 storage efficiency ratio, EC2 compute efficiency ratio

For more detailed information, watch this AWS CFM Talks webinar, “Creating an Effective Metrics Strategy for your Business.”

Related resources:

Unit Metric – The Touchstone of your IT Planning and Evaluation

Well-Architected Lab: Cloud Intelligence Dashboards- KPI Dashboard