Unit metrics in practice – lessons learned

Voiced by Amazon Polly

We hope by now you have a good grasp of what a unit metric is and how you can create your own to evaluate the efficiency of resources usage or project your future spend. Before you jump into action, here are some of the lessons learned along the way we’ve experienced both creating and implementing unit metrics.

As you come to rely on the value that unit metrics deliver, the details of how they are prepared become increasingly important. To keep our friends in accounting and auditing happy, there needs to be a level of precision and formality about the process of counting the demand driver in a given period. This post will focus on the operational aspects of creating a well-functioning denominator for our unit metric.

The Book of Record

There should only be one source of truth for the production and delivery of a denominator that is used in a unit metric. That source of truth is often known as the Book of Record (BoR).

Ownership – who in the business, whether it be engineering, product, or finance, owns the denominator? Someone or some team needs to own the measurement, reporting and analysis of your unit metric’s denominator. There must be an accountable party.

Counting – a reliable mechanism must be developed to deliver an accurate count of the number of occurrences of the demand driver. The process also needs be timely with respect to the accounting period close. If your unit metrics are used to calculate product margins or if there is a variance analysis process that is a part of the month end close, you need to make sure that the timing of the BoR process is coordinated with the month end close.

For example, if accounting needs data finalized on “work day 2” and the BoR process that provides the denominator is not complete until work day 4, we’re going to have a problem. Partner with accounting or financial planning and analysis to make sure your timelines align with theirs. Please allow us to give you an important hint… financial reporting timelines are more important than yours. Figure out how to get it done faster.

Reporting – How do you present this data to all the stakeholders? Preferably it should be done using self-service dashboards using a BI tool like Amazon QuickSight. But it can be as simple as an e-mail or Excel workbook. Users should know the cadence of when the data are ready (locked down) as well as where to go to find it if it isn’t pushed to them.

Scale Factor – Selecting a scale factor that represents the information in a meaningful way will make it easier for the consumers of the unit metric to digest and remember it.

For example, my unit metric is cost per API response. It can be represented at a single unit level:

$ / API Response = $0.00024681

or it can represented as cost per million API responses, which would be shown as:

$ / M API Responses = $246.81

The latter is much easier to identify with unless you’ve been a cost accountant your whole career.

Establish a Cut-off Process

This is one of the most important concepts to properly implement as far as our partners in accounting and finance are concerned. Cut-off is the process of determining what data belongs to a given reporting period and what data does not. Having what is known as a “clean cut-off” will endear you to both accounting and internal audit. It is essential to have a clean cut-off especially if the data you develop is used to make strategic decisions such as pricing or where to allocate budget.

Fiscal calendar vs. the calendar I use every day – The calendar that defines a month for accounting is not always aligned to the good ol’ Gregorian calendar (no offense to users of the Julian calendar).  Not only that, but accounting may define the beginning or ending of a week different from what most of us use day to day.

Ensure your BoR counting week, month, or quarter aligns to the reporting week, month, or quarter that supports accounting. If fiscal calendars are new to you, use this as an opportunity to reach out to your partners in accounting and finance and have them teach you how they work at your company.

The timestamp – we have a book of record and a calendar of record, let’s also have a clock of record. Yes, the timestamp associated with transactional data that is produced matters especially if you are in a global organization.

What is the time zone that establishes when the business day begins or ends? Home office time? Universal Coordinated Time (UTC)? Perhaps it is the time zone of your largest AWS region. Whatever it is, it needs to be consistently applied in your BoR process. Per best practices, timestamps should be stored as UTC plus a regional offset. You need to ensure the counting processes that are developed use a single time reference when establishing temporal boundaries that comply with company’s standards for reporting.

What happens when data from a given period arrives after cut-off? You have a couple of options, and like all things Cloud Financial Management, it depends. In finance and accounting, there is the concept of materiality. Simply stated, if data shows up late (after cut-off) from the period we are about to report, is there enough of it such that it will impact our ability to make a well-informed decision? Could the consumers of this data make a bad decision or be misled as a result of late arrivals not being included in the proper reporting period?

Depending on how your process works, you may require anywhere from seconds to days for all the data that belongs in the reporting period to arrive. How long do you wait for it all to show up before you say it is good enough? This is something you will need to work out since the answer is different for every company. However, it is something you need to think through and build into the process of counting the demand driver. Remember, one of the goals is to publish a reliable metric that every team can trust so it can be confidently used for decision making. It also should be noted that AWS Cost Explorer and the AWS Cost & Usage Report are updated at least once per day. Make sure the data for the period to be reported has arrived before pulling the data from either of these two sources.

What happens if there is a “log jam” and some amount of prior period data arrives after cut-off and after the closing process has concluded? It doesn’t happen often, but it can happen. What is the number of late arriving records or the change to the demand driver as a result of late arriving data such that you need to restate the officially published demand driver and subsequently, a period’s unit metric? Restatement thresholds should be worked out in advance. Trust us, this isn’t something you want to do on the fly. There should be an established set of standards and processes to restate that are developed at the same time a unit metric process is being developed. This creates another wonderful opportunity to work with your partners in accounting and audit. It also shows consideration for their role in the business and makes you a valued partner in their success.

Design your process with an audit in mind

Hopefully, the unit metrics that have been developed become core KPIs for the products and services that you offer. If this is the case, it would be good to be prepared to be on the receiving end of an audit. If your unit metric is being used to calculate the gross margin of a product or service, it is a reasonable expectation that there will be a review of processes used to calculate your unit metric. Don’t fear an audit! Be prepared for it and embrace the opportunity for an arm’s length assessment of your system and the feedback an audit provides to make your systems better.

Build your unit metric processes with prospect of an audit taking place sometime down the road. First and foremost, have a plan to archive the data used to calculate your unit metrics in a cold storage class like Amazon S3 Glacier or S3 Glacier Deep Archive. Keep the data for the required amount of time while reducing the storage retention costs.

After the audit team reviews your unit metrics creation process, testing the data will be on the agenda. Develop a storage scheme that makes it easy to find the data that are to be reviewed. Rack, pack, and stack the data to be archived. By that we mean, tag it, compress it, and store it using a methodology that makes it easy to find when the call comes to retrieve it. Tag your S3 Glacier Vaults and take advantage of built in mechanisms like S3 Glacier Vault Locks to ensure your compliance retention policies are enforced. We promise you, your auditors will appreciate the fact that the data retention process employs strong enforcement mechanisms to ensure controls work as designed.

It is important to develop an operationally robust set of processes and controls around the counting of unit metric denominators. These processes and controls extend well beyond engineering especially if a given unit metric is used in your organization to make pricing decisions, drive cost allocations for shared services, or help product managers shape how development backlog is prioritized. The details matter!

In our last installment of this blog series, we will look at how unit metrics can help to create alignment across business functions within your enterprise.