Measuring performance
Overview
Measuring performance is the starting point to identify opportunities for improving the speed of web applications. The measured performance is influenced by the quality of content delivery using a CDN, and by the way a web application is designed. This article focuses on the best practices for measuring the Content Delivery performance. This practice is important for benchmarking CDNs (e.g. rolling out a new CDN), for switching CDNs in a multi-CDN architecture, and simply for ongoing CDN configuration changes (e.g. rolling out new protocol such as HTTP/3). To obtain the most relevant and accurate CDN performance data, testing conditions must be carefully selected: Measurement techniques and testing conditions.
Measurement techniques
Synthetic testing for application-level performance measurement
Synthetic testing is a measurement technique in which operators simulate the scenario of a user loading their web application. It can be done simply from the operator's computer using tools like Chrome's Lighthouse, or from a datacenter using tools like Webpagetest.org, or from a sophisticated network of probes deployed in datacenters and ISP backbones across the world, using tools such as Catchpoint or CloudWatch Synthetics.
This approach is beneficial to understand the performance of your web application's code (e.g. impact of javascript on page load). It's a recommended approach to monitor the improvement or regression of your application's performance after a code update, in the different steps of your code release process (e.g. synthetic testing in pre-prod environment).
However, synthetic testing falls short when measuring the performance of CDNs. For example, synthetic testing probes usually sits in networks which does not carry real user traffic, for which not all CDNs' routing systems are optimized. In addition, traffic generated by synthetic testing is relatively low compared to real traffic load, which doesn't reflect well how CDNs warm their network with ramped up traffic. For such reasons, it's recommended to use Real User Measurement (RUM) techniques for benchmarking CDNs. Read the
Real User Measurement for CDN-level performance measurement
If you want to measure the performance of your CDN such as CloudFront, consider using Real User Measurement (RUM) technique, using tools like CloudWatch RUM. RUM gives you the most accurate data about how your web application is performing from users' perspective. CloudWatch RUM allows you to understand how your CDN's performance is impacting your application's Core Web Vitals such as Largest Content Paint metric (LCP). You can also use RUM to correlate the CDN performance with your business KPI (e.g. number of user visit, sales, etc...). To use CloudWatch RUM, you need to add a javascript tag to your web pages. The javascript collects data from browser APIs, such as page load times with breakdown in connection steps (DNS lookup, TCP connection, etc..) or Core Web Vitals, and then send them to CloudWatch RUM for dashboarding. You can analyze your CDN performance by filtering on specific dimensions such as the browser type, user country, or a specific page id.
When your CDN is showing latencies in one of these dimensions (e.g a specific page or geographic region), drill down the networking metrics of requests within the outlying dimension. Slow TCP connection times might indicate sub-optimal routing, Slow time to first byte on cache misses might indicate a slowing origin. Use CloudFront server side troubleshooting in combination with CloudWatch RUM to scope down further and understand any latency issues.
Best practices for CDN benchmarking
Consider the following best practices to setup a non-biased testing conditions for benchmarking CDNs like CloudFront:
- Send a significant portion of your traffic (20% at minima) to each CDN, to measure the actual performance of the CDN at scale. CDNs like CloudFront show better performance at scale when warmed (e.g. cache populated, TCP connection pool warmed, and DNS entries cached by ISPs)
- Conduct the benchmarking testing in same conditions for all CDNs. Send the similar amount of traffic to each of your CDN, in the same time, and to the same user base.
- Conduct the testing for a relevant amount of time. For example, If you deliver long tail content, CDNs might need days to populate their cache and ramp up their Cache Hit Ratio to stable state.
- Configure CDNs with the same capabilities, such as compression, image optimization, protocols (HTTP/3 vs H2, TLS1.3 vs TLS1.2, IPv6 vs IPv4), origin acceleration (Origin Shield), etc..
Some 3rd party solutions for traffic steering, such as Pulsar by NS1 or Citrix Intelligent Traffic Management, include their own RUM data collected from sources outside of your application. If you use such tools, consider the following gotchas when thinking about CDN performance benchmarking:
- Are CDNs tested in the same conditions? similar configurations and optimizations.
- How much data points are collected for every dimension? More data points leads to more accurate benchmarking (warming CDNs, statistical errors, etc..)
- Do the tested objects have the same characteristics of your application (e.g. Static vs Dynamic, small object vs large object, popular objects vs non popular objects etc...)
- Some of these tools offer the possibility to do RUM testing on your own CDNs. Consider this option, as it will better reflect the performance that your users will experience with your CDNs.
- Use the most appropriate combination of performance metrics (latency, throughput, P50 vs P90) and availability that is relevant to your application. For example, large object downloads are more sensitive to throughput than to latency.