Fact-checking GigaOm’s Microsoft-sponsored benchmark claims

SQL Server on AWS delivers 40% price/performance advantage over Azure

In this blog, we will review a recent benchmark that Microsoft sponsored and GigaOm published on 12/2/2019. This benchmark is not credible because Microsoft and GigaOm use configurations of AWS that generate weaker performance, they have not been transparent on how it was run, and the benchmarks are not reproducible.

AWS is committed to providing objective, transparent, and replicable benchmarking data so you can make an informed decision for why to run SQL Server on AWS. The latest AWS performance benchmark shows that AWS has up to a 1.75x performance advantage and up to 40% price/performance advantage over Azure (see Appendix).

The GigaOm/Microsoft benchmark is not an accurate, head-to-head comparison. It claims that Azure has over 3x performance and 80% price/performance advantage compared to AWS. This claim by GigaOm and Microsoft is not reproducible, and was created by utilizing a modified TPC-E benchmark which uses Microsoft’s proprietary benchmark tool. These benchmark results are also inaccurate comparisons due to significant mismatches in the configurations and the price calculations for Azure and AWS. These inaccuracies include:

1. The benchmark uses four striped disks for Azure, but did not apply any striping to AWS. Striping is a technique that is commonly used to enhance performance.
2. The benchmark uses an older generation AWS instance (R4), ignoring AWS hardware innovations that the latest generation comparable instance R5d delivers. R5d also has up to 3.6TB local storage for additional performance benefits.
3. The benchmark leaves out significant cost components for Microsoft, resulting in an inaccurate price/performance result. The Microsoft cost does not account for the original cost of the Window Server licenses or the Software Assurance required to use Azure Hybrid Benefits. It also does not take into account the AWS programs that provide similar benefits, such as the Migration Acceleration Program (MAP).

AWS ran a performance analysis with comparable instance type and storage configurations using the publicly available TPC-C HammerDB benchmark tool. The benchmark running SQL Server on AWS delivers 1.75x better performance and delivers up to 40% better price/performance than Azure. You can use the same HammerDB benchmark tool and run your own TPC-C tests to replicate the latest results using this whitepaper. Similarly, you can use the same tool to replicate prior TPC-C like benchmarks from DB Best which showed that AWS delivered 2-3x better performance over Azure.

Performance is just one of the reasons customers such as NextGen Healthcare and Pearson choose AWS to run their Windows workloads, and according to a report by IDC, we host nearly two times as many Windows Server instances in the cloud as Microsoft. More and more enterprises are entrusting their Windows workloads to AWS because of its greater reliability, security, and performance. Publishing misleading benchmarks is just one more old-guard tactic by Microsoft, in addition to license complexity and licensing restrictions, to try to prevent customers from using the best cloud for their Windows workloads.

It is important that you have the facts when making a decision about your cloud provider. AWS encourages you to look for replicable research and benchmark your own workloads that help you verify which provider offers the best price, performance, and reliability. Don’t be misguided by vendor claims that cannot be validated.

Appendix

HammerDB TPC-C tests run internally on the same server and storage configurations as the GigaOm report deliver the following performance improvements over Azure when more optimal options are chosen:

About the Author

Fred Wurden is the GM of Enterprise Engineering (Windows, VMware, RedHat, SAP, benchmarking) working to make AWS the most customer-centric cloud platform on Earth. Prior to AWS, Fred worked at Microsoft for 17 years and held positions, including: EU/DoJ engineering compliance for Windows and Azure, interoperability principles and partner engagements, and open source engineering. He lives with his wife and a few four-legged friends since his kids are all in college now.