Powering .NET 8 with AWS Graviton3: Benchmarks

Introduction

Pioneering computer scientist Alan Kay said in 1982, “People who are really serious about software should make their own hardware.” At AWS, we’ve designed quite a bit of our own hardware, and have increasingly moved to use our own custom-designed silicon, including the AWS Graviton, AWS Inferentia and AWS Trainium processors. The newest Graviton processor, Graviton4, was announced at re:Invent 2023 with new r8g instances in preview.

In December 2020, we published the blog Powering .NET 5 with AWS Graviton2: Benchmarks that compared the performance of ASP.NET Core with .NET 5 on then-current Graviton2 (arm64) and Intel-based (x86-64) processors and we found that Graviton2 outperformed the x86-64 instances. Since that blog was written, .NET has advanced to version 8.0, two generations of Intel-based EC2 instances have launched, and Graviton3 EC2 instances became widely available. We therefore felt it was time to revisit the comparative performance of .NET—in this case, .NET 8.0—on AWS instances running Graviton3 and current-generation Intel silicon. As in the 2020 tests, we included tests from the ASP.NET benchmarks GitHub repo, which implement scenarios from the TechEmpower Web Framework, with a key difference being that we included tests that query an Amazon Aurora PostgreSQL database running on db.m5d.large instance, which uses physically-attached Non-Volatile Memory Express (NVMe)-based SSD storage for high performance. We also included the most demanding test that we used in 2020, which encrypts a weather summary pulled from an EF Core in-memory database. We tested the applications on compute-optimized instances: c7g.xlarge and c7i.xlarge.

Testing infrastructure and setup

We used four EC2 instances in total, two each for arm64 and x86-64, plus another EC2 instance on which we ran bombardier, the popular open-source HTTP(S) tool which does the actual performance testing. See the following infrastructure diagram. All the instances were running Ubuntu 22.04, and were launched—along with the Amazon Aurora database—in the same Availability Zone (AZ) to minimize any other sources of latency. We installed the final .NET 8.0 for Linux binaries on all four instances, and loaded the (already-compiled) TechEmpower ASP.NET Core MVC benchmark application on one each of the x86-64 and arm64 instances, and the encrypted-weather-summary ASP.NET Core Web API application on the other pair. All the applications were re-targeted to net8.0 and NuGet packages updated prior to building them.

Testing infrastructure setup

For EC2 instance types based on Graviton, each virtual CPU (vCPU) corresponds to a physical processor core, while for those based on the x86-64 architecture, each vCPU corresponds to a logical core (via simultaneous multi-threading). When comparing instances with an equal number of vCPUs, the Graviton-based instances provide twice as many physical cores.

Hourly cost comparison

The Graviton3 instances have lower hourly costs than their x86-64 equivalents, as shown in the table below. The Intel-based c7i.xlarge is ~23% more expensive than the equivalent arm64-based c7g.xlarge:

Note: prices are for on-demand compute in the us-west-2 (Oregon) region, where testing was performed, as of November 2023.

Test details

Bombardier lets you specify the length of the test and the maximum number of simultaneous connections it will use to hammer the target. For all the tests, as in 2020, we chose 20 seconds as the duration, and we “warmed up” each application using curl to hit each endpoint first. At low numbers of connections, the CPU on the target instances wasn’t being stressed for some tests, notably the TechEmpower Fortunes test which is hard-coded to return only five rows in a simple HTML view. The other tests all returned JSON, and allowed specifying the number of rows. We therefore used higher numbers of concurrent connections for some tests. The following table shows the values we used for each parameter for the four different tests:

Test results

Benchmark runs and units

We used Bombardier to run three tests for each benchmark on each tested instance and averaged the results. There was only small variation between each run, and in fact the results for the TechEmpower Fortunes and Updates tests were nearly identical for the x86-64 and arm64 instances. We believe this is because these tests are so computationally inexpensive, that we were saturating network bandwidth before stressing the processor.

We then charted the average values for latency, requests per second, and total successful (HTTP 200) requests per hourly cost. The latency values are in milliseconds (ms). Note that we had to normalize the values (that is, multiply or divide by a scaling factor) in order for the results of different tests to be shown on the same chart. The scaling factor is noted on the charts.

Summary of results

We found it interesting that the c7g instances—which became available in May 2022—were able to outperform the c7i instances that launched in September 2023. When c8g instances based on Graviton4 become available, we’ll repeat these tests to see how much of an advantage they confer.

Latency

The measured latency for two of the tests—the TechEmpower (TE) Fortunes and TE Updates—were virtually identical, likely due to the CPU not being stressed as noted earlier. For the computationally-expensive encrypted-weather-summary test, the Graviton3 instance had roughly 34% lower latency, and for the TE MultipleQueries test, it had 7.6% lower latency. See the following chart.

Latency chart

Requests per second

The results were broadly similar looking at the number of requests per second measured: almost exactly the same for the TE Fortunes and Updates tests, and the Graviton3 serving 52% more requests/sec for the encrypted-weather-summary test, and 7.9% more on the TE MultipleQueries test.

Requests per second chart

Price-performance

The real advantage of the Graviton3 instances becomes even more apparent when you look at the performance per cost by dividing the number of requests served by the hourly price of the instance. Here, due to the significantly lower cost of the c7g, the Graviton instances handled 23% more requests/dollar for the TE Fortunes test, 23.8% more for TE Updates, 32.7% more for the TE MultipleQueries test, and a staggering 85.6% more for the encrypted-weather-summary test.

Price-performance chart

Conclusion

One key takeaway from running these tests is that, for multi-threaded applications under heavy load, the Graviton3 processor with its greater number of physical cores exhibits superior performance. Given the lower pricing for Graviton3, it’s likely still worth evaluating your ASP.NET Core applications on arm64 even if they’re not running under a heavy load. Modern, cross-platform .NET versions can run locally on developers’ Windows or macOS machines running on x86-64 or arm64 processors as well as cloud-based Linux and Windows EC2 instances, containers and Linux-based serverless Lambda functions, freeing your teams to develop on the environments they love, and seamlessly deploy to AWS without code changes.

Learn more about Graviton at AWS Graviton Processors. Many AWS managed services are available on Graviton, so you can incorporate the price-performance benefits into almost any architecture or workload.