Automated Performance Regression Detection in the AWS SDK for Java 2.0

We are happy to share that we’ve added automated performance regression tests to the AWS SDK for Java 2.0. With this benchmark harness, every change to the SDK will be tested for performance before release, to avoid potential performance regressions. We understand that performance is critical to our customers and we’ve prioritized improving various performance aspects of the AWS SDK. In the past, we’ve relied on Pull Request reviews to catch any code changes that looked like would cause performance issues. With this approach, although rare, sometimes a simple line of code might get overlooked and end up causing performance issues. Also, at times, performance regressions were caused due to newer versions of SDK dependencies, and there was no easy way to monitor and quantify these performance impacts to the SDK. With the benchmark tests, we are now able to detect performance regressions with changes before they are merged into the master. The benchmark harness code is open-source and resides in the same repository as the AWS SDK for Java 2.0, and is implemented using the Java Microbenchmark Harness or JMH.

How to Run Benchmarks

To run the benchmarks, you need to build it first using mvn clean install -P quick -pl :sdk-benchmarks –am. Then, trigger the benchmarks using one of the following options:

Option 1:  Use the executable JAR

cd test/sdk-benchmarks
# Run a specific benchmark
java -jar target/benchmarks.jar ApacheHttpClientBenchmark
 
# Run all benchmarks: 3 warm up iterations, 3 benchmark iterations, 1 fork
java -jar target/benchmarks.jar -wi 3 -i 3 -f 1

Option 2: Use maven command to invoke BenchmarkRunner main method to run all benchmarks

mvn install -pl :bom-internal
cd test/sdk-benchmarks
mvn exec:exec

Option 3: Run the main method within each Benchmark class from your IDE

You can also run the main method within each Benchmark class from your IDE. If you are using Eclipse, you might need to set up build configurations for JMH annotations (check out the JMH page to learn how). You’ll note that per JMH recommendations, using Maven and executable JARs (options #1 & #2 above) are preferred over running it from within an IDE. IDE set up is a bit complex and could yield less reliable results.

How the Benchmark Harness Works

When the benchmark tool gets triggered, it first runs a set of predefined scenarios with different http clients sending requests to local mock servers. It then measures the throughput of the current revision and compares the results with the existing baseline results computed from the previously released version. The performance tests will fail if the throughput of the new change decreases by a certain threshold. Running the benchmark tests for every pull request allows us to block those with problematic changes. As an added bonus, the benchmark tool makes it easier to monitor the SDK performance over the time, and when performance improvement is made, we can inform customers with quantified performance gains so that they can benefit immediately from upgrading the SDK. Finally, the baseline data generated from the benchmark harness provides useful information, such as which HTTP client has the best performance or how many percentages of throughput gains it can achieve by tuning SDK configurations. For example, changing to use OpenSSL provider for NettyAsyncHttpClient can reach 10% more throughputs according to our benchmarks. With automated performance checks, we expect to limit unanticipated performance degradation with new version releases. For contributing, we also encourage you to run the benchmarks harness locally to check if there is a performance impact by the changes.