Run Selenium tests at scale using AWS Fargate
Tools such as Selenium eliminate repetitive manual testing that consumes lots of time and effort. Automated testing conforms to the ideas of Agile and DevOps, which endorse a continuous delivery workflow. Selenium has become a popular tool for testing, because it is open source and meets the requirement of quick and reliable testing, which helps enterprises save time and money.
What does a testing workflow look like?
Let us consider a scenario where a developer has added a new feature or made a bug fix to an existing product and checked in the changes to Git. As part of the CI/CD pipeline, the following manual and automated steps would typically occur:
- Unit tests are run and then the new code gets built.
- The built code gets deployed in a QA or staging environment.
- Once the deployment is healthy the QA team will run the integration and regression test cases to certify the build.
In a typical case, the number of test cases may vary between a handful to a few thousand depending on the application. Executing these test cases can take lots of time, which slows the process of getting features deployed in production.
The bottleneck when using automated testing is hardware resources, such as virtual machines (VMs) and vCPU, because you can run as many tests as your hardware will support. The traditional ways to accelerate this test case execution are:
- Start more of these test case executions in parallel from multiple machines, monitor them, and collect results.
- Add more physical machines or VMs, so you have enough resources for the browser agents to run these tests.
These methods are not scalable and cost both money and time. The approach outlined in this blog post using Fargate Spot to add more resources in parallel to overcomes these specific problems.
The following shows a high-level view of all the components:
Selenium Hub is the central point in Selenium that routes the JSON test commands to Fargate and is used to parallelize and distribute the load between the browser agents for test case execution. The whole infrastructure gets deployed in an ECS cluster with the following strategy for the default capacity provider:
- FARGATE -> Base: 4, Weight: 1
- FARGATE_SPOT -> Weight: 4
Note: Out of five instances, four of them will get provisioned as SPOT, and one of them will get provisioned as ON_DEMAND.
In this setup, Selenium Hub, Chrome Node (Selenium Docker image with headless Chrome), and Firefox Node (Selenium Docker image with headless Firefox) are deployed as ECS services. All ECS services have autoscaling enabled, with the following scale-in and scale-out policies:
- Add one instance if
max(CPUUtilization) >= 70in last 1 minute.
- Remove one instance if
max(CPUUtilization) <= 30in last 1 minute.
The Selenium Hub is backed by an application load balancer to which WebDriver clients connect to run the tests, and CloudWatch logs enable observability for any errors. The code in this repository is written as an AWS Cloud Development Kit (AWS CDK) construct.
Note: Constructs are the basic building blocks of AWS CDK apps. A construct represents a “cloud component” and encapsulates everything AWS CloudFormation needs to create the component.
Benefits of using Selenium on Fargate
Key benefits of this approach are:
- Depending upon the number of concurrent execution,
FirefoxNodewill scale out and in automatically (as it impacts the
CPUUtilizationmetrics), so customers pay only for the duration they use (no standing cost).
- Customers can deploy different browsers or different versions of the same browser depending upon their business need without thinking about the infrastructure or cost.
- Based on the capacity provider strategy, most of the instances get provisioned as
FARGATE_SPOT, which costs less money, to facilitate quicker execution and quicker scaling with more nodes.
- With this approach, customers can now run the regression and integration test cases as part of their nightly builds, which enables a daily release cycle to support growing business needs.
A headless browser refers to a web browser without a user interface. In other words, a headless browser can access the web page, but the GUI is hidden from the user. Otherwise, headless browsers are just like other browsers, and we use them to run WebDriver tests. Although headless browsers have many advantages, such as faster page loading, they also have limitations, including:
- Due to its faster page loading ability, sometimes it is difficult to debug issues.
- Real browser testing includes performing test cases in the presence of GUI. These tests are performed in front of the user, so the user can interact with the team, referring to the GUI and discussing where changes or corrections are required. In these cases, headless browsers cannot be used.
- Because headless browsers don’t have a GUI, it is troublesome to report errors with the help of screenshots. A conventional browser helps present defects by generating screenshots, which are a must in testing.
- In the case where a lot of browser debugging is required, the use of headless browsers can be challenging.
Test case execution
Here is the sequence of events that happens when we execute a test case using this architecture:
In this setup, the WebDriver client can talk to Selenium Hub with test cases results via the application load balancer URL. Once the request is received, Selenium Hub will direct the request to the Firefox Node or Chrome Node running as an ECS Fargate task to process the request. The browser node will then launch the browser in headless mode and execute the tests.
Build and deploy
The following prerequisites will be needed:
- AWS CDK should be installed for testing. You can read more about it in the Getting started with the AWS CDK documentation.
Yarnneeds to be installed; you can check the installation status by running the following command:
npm install -g aws-cdk cdk --version 1.87.1
Yarnis not installed, run the following command:
npm install -g yarn yarn version >1.22.10
- An AWS account and console access are also required.
Check out the code from this repository using this command:
git clone https://github.com/aws-samples/run-selenium-tests-at-scale-using-ecs-fargate cd scaled-test-execution/
Because the code is created as an AWS CDK construct, the following parameters can be customized as part of the deployment:
Next, run the following command to start the deployment:
cdk deploy --require-approval never
Note: Once the deployment is successful, you should see the Selenium-Hub-DNS in the CfnOutput.
The complete Selenium Hub load balancer URL will look like:
Unit test cases can be executed by running the following command from the root directory:
$ npx projen test ? test | rm -fr lib/ ? test » test:compile | tsc --noEmit --project tsconfig.jest.json ? test | jest --passWithNoTests --all --updateSnapshot PASS test/hello.test.ts ✓ create app (730 ms) ----------|---------|----------|---------|---------|------------------- File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s ----------|---------|----------|---------|---------|------------------- All files | 100 | 72 | 100 | 100 | index.ts | 100 | 72 | 100 | 100 | 61-70 ----------|---------|----------|---------|---------|------------------- Test Suites: 1 passed, 1 total Tests: 1 passed, 1 total Snapshots: 0 total Time: 5.163 s Ran all test suites. ? test » eslint | eslint --ext .ts,.tsx --fix --no-error-on-unmatched-pattern src test build-tools .projenrc.js ✨ Done in 17.45s.
Integration testing (using WebDriver)
A sample WebDriver test case can be found under
sample-test-function folder, and we can run the following commands to build and execute the tests against the Selenium Hub load balancer URL:
cd sample-test-function && npm install npx wdio --hostname <>
The test case will navigate to google.com and perform a search.
All test cases should successfully pass, and the results should look like:
[chrome 87.0.4280.88 linux #0-0] Running: chrome (v87.0.4280.88) on linux [chrome 87.0.4280.88 linux #0-0] Session ID: 80329f4643463a93a4628a35de4aaab4 [chrome 87.0.4280.88 linux #0-0] [chrome 87.0.4280.88 linux #0-0] Play with google [chrome 87.0.4280.88 linux #0-0] ✓ navigate to the site [chrome 87.0.4280.88 linux #0-0] ✓ start a search [chrome 87.0.4280.88 linux #0-0] [chrome 87.0.4280.88 linux #0-0] 2 passing (8.2s) Spec Files: 1 passed, 1 total (100% completed) in 00:00:10
Load testing (using webdriver)
To simulate load testing, we ran the above mentioned test case in parallel (closer to 10 concurrent session), which made the
CPUUtilization go above 70 percent, resulting in autoscaling.
The following screenshots were captured using Container Insights and AWS ECS console.
AWS ECS console (ECS tasks):
AWS ECS console (ECS service):
Container Insights (ECS cluster):
Container Insights (ECS tasks):
Container Insights (Map view):
After successful test case execution, the cluster will automatically scaling down when the
CPUUtilization goes below 30 percent with a cool-down interval of 180 seconds. The following shows a preview of ECS services running with just one instance after scaling down to the desired capacity:
Run the following command from the root directory to delete the stack:
We have shown how a combination of AWS Fargate and Spot instances can help customers run tests in scale with low cost. This approach also allows customers to innovate quickly, fail fast, and reduce their release cycles to introduce new features into their product.