Load testing applications built with the Amazon Chime SDK

Builders creating audio-video communication applications can find it challenging to load test using a large number of attendees because of the challenges with inviting large number of unique individuals to join a meeting. To overcome this issue, we have built a solution to let the builders test their application with a large number of attendees without needing actual people.

In this blog, we will teach you how to test applications built with the Amazon Chime SDK for Javascript (Amazon Chime SDK for JS) at scale under a variety of load conditions. We provide an open-source tool, the Amazon Chime SDK client launcher tool, to simplify your approach. The tool can be used for the following scenarios:
a) test your app with large number of participants; or
b) test multiple independent sessions of your app

Note: Deploying the demos created in this post can incur AWS charges.

Solution Overview

The Amazon Chime SDK client launcher tool eliminates the need for humans to join as participants in a meeting in order to test features and functionalities of applications built using the Amazon Chime SDK for JS. It simulates real attendees by launching test Amazon Chime SDK for JS clients, each in a new browser tab. The features of each client can be dynamically controlled while a load test is active.

Here are some scenarios the tool is designed to support:

Simulate attendees in a meeting

The Amazon Chime SDK client launcher tool can help builders and developers launch their application to start a meeting with multiple participants and observe how the app behaves.

Optimize Amazon Chime SDK for JS client app

The Amazon Chime SDK client launcher tool can help builders and developers launch their application to start multiple test meetings with multiple participants, collect metrics on these tests to understand how the application can be optimized for a better user experience.
Example: The tool can be used to monitor the average time taken for the clients to launch on a browser or track how long it takes for the audioVideoDidStart() to get triggered.

Simulate different scenarios on client app

The tool can also help simulate different scenarios on each individual attendee dynamically while a load test is active.
Some scenarios that can be already tested using the tool are:

1. Join a meeting
2. Self Mute
3. Start Local Video
4. Self Unmute
5. Stop Local Video
6. Leave Meeting

The developers can add custom scenarios to the load test tool by writing a custom methods in the ClientController class and invoking it in the performActivity() of the ChildActivity.

Prerequisites:

1. Node JS on the local machine

1. Set up and install Node JS in the local machine by following the instructions mentioned here.

2. AWS Command Line Interface on the local machine

1. Set up and install AWS CLI in the local machine by following the instructions mentioned here.

3. App built using the Amazon Chime SDK for JS

1. Build a client using the Amazon Chime SDK for JS and host it on the internet. Alternatively, you can deploy a serverless version of the same using the guide mentioned here. The app should be compatible to run on Google Chromium based browsers.

4. AWS Account with necessary permissions

Once the AWS CloudFormation Stack is setup using the CDK script provided in the package, the necessary AWS Identity and Access Management (AWS IAM) policies are automatically set up.
These are the permissions that will be added:

1. Amazon Chime SDK – to support Amazon Chime SDK for JS clients
2. Amazon Elastic Compute Cloud (EC2) – to launch instances of the clients
3. Amazon Simple Storage Service (S3) – to host the latest/modified code for the tool so that it can be pulled into the EC2 instances upon launch and a medium for the launcher tool to listen and act ontrol
4. Amazon Systems Manager – to be able to send commands to EC2 instances from local machine

Solution Architecture:

The framework needs two major components:

1. Launcher Tool: AmazonChimeSDKMeetingsLoadTest tool which is responsible for launching the clients
2. Client: URL to clients based on Amazon Chime SDK for JS built by the builder

Once you spawn the clients on remote EC2 instances using the launcher tool, you can control the features of the client from the local machine. The load test tool launcher tool uses puppeteer to spawn the desired number of headless Chromium browsers. Each browser represents an attendee joining the client.

The sequence flow is as follows:

1. Send pre-defined commands with mapping of the meeting name and attendee name from local machine to the S3 bucket
2. Launcher tool running on the EC2 instances reads the commands from the S3 bucket
3. Based on the commands read by the launcher running on each EC2 instance, the launcher tool changes the functionality of the clients

The following diagram represents how the launcher tool works at a high level on each EC2 instance launched using the CDK provided in the package. More details provided in the Launching a test section described later.

Setting up the tool:

1. Download the tool from Github from here on your local machine.

1. git clone https://github.com/aws-samples/amazon-chime-sdk-meetings-load-test

2. Install all the necessary dependencies in the packages

1. cd AmazonChimeMeetingsSDKLoadTest
2. yarn install

3. Update the element names in Constants file

Update the element names used in the builder created app by setting the constants in the configs/Constants.js file. The ones that have been set now are with respect to the Amazon Chime SDK for JS demo app.
For example, the local camera toggle button in Amazon Chime SDK for JS demo app is identified by button-camera here. So the `cameraButton` in the configs/Constants.js is set to button-camera over here.

Launching a test:

1. Link the client application URL

Once the package is downloaded, set the “clientUrl” in the configs/Constants.js file with the URL to the client built on the Amazon Chime SDK for JS client app.

export const clientUrl = 'client-url-link'; // this variable should point to the client url

2. Deploy CDK

The CDK script provided with the tool will help set up the infrastructure needed to run the load test. Based on capacity estimation, estimate the number of required EC2 instances and set the NO_OF_EC2_INSTANCES here. The stack name used here is CCLStack – this can be changed from the configs/Constants.js file in the configs directory.

1. cd CDK
2. Runyarn install(only for the first time)
3. Ensure AWS credentials of the desired account is already setup.
4. Runyarn deploy

Note: Once the deployment is complete wait for the Amazon EC2 instances to get to the ‘running’ state with all checks ‘passed’. No attendee clients have been launched yet.

3. Launch Amazon Chime SDK for JS clients on each EC2 instance

The clients will be invoked when the ClientLauncher.js is invoked on the EC2 instances. The following command spawns the clients:

node ClientLauncher.js --activeVideosPerMeeting 10 --loadTestSessionName “MeetingName”

To test and try the command on the local machine, append the parameter –localMachine

Run the launcher using node ClientLauncher.js <optional parameters>

Optional parameters:

A simple way is to run the command on all the EC2 instances associated to the AWS CloudFormation Stack. SSM command from the local machine can help achieve the same.
A script that does this, has been provided in the scripts directory. The following steps helps achieve this:

1. cd scripts
2. Ensure AWS credentials of the desired account is already setup
3. Runnode InvoleClientsUsingSSM.js

4. Dynamically control Amazon Chime SDK features by sending Commands to S3

Once the clients are up on the EC2 instances, they are dormant. In order to control the functionalities of each active client, the user can send commands from the local machine to the launcher. Each command can target many or all the clients associated to a meeting.

This is achieved by sending the following commands to the S3 bucket. The load test tool listens to changes in the files in the S3 bucket and controls the client associated to it accordingly. A list of commands currently supported by the tool are mentioned below.

The following section below, describes the process to achieve dynamic control of the Amaon Chime SDK features:

1. After ensuring that the Amazon Chime SDK for JS clients are running (from step 3 above), the activity commands needs to be generated by the user/tester and written in the Activity.json file. The next section gives an example of writing the activity commands to the Activity.json file.
2. Go to the scripts directory cd scripts
3. Ensure desired AWS credentials are set
4. Run node UploadActivityCommandsToS3.js to upload the activity commands to the AWS S3 bucket.

You can modify the Activity.json file with new set of activity commands and upload to the AWS S3 bucket multiple times to simulate desired scenarios on the clients while a load test is running.

Adding custom activity scenarios:
The developers can add custom scenarios to the load test tool by writing custom methods in the ClientController class and invoking it in the performActivity() of the ChildActivity.

Example:

async toggleVideoOnPage(videoButton, browserTab, page) {
    try {
        if (page) {
            const videoToggle = await page.evaluate(async (videoButton) => {
                return new Promise((resolve, reject) => {
                    try {
                        document.getElementById(videoButton).click();
                        resolve('Success');
                    } catch (err) {
                        resolve('Fail');
                    }
                });
            }, videoButton);
        }
    } catch (err) {
        this.support.error('Failed to toggle video', browserTab);
    }
}

Example to run a simple test:

Let us walk through an example to have 10 attendees join a meeting and turning on the local camera. Attendee_1 turns off the local camera after 10 seconds and then leaves the meeting after 30 seconds.
The constants set up in the configs/Constants.js file is based on the element names used in the Amazon Chime SDK for JS meetings demo.

The corresponding Activity.json file looks like this:

{
  "activityCommands": [
    {
      "meetingName": "MeetingLoadTest",
      "attendeeName": "*",
      "commands": [
        {
          "activity": "Wait",
          "duration": 1000
        },
        {
          "activity": "JoinMeeting"
        },
        {
          "activity": "LocalMute"
        },
        {
          "activity": "TurnOnLocalCamera"
        }
      ]
    },

    {
      "meetingName": "MeetingLoadTest",
      "attendeeName": "Attendee_1",
      "commands": [
        {
          "activity": "Wait",
          "duration": 10000
        },
        {
          "activity": "TurnOffLocalCamera"
        },
        {
          "activity": "Wait",
          "duration": 30000
        },
        {
          "activity": "LeaveMeeting"
        }
      ]
    }
  ]
}

Note that, the meetingName set in the configs/Constants.js file is “MeetingLoadTest” and the attendeeNamePrefix is “Attendee”.

For simplicity, it is recommended that you create a meeting before running a load test for example by running a version of the client on the local machine. Here is how the 16 video tiles appear on the Amazon Chime SDK for JS demo app for 16 attendees look like when launched using the tool.

Capacity Estimates on the EC2 instances:

In order to simulate the ideal scenario of every attendee joining and turning on the video, we recommend to run at most two meeting attendee sessions per EC2 instance of C5 xl as the WebRTC threads utilizes a lot of the CPU resources.

The cost for running C5 instances per hour can be found here.

Monitoring Metrics:

You should monitor the active Amazon EC2 instances for the CPU and memory usage monitors on the AWS Console to ensure none of the resources are being over utilized. Over utilization of resources can result in skewed behavior of the clients thus result in bad audio video experience in each of the active meetings.

Following this AWS public documentation on meeting events, the clients launched during the run using the Amazon Chime SDK client launcher tool, you can monitor the successfully started and ended meeting with the relevant events.

Launcher:
The launcher is sending the following metrics:

Looking into the metrics emitted during the Load Test

Once the load test has started, the metrics emitted from the launcher and the client can be viewed on the AWS Cloudwatch dashboard. To view the metrics: Head to the AWS Cloudwatch console of the corresponding AWS account that was running the load test. Choose the Metric tab. Select aws-embedded-metrics. Filter the metrics based on the LoadTestSessionID. Further filter out the metrics based on the metrics that need to be verified.

Clean Up:

Terminate or stop the EC2 instances and other associated resources to prevent recurring charges.
The stack created should be deleted to destroy all associated resources. To do so,
1. Runcd CDK
2. Runyarn destroy

Conclusion:

After going through this blogpost, you should be able to use the Amazon Chime SDK client launcher tool to test your application built on the Amazon Chime SDK for JS for large meeting settings. You should also be able to add automated scenarios that you wish to perform during the test. To get started, follow the instructions listed above or you can read more about the Amazon Chime SDK here.