How to implement recording functionality in applications built using the Amazon Chime SDK for Javascript

Wataru Okada, technical researcher and engineer, R&D team, FLECT Co., Ltd.

This is a guest blog written by Wataru Okada, technical researcher and engineer, R&D team at FLECT Co., Ltd (FLECT). The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.

FLECT is a cloud native integrator based in Tokyo, Japan. We have built a wrapper component for FlectVideo Chat (Flvc) for the Amazon Chime SDK to offer our original functionalities such as virtual background. We can offer highly functional online meeting products to our customers quickly and easily with our wrapper component.  As one new feature, we are developing a functionality to record online meetings. One use case we are envisioning is to allow customers to analyze the quality of a conversation by reviewing the recorded meeting after the meeting has ended.

Some customers want to use online meetings with no additional software, other than their browser, for reasons such as security rules or other considerations.  We have developed a new recording functionality that works in the browser. This blog shows how to implement the recording functionality in applications built using the Amazon Chime SDK for JavaScript.

Overview of the solution

Online meetings have become quite common nowadays, and are widely used for work, classes, shopping[MA1] , and more. The ability to record these meeting sessions can provide great benefits to the host/users, such as, for example:

1. The recording can be used as a substitute for meeting minutes.
2. Those who could not attend the meeting can watch the recording later.
3. The recording can be analyzed later and utilized for future improvement studies.

We use the MediaRecorder interface of the MediaStream Recording API to record online meetings. With this interface, users can record online meetings without additional software. The overview of the solution is shown in the figure below:

Online meeting application with Amazon Chime SDK for JavaScript

First, we will briefly review how to implement an online meeting application using the Amazon Chime SDK for JavaScript.

The diagram above shows the main components of an online meeting with the Amazon Chime SDK for JavaScript and the data flow.

To start an online meeting, we do the following.

1. create a session for the meeting.
2. register the ID of the video input device and the audio input device to the session.
3. register HTMLVideoElement, HTMLAudioElement, and the ID of the audio output device to the session.

By step (2), the video data and audio data can be sent to each client through the backend by the Amazon Chime SDK. By step (3), the Amazon Chime SDK associates the registered HTMLAudioElement with the audio output device. When the Amazon Chime SDK receives video data and audio data, the Amazon Chime SDK plays them back with the registered HTMLAudioElement and HTMLVideoElement.

For example, to register each device and HTMLElement, one needs to do the following. For more information, see API Overview.this.meetingSession = new DefaultMeetingSession(configuration, logger, deviceController)
...
await this.meetingSession.audioVideo.chooseAudioInputDevice(audioInputDeviceId)
await this.meetingSession.audioVideo.chooseVideoInputDevice(videoInputDeviceId)
await this.meetingSession.audioVideo.chooseAudioOutputDevice(audioOutputDeviceId)

await this.meetingSession.audioVideo.bindAudioElement(outputAudioElement)

this.meetingSession.audioVideo.getAllVideoTiles().forEach((tile, index)=>{
    tile.bindVideoElement(outputVideoElement[index])
})

Implementation of recording functionality

Now, I will explain how to implement the meeting recording functionality. Roughly, the configuration is shown as in the following figure.

Most modern browsers support MediaRecorder API (CanIUse).

MediaRecorder can record video and audio using a MediaStream as input. MediaStream can be retrieved from HTMLAudioElement, HTMLVideoElement or HTMLCanvasElement. Therefore, we can also retrieve it from HTMLVideoElement and HTMLAudioElement, which we registered for the meeting session above. This time, we will get the MediaStream from these HTMLVideoElement and HTMLAudioElement and input it into the MediaRecorder. This will allow us to record within the browser.

If you want to record multiple HTMLVideoElement as a single media file, you can also write the HTMLVideoElement to HTMLCanvasElement in any layout and retrieve the MediaStream for the recording from that HTMLCanvasElement.

For example, the source code would look like this.

const stream =  new MediaStream();

// capture audio stream
const audioElem = document.getElementById("for-speaker") as HTMLAudioElement
const audioStream = audioElem.captureStream() as MediaStream
// capture video stream
const canvasElem = document.getElementById("for-recorder") as HTMLCanvasElement
const videoStream = canvasElem.captureStream() as MediaStream
// merge audio stream and media stream into one MediaStream
[audioStream, videoStream].forEach(s=>{
     s?.getTracks().forEach(t=>{
          stream.addTrack(t)
     })
});

// Setup MediaRecorder
const options = {
     mimeType : 'video/webm;codecs=h264,opus'
};
this.chunks = []   // storage of the recorded data

this.recorder = new MediaRecorder(stream, options)
this.recorder.ondataavailable = (e:BlobEvent) => {
this.chunks.push(e.data)
}
this.recorder.start(1000)

Demo

The recording feature described in this article is implemented in a sample application for online meetings in the repository below:

https://github.com/w-okada/flect-chime-sdk-demo

In this sample application, we will build the backend with a script using AWS Cloud Development Kit (CDK). Deploying is very easy if you have an AWS account. Let’s take a look at the process in detail.

Note: Deploying and receiving traffic from the demos created in this post can incur AWS charges.

Prerequisites

(1) Setup AWS Credentials

We will build a backend in AWS. For this reason, it is assumed that you have set up your AWS credentials before deploying. For more details, please see here.

(2) Install AWS CLI

We will use AWS Command Line Interface (AWS CLI). For this reason, it is assumed that you have installed AWS CLI. For more information, please see here.

Procedure

(1) Clone the repository

In this article, we will use the version with the aws-demo01 branch, so please specify the aws-demo01 branch and clone.

$  git clone https://github.com/w-okada/flect-chime-sdk-demo.git -b aws-demo01
$ cd flect-chime-sdk-demo/

(2) Configuration

Specify the CloudFormation stack name for the backend we will build in backend/bin/config.ts. This stack name can be anything you want, but it must be globally unique since we will be creating an Amazon Simple Storage Service (Amazon S3) bucket with this name as the prefix.

$ cat backend/bin/config.ts

export const BACKEND_STACK_NAME = 'BackendStackName' # <-- You should change. 
export const FRONTEND_LOCAL_DEV = false          # <-- Set false for deployment.

(3) Build the backend and deploy

Now, let’s build the backend. As mentioned earlier, we will build it using a script that uses AWS Cloud Development Kit (CDK), so you can complete the build by executing the following command. The process will take a few minutes.

$ cd backend
$ npm install
$ npm run build_all

(4) Build the frontend and deploy

The next step is to build and deploy the frontend. Execute the following command to build it.

$ cd frontend3
$ npm install
$ npm run build

After the build is completed, deploy it.

$ sh sync.sh

This completes the build and deployment. Now, let’s try to use it.

(5) Open the demo application

The URL of the deployed demo application can be found in demo_url.txt. Please access it with a browser.

$ cat demo_url.txt

$ cat backend/cfn_outputs.json 
{
  "xxxxxxxxxxxxxx": {
    "UserPoolClientId": "xxxxxxxxxxxxx",
    "UserPoolId": "xxxxxxxxxxxxxxx",
    "BucketDomainName": "xxxxxxxxxxx",
    "Bucket": "xxxxxxxxxxxxxxxx",
   <snip>
  }
}

$ cd backend/
$ npx cdk destroy