Blur your background and suppress noise in an online meeting application built with the Amazon Chime SDK for JavaScript

FLECT is a cloud native integrator based in Tokyo, Japan. We have built a wrapper component for FlectVideo Chat (Flvc) for the Amazon Chime SDK to offer our original functionalities. We can offer high functional online meeting products to our customers quickly and easily with our wrapper component. As new features of our component, we are developing a functionality to blur background and suppress noise. This functionality helps us to focus on the meeting by reducing the risk of distraction and also helps to increase visual privacy.

Overview of the solution

Online meetings are quite common nowadays, and are widely used for work, classes, and shopping, among other things. During online meetings, users want to focus on the discussion and not be distracted by busy backdrops or background noise. Amazon Chime SDK for JavaScript provides the functionalities to blur background and suppress noise, to help developers build more productive and comfortable online meeting applications while using the Amazon Chime SDK.

These features help expand the range of possible uses for online meetings and help increase the value of your application. For example, users can hold an online meeting in situations where there are distracting objects or sounds around, which distracts other meeting attendees from the discussion. Users may also wish to protect their privacy by not showing the background behind them or noises in their environment.

We can use the BackgroundBlurVideoFrameProcessor and the Video Processing API to blur background, and Voice Focus to suppress noise. These features are enabled in the Amazon Chime SDK for JavaScript. The Video Processing API allows various frame-by-frame processing to be applied to the output video stream. The BackgroundBlurVideoFrameProcessor is a processor within the Video Processing API, it identifies the background and the person in a frame. Amazon Voice Focus reduces unwanted background noise in your users’ microphone input.

Implementation of background blur and noise suppression functionality

It is easy to implement both background blur and noise suppression with Amazon Chime SDK for JavaScript. In both cases, we create a virtual device called TransformDevice that converts the input data, and register it as an input device just like a normal device. First, we will explain how to implement the blur background functionality. Then we will explain how to implement the noise suppression functionality.

Background Blur

We implemented blur background functionality in the following steps.

Create BackgroundBlurVideoFrameProcessor. In this step, we can choose the strength of the blur.
Create DefaultVideoTransformDevice with BackgroundBlurVideoFrameProcessor created at (1) and video device to be used.
Set DefaultVideoTransformDevice to MeetingSession.

That’s all. The actual code will look something like this.

if(await BackgroundBlurVideoFrameProcessor.isSupported()){
    const processor = await BackgroundBlurVideoFrameProcessor.create(undefined, { blurStrength: backgroundBlurLevel as BlurStrength });   // <----- Step (1)
    if (!processor) {
        await state.meetingSession!.audioVideo.chooseVideoInputDevice(deviceId); 
    } else {
        const chosenVideoTransformDevice = new DefaultVideoTransformDevice(state.meetingSession!.logger, deviceId, [processor]); // <----- Step (2)
        await state.meetingSession!.audioVideo.chooseVideoInputDevice(chosenVideoTransformDevice); // <----- Step (3)
    }
}else{
    await state.meetingSession!.audioVideo.chooseVideoInputDevice(deviceId);
}

Note1: The supported environment is shown in official document. Please see it.

Noise Suppression

We implemented noise suppression functionality in the following steps.

Create VoiceFocusDeviceTransformer. In this step, we can choose the complexity level of noise suppression.
Create VoiceFocusTransformDevice with VoiceFocusDeviceTransformer created at (1) and audio device to be used.
Set VoiceFocusTransformDevice to MeetingSession.

That’s all, it’s very straightforward. The actual code will look something like this.

let voiceFocusDeviceTransformer: VoiceFocusDeviceTransformer | null = null;
if (await VoiceFocusDeviceTransformer.isSupported()) {
    voiceFocusDeviceTransformer = await VoiceFocusDeviceTransformer.create({ variant: noiseSuppressionLevel });   // <----- Step (1)
    if (voiceFocusDeviceTransformer.isSupported() === false) {
        voiceFocusDeviceTransformer = null;
    }
}

if (voiceFocusDeviceTransformer === null) {
    state.meetingSession!.audioVideo.chooseAudioInputDevice(deviceId);
    throw "VoiceFocus is not supported";
} else {
    let voiceFocusTransformDevice = await voiceFocusDeviceTransformer.createTransformDevice(deviceId); // <----- Step (2)
    if (voiceFocusTransformDevice) {
        state.meetingSession!.audioVideo.chooseAudioInputDevice(voiceFocusTransformDevice); // <----- Step (3)
    } else {
        state.meetingSession!.audioVideo.chooseAudioInputDevice(deviceId);
        throw "VoiceFocus is not supported";
    }
}

Note1: The supported environment is shown in official document. Please see it.

Note2: To use Voice Focus, WebAudio should be enabled.

Demo

The background blur and noise suppression feature described in this article is implemented in a sample application for online meetings in the repository below:

https://github.com/w-okada/amazon-chime-bg-blur-demo

Note: Receiving traffic from the demos created in this post can incur AWS charges.

Prerequisites

(1) Setup AWS Credentials

We will build a backend in AWS. For this reason, it is assumed that you have set up your AWS credentials before deploying. For more details, please see here.

Procedure

(1) Clone the repository

In this article, we will use the version with the aws-demo01 branch, so please specify the aws-demo01 branch and clone.

$ git clone https://github.com/w-okada/amazon-chime-bg-blur-demo.git -b aws-demo01
$ cd xxxxxx/background-blur-demo

(2) Install npm package

After cloning the repository, you should install required npm packages.

$ npm install

(3) Run the demo

Run the command below to start the demo.

$ npm run start

(4) Open the demo application

Access https://<server_ip>:9000 or https://localhost:9000 with a browser.

$ cat demo_url.txt

When you access this page with a browser, you will enter the join meeting screen. Input the meeting name and user name and push join button. If there is no meeting room with the name you entered, a new one will be created. Set up your device in the device settings screen. The first time you start the application, you will be asked for permission to use the camera and microphone, so please allow it.

(5) Use the features

To use the feature, you change the strength of background blur or the complexity level of noise suppression. These can be changed by clicking on the menu shown in the figure below.

The figure below shows the result of applying a medium level of blur. On the left is the image before applying the blur. On the right is the result after applying the blur.

Cleanup

Cleanup is not needed for this demo.

Conclusion

In this article, we showed you how to blur background and suppress noise in an online meeting application created with the Amazon Chime SDK for JavaScript. We hope that this article will help create a better online meeting experience.

This is a guest blog written by Wataru Okada, technical researcher and engineer, R&D team at FLECT Co., Ltd (FLECT). FLECT is a cloud native integrator based in Tokyo, Japan. The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.

Business Productivity