Analyze content using AWS Elemental MediaLive thumbnails with Amazon Rekognition

Amazon Web Services (AWS) recently launched a Thumbnail Image Preview feature as part of the AWS Elemental MediaLive service. This new feature not only provides a mechanism for quality control of live content, but also enables integration with media analysis services, like Amazon Rekognition, to automate content review and validate workflows.

MediaLive automatically generates thumbnails every two seconds. Thumbnails can be viewed within the running channel via the MediaLive console. You can also retrieve thumbnails programmatically via the AWS CLI, API, and SDK. Amazon Rekognition makes it easy to add image and video analysis to your applications. You can perform image analysis to detect objects, concepts, people, scenes, and detect inappropriate content. By using Amazon Rekognition to evaluate the MediaLive thumbnail preview, you can perform content moderation at scale on both organizational and user-generated content (UGC). For example, organizations that integrate MediaLive services into their live streaming offerings with Rekognition can now more easily safeguard use of their offerings to reduce risk of liability and brand damage.

In this blog post, we discuss how to integrate the artificial intelligence (AI) and machine learning (ML) capabilities of Amazon Rekognition for automatic detection of unauthorized content streaming via thumbnail previews generated by active AWS Elemental MediaLive channels. When unauthorized content is detected, an automated response can be triggered to alert or stop the offending channel. The example provided makes use of the DetectLabels API, which detects entities, activities, and events, and provides a response that includes not only what’s detected, but also a confidence level of what’s been detected. While we discuss how a company may use a specific API to detect unauthorized streaming of content, another API, such as the DetectModerationLabels, can easily be added or substituted to better fit your use case.

Solution overview

The following diagram provides a high-level architecture of the solution.

The solution uses OBS Studio as the input source for the MediaLive channel, and the MediaLive channel must be in a started state.

The following steps are required for the solution.

1. An Amazon EventBridge Scheduler is a serverless scheduler that makes it simple to manage scheduled tasks at scale. The schedule runs at a specified rate and delivers information, similar to the following object, for a specific MediaLive channel to an AWS Lambda.

a. Note: As opposed to hard-coding MediaLive channel values in the Lambda code, sending the channel information in a payload from the Scheduler reduces complexity by enabling easy reuse of the compute resources.

<pre><code class="lang-json">

{
  "action": "create",
  "schedulerName": "EveryHourFrom9To5CSTWeekdays",
  "cron": "0 9-17 ? * MON-FRI *",
  "timeZone": "America/Chicago",
  "channelInfoPayload": {
    "AWS_REGION": "us-west-2",
    "ChannelId": "3284674",
    "PipelineId": "0",
    "ThumbnailType": "CURRENT_ACTIVE"
  }
}

</code></pre> 

RunWeekdaysEveryHourFrom9To5CST.json

2. The Lambda function receives the payload as event object, parameterizes its properties, and calls the describe Thumbnails method for the MediaLive Channel specified in the params object.

<pre><code class="language-typescript">
 // Parameterized properties of the event object: 
  const { AWS_REGION, ChannelId, PipelineId, ThumbnailType } = event;
  AWS.config.update({ region: AWS_REGION });

  //Create the parameter object for the MediaLive API call
  const params = {
    ChannelId,
    PipelineId,
    ThumbnailType
  };
  
  try {

    let response: object | undefined

    // 1. Call MediaLive describeThumbnails() api to retrieve the latest thumbnail 
    const data: any = await describeThumbnails(params);

</code></pre> 

Location: /aws-medialive-thumbnail-amazon-rekognition-analysis/lib/lambda-functions/MediaLiveThumbnailDetector/index.ts

3. The image is decoded into binary data. In our example, the decoded image is stored as a variable in ephemeral storage, but optionally can be archived to long-term storage such as Amazon S3.

<pre><code class="language-typescript">
    //2. Use the built-in Buffer class to decode the binary data into an image
    const decodedImage = Buffer.from(thumbnailBody, 'base64');
    // Optionally archive image in S3 bucket 
  
</code></pre>

Location: /aws-medialive-thumbnail-amazon-rekognition-analysis/lib/lambda-functions/MediaLiveThumbnailDetector/index.ts

4. The decoded image is included as a parameter of the last function call within the Lambda handler, detectUnauthorizedContent, which will call the Rekognition service.

<pre><code class="language-typescript">
    //3. Detect sport events using rekognition
      response = await detectUnauthorizedContent(decodedImage, ChannelId);
</code></pre>

Location: /aws-medialive-thumbnail-amazon-rekognition-analysis/lib/lambda-functions/MediaLiveThumbnailDetector/index.ts

5. In detectUnauthorizedContent function, the following methods are executed.

a. The Rekognition DetectLabels function is called and results returned.

b. The returned results are evaluated against three conditions. First, ensuring labels were in fact returned. Second, comparing the returned label against the unauthorized content we are looking for, in this case ‘Sport’. And finally verifying that the confidence score for this returned content is greater than 90%. A parameter could be used to replace ‘Sport’ in the code in order for this solution to be more flexible.

c. If the returned label and confidence score determine that unauthorized content is true, the sendSnsMessage function is called.

Amazon Simple Notification Service (Amazon SNS) allows you to send messages between decoupled systems, such as IT support systems, monitoring systems, IT automation systems such as AWS Systems Manager, or communication systems such as Slack, or email. In the case of the example application, an email address is required as a deployment parameter. The specified email will receive an Amazon SNS Subscription Confirmation email upon successful deployment. Be sure to click the confirmation link in the email to receive notifications from the solution when a ‘Sport’ event is detected.

d. Finally, if a label matching the condition is not found, labels that are returned are logged, along with their corresponding confidence scores to the Amazon CloudWatch log associated with the Lambda function.

<pre><code class="language-typescript">
// Function to detect sporting events using rekognition
async function detectUnauthorizedContent(imageBuffer: Buffer, channelId: string): Promise<{ statusCode: number; body: string; } | undefined> {
  try {
    const params = {
      Image: {
        Bytes: imageBuffer
      },
      MaxLabels: 10,
      MinConfidence: 70
    };
    const response = await rekognition.detectLabels(params).promise();
    const labels = response.Labels;

    if (labels) {
      let unauthorizedContent = false;
      let unauthorizedContentConfidenceScore: number
      for (let i = 0; i < labels.length; i++) {
        if (labels[i].Name === 'Sport' && labels[i].Confidence > 90) {
          unauthorizedContent = true;
          unauthorizedContentConfidenceScore = labels[i].Confidence
          break;
        }
      }

      if (unauthorizedContent) {
        console.log("Sporting Event Streaming Detected");
        const snsResponse = await sendSnsMessage(channelId, `Sporting event streaming detected with a ${unauthorizedContentConfidenceScore!} confidence level.`); 
        const response = {
            statusCode: 200,
            body: JSON.stringify({
                channelId,
                Message: snsResponse, 
            }),
        };
        return response

      } else {
        console.log('No sporting events detected. Detected labels are: ', labels.map((label: any) => `${label.Name}: ConfidenceLevel=${label.Confidence}`).join(', '));
        const response = {
            statusCode: 200,
            body: JSON.stringify({
                channelId,
                Message: 'No sporting events detected. Detected labels are: ',
                Labels: labels.map((label: any) => `${label.Name}: ConfidenceLevel=${label.Confidence}`).join(', ')
            }),
        };
        return response
      }
    }
  } catch (error) {
    console.log('Error occurred while detecting labels', error);
  }
}
</code></pre>

Location: /aws-medialive-thumbnail-amazon-rekognition-analysis/lib/lambda-functions/MediaLiveThumbnailDetector/index.ts

The solution monitors your content in near-real time and sends an alert based on the content you have identified as being unauthorized.

Setup and Installation

Prerequisites:

1. An AWS Account
2. An active AWS Elemental MediaLive Channel with the Thumbnail feature enabled. Visit this blog for quick start in streaming live content from your laptop with OBS Studio and AWS Media Services.
3. Install AWS CDK version 2.73.0 or later

We implemented the solution as an AWS CDK app in Typescript. You can specify the configuration parameters in the CDK context. Installing and setting up this solution in your AWS account only requires a few simple steps.

Verify that you have permissions to create a CloudFormation stack and the resources defined in this solution. Also, make sure to install the CDK Toolkit, Python 3, and the Docker CLI.

Next, clone the repository to your local file system

$ git clone https://github.com/aws-samples/aws-medialive-thumbnail-amazon-rekognition-analysis

Change Directory to CDK solution and Install

$ cd .\aws-medialive-thumbnail-amazon-rekognition-analysis\

$ npm install

Gather information for the parameters to deploy the solution

snsEmail: Email which will receive notification alerts

channelId: Channel Id of an active MediaLive channel in your AWS account.

pipelineId: Standard channels typical have two pipelines for input

Deploy the solution into your AWS Account:

$ cdk deploy --parameters snsEmail=myemail@example.com --parameters channelId=1234567  --parameters pipelineId=0

After the deployment completes, you can verify the solution is working by playing unauthorized content on your channel. In this example we are looking for sports content so you can play sports-related content on your channel to trigger the workflow.

You can easily adapt this solution to your specific requirements. To change the Rekognition labels being evaluated, you can modify the Lambda function call ‘MediaLiveThumbnailDetector’ under the /lib folder.

Cleaning Up

Delete all resources created by the CDK Project by running the following commands in the project terminal:

• cdk destroy
• type ‘y’ to confirm deletion

Make sure to run the terminal commands labeled with delete scheduler: text for each additional scheduler created via the AWS CLI and AWS Lambda as described here.

Alternative Solution

While this blog focuses on using AWS Elemental MediaLive thumbnails, there is an alternative solution that uses the MediaLive frame capture output group. Frame capture output groups allow for additional input controls like image resolution and output interval. It also saves the images as JPG to your defined Amazon S3 bucket.

Summary

In this post we demonstrated how Amazon Rekognition machine learning can be used to analyze an AWS Elemental MediaLive thumbnail. This analysis can trigger a notification if unauthorized content is detected on your channel. More can be accomplished with this solution. For instance, you can add automation to stop the channel automatically when unauthorized content is found, in addition to sending the notification. This solution can also be integrated into your existing channel automation code.

For more information on accessing and using thumbnails, as well as exploring other features within AWS Elemental MediaLive, please refer to the AWS Elemental MediaLive documentation.