AWS Machine Learning Blog

Easily perform facial analysis on live feeds by creating a serverless video analytics environment using Amazon Rekognition Video and Amazon Kinesis Video Streams

Video often gets captured and saved, but it isn’t analyzed until someone has the time to sit down and watch the video looking for key people, places, or things.  What if you could streamline and automate the process of analyzing video, using an easy-to-use service powered by deep learning?

Amazon Rekognition Video is a deep-learning-powered video analysis service that tracks people, detects activities, and recognizes objects, celebrities, and inappropriate content. Amazon Rekognition Video can detect and recognize faces in live streams. Rekognition Video analyzes existing video stored in Amazon S3 and returns specific labels of activities, people, faces, and objects with time stamps, so you can easily locate the scene. It can also perform facial recognition on live video from Amazon Kinesis Video Steams. Using Amazon Kinesis Video Streams you can securely stream video from connected devices to AWS for analytics, machine learning (ML), and other processing.

In this blog post, we show you how you can test the facial recognition feature yourself. This feature allows you to also identify if the faces in the video match a particular collection of known individuals from a live video feed. These could be VIPs, persons of interest, known folks in a company or organization, or any other type of collection that makes sense for your use case.

Serverless architecture overview

The following diagram shows the video analysis flow that we demonstrate in this blog post. We use a single face in the collection, but this could easily be expanded to a collection of millions of faces.

In this blog post, we’ll use your webcam on your laptop to send a live feed to an Amazon Kinesis Video Stream. From there, a processor within Amazon Rekognition Video analyzes the feed and compares it to a collection we create.  The output matches will get sent to us via an email through an integration with AWS Lambda and Amazon Simple Notification Service (Amazon SNS).

Understanding the results

Next, we’ll go over the results from  Amazon Rekognition Video when a face is identified in the live video stream. This result is sent to our Lambda function in order to send the notification through Amazon SNS when either a known or unknown face shows up in the webcam feed.

The Rekognition Video stream processor will output messages to the Kinesis Data Stream for each video stream fragment. We’ll step through each part of this output next.

InputInformation: This includes the information on which video stream had the face on it, the fragment number from the live stream, the timestamps for both the producer of the video and the server, as well as the offset in seconds.

StreamProcessingInformation: Status of the Rekognition Stream processor, which is essentially is always RUNNING while reading and outputting results.

FaceSearchResponse: This section contains the list of faces found in the live video feed. Each face found in the feed includes metadata for the bounding boxes, confidence scores, landmark locations, as well as which faces it matched (if any) from the faces collection used when starting the stream processor.

In the following example, the segments contained faces, but none were a match from our face collection.

Here is a separate example where the segment had two faces identified, but only one of them matched the collection:

In the previous example, if we examine the MatchedFaces information, we see the external image id (“Snively” in my example) as well as the matched confidence score that determined that the face was a match.

Here is an example:

Setting up the environment

Note: The following steps assume that you already have the AWS CLI configured on your machine. If you don’t, see detailed instructions here: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html.

It’s easy for you to set this up yourself. Just follow a few steps that get your environment running and recognizing videos from the webcam of your computer.

There is an AWS CloudFormation stack that will set up much of this video analysis stream / solution from this blog post up for you. The stack will create the components that are included in the dotted section in the following image.

After you launch the CloudFormation script, we’ll step you through setting up the rest of the cloud services as well as turning your webcam into a video producer for Kinesis Video Streams.

  1. Using the following link will automatically launch a new CloudFormation Stack in us-east-1: CloudFormation Stack.

  1. Select Next:

  1. Enter your email address (used for the SNS notification) and choose Next:

  1. Choose Next.
  2. Select the check box and choose Create.

  1. Wait until the CloudFormation Stack is created.

When the status says CREATE_COMPLETE, this means it’s done.

  1. Now, using the AWS CLI on the command line, let’s create a face collection that we’ll use for the blog.

aws rekognition create-collection --collection-id rekVideoBlog --region us-west-2

  1. Now, let’s add a face to it (please use a picture of yourself).

First, upload a picture of yourself to an Amazon S3 bucket in the us-west-2 Region. Then replace the S3BUCKET, MYFACE_KEY with the bucket and location within the bucket from which it was uploaded. For YOURNAME, type your name, and it will be used as the external image ID when it finds your face.

aws rekognition index-faces --image '{"S3Object":{"Bucket":"<S3BUCKET>","Name":"<MYFACE_KEY>.jpeg"}}' --collection-id "rekVideoBlog" --detection-attributes "ALL" --external-image-id "<YOURNAME>" --region us-west-2

You should see a FaceDetail record get returned.  This is the beginning of an example:

We now have a face in our collection that we will use to be notified whenever the face shows up in the live camera feed. Of course, millions of faces could be added to the collection.

  1. Now, let’s create the Kinesis Video Stream for our webcam to connect to.

a. Open the Kinesis Video stream console and launch into the us-west-2 Region:

https://us-west-2.console.aws.amazon.com/kinesisvideo/streams?region=us-west-2

b. Choose Create.

c.

c. Give it the name, “LiveRekognitionVideoAnalysisBlog.”

d. Leave the defaults and create your video stream.

Creating the video processor

The stream processor contains information about the Kinesis data stream and the Kinesis video stream. It also contains the identifier for the collection that contains the faces you want to recognize in the input streaming video.

The following is an example of where you would find ARN_VIDEO_STREAM_YOU_CREATED:

Kinesis Video Stream ARN:

Example of where you find the Outputs from the CloudFormation script:

  1. Create a JSON file that contains the following information:
{
       "Name": "streamProcessorForBlog",
       "Input": {
              "KinesisVideoStream": {
                     "Arn": "<ARN_VIDEO_STREAM_YOU_CREATED>"
              }
       },
       "Output": {
              "KinesisDataStream": {
                     "Arn": "<KINESIS_DATA_STREAM IN_CLOUDFORMATION_OUTPUT>"
              }
       },
       "RoleArn": "<IAM_ROLE_ARN_IN_CLOUDFORMATION_OUTPUT>",
       "Settings": {
              "FaceSearch": {
                     "CollectionId": "rekVideoBlog",
                     "FaceMatchThreshold": 85.5
              }
       }
}

 

  1. Run the CLI command to create the stream processor:

aws rekognition create-stream-processor --region us-west-2 --cli-input-json file://<PATH_TO_JSON_FILE_ABOVE>

  1. Now let’s start the stream processor:

aws rekognition start-stream-processor --name streamProcessorForBlog --region us-west-2

You’ll notice in this command that the stream name matches the name from the JSON file we used to create the stream processor.

  1. You can see if the stream processor is in a running state by doing a list:

aws rekognition list-stream-processors --region us-west-2

It’s still starting if you see this:

{
    "StreamProcessors": [
        {
            "Status": "STARTING", 
            "Name": "streamProcessorForBlog"
        }
    ]
}

This means it’s running:

{
    "StreamProcessors": [
        {
            "Status": "RUNNING", 
            "Name": "streamProcessorForBlog"
        }
    ]
}

Configuring the live feed, your webcam

This blog uses the standard Amazon Kinesis Video Streams Producer SDK from the AWS Labs GitHub repository located here:

https://github.com/awslabs/amazon-kinesis-video-streams-producer-sdk-cpp#building-from-source

The repository includes various ways of writing to a Kinesis Video Stream. These include native SDKs, a gstreamer application that uses the webcam on your Mac or PC, a Raspberry Pi project, and other methods.

We’ll be building the native section of the repository from the source, but we’ll leverage the gstreamer application that uses the webcam for this blog post. You don’t need to use a Raspberry Pi or other methods in order to run this.

First, make sure that all of the prerequisites are met when building the native project. After you build the native project, set the following environment variables:

NOTE: The AWS variable names must be capitalized as noted below:

export AWS_ACCESS_KEY_ID=<FILL_IN>

export AWS_SECRET_ACCESS_KEY=<FILL_IN>

And run this command:

./kinesis_video_gstreamer_sample_app LiveRekognitionVideoAnalysisBlog

NOTE: The default Region is set to us-west-2. If you are using a different Region than the one that we use in this blog, you’ll need to set the Region variable too.

Confirming the SNS subscription

When you launch the CloudFormation script, it will automatically use the email address you entered as part of the SNS subscription. You should see a message like this to confirm:

Get notified whenever known and unknown persons show up in your webcam.

Here is an example:

Cleanup

Running the following cleanup will stop the Rekognition stream processor an remove any components that were created during this blog post:

  1. Stop and Delete the stream processor.

aws rekognition stop-stream-processor --name streamProcessorForBlog --region us-west-2

aws rekognition delete-stream-processor --name streamProcessorForBlog --region us-west-2

  1. Delete the Rekognition collection.

aws rekognition delete-collection --collection-id rekVideoBlog --region us-west-2

  1. Delete the Kinesis Video Stream

Go to the Kinesis Video Console.

Select the ‘LiveRekognitionVideoAnalysisBlog’ video stream and choose Delete.

  1. Delete the CloudFormation stack.

Go to the CloudFormation console and select RekognitionVideoBlog. Choose the Actions button, then Delete Stack, and confirm.

Conclusion

Without spinning up a single server, we set up a real-time live video feed, facial analysis, and notification system. Amazon Kinesis Video Streams makes it easy to securely stream video from connected devices to AWS for analytics, machine learning (ML), and other processing.

About the Author

Ben Snively is an AWS Public Sector Specialist Solutions Architect. He works with government, non-profit, and education customers on big data/analytical and AI/ML projects, helping them build solutions using AWS.