Amazon Rekognition Documentation

Amazon Rekognition Video

Amazon Rekognition Video is a machine learning powered video analysis service that is designed to detect objects, scenes, celebrities, text, activities, and inappropriate content from your videos stored in Amazon S3. Rekognition Video is also designed to provide facial analysis and facial search capabilities to detect, analyze, and compare faces, and help understand the movement of people in your videos. Each result or detection is paired with a timestamp so that you can create an index for detailed video search, or navigate quickly to an interesting part of the video for further analysis. For objects, faces, text, and people, Rekognition Video also returns bounding box coordinates, which is the specific location of the detection in the frame.

Key features

Object, scene, and activity detection

Amazon Rekognition Video is designed to identify thousands of objects such as vehicles or pets, scenes like a city, beach, or wedding, and activities such as delivering a package or dancing. For each label detected, you get a confidence score. For common objects such as 'Person' or 'Car', you also get object bounding boxes to enable counting and object localization. Amazon Rekognition Video relies on motion in the video to accurately identify complex activities, such as “blowing out a candle” or “extinguishing fire”. Using this rich metadata, you can make your content searchable or serve advertisements that best match the context of the content preceding it.

Content moderation

Amazon Rekognition Video is designed to detect inappropriate content such as nudity, violence, or weapons in videos, and provides timestamps for each detection. You also get a hierarchical list of labels with confidence scores, describing sub-categories of unsafe content. For example, 'Graphic Female Nudity' is a sub-category of 'Explicit Nudity'. Confidence scores and detailed labels allow you to set up varied business rules to serve the compliance needs of different target segments and geographies.

Text detection

Amazon Rekognition Video is designed to detect and read text in videos, and provides the detection confidence, location bounding box, as well as the timestamp for each text detection. In addition, you get convenient options to filter out words by regions of interest (ROIs), word bounding box size, and word confidence score. For example, you may only want to detect text in the bottom third region for on-screen graphics or only the top left corner for reading scoreboards in a soccer game.

Celebrity recognition

Amazon Rekognition Video is designed to detect and recognize when and where certain well-known persons appear in a video. The time-coded output includes the name and unique id of the celebrity, and URLs pointing to related content for the celebrity, for example, the celebrity's IMDB link.

Face detection and analysis

Amazon Rekognition Video is designed to detect up to 100 faces in a video frame, and return the bounding box location. For each detected face, you can also get additional attributes such as gender, emotions, estimated age range, and whether the person is smiling, along with timestamps for each detection.

Amazon Rekognition Video is designed to identify known people in a video by searching against a private repository of face images. You get a similarity score for each match, and timestamps for each instance where the same person is identified during the video. Amazon Rekognition Video can also cluster all unknown people in a video who don’t have any matches in the repository, and return timestamps with unique identifiers for each such person.

Person pathing

Amazon Rekognition Video is designed to capture where, when and how each person is moving in your video. Amazon Rekognition also provides a unique index for each person found, allowing you to count the number of people in the video.

Live stream video analysis

Amazon Rekognition Video is designed to analyze your live video streams in real time to detect and search for faces. By providing a stream from Amazon Kinesis Video Streams as an input to Rekognition Video, you can perform face search against a repository of your own images with very low latency.

Amazon Rekognition Image

Rekognition Image is a deep learning powered image recognition service that is designed to detect objects, scenes, and faces; extract text; recognize celebrities; and identify inappropriate content in images. It also allows you to search and compare faces. The service returns a confidence score for everything it identifies so that you can make informed decisions about how you want to use the results. In addition, all detected faces are returned with bounding box coordinates, which is a rectangular frame that fully encompasses the face that can be used to locate the position of the face in the image.

Object and Scene Detection

Rekognition Image is designed to identify thousands of objects such as vehicles, pets, and furniture. Rekognition is also designed to detect scenes within an image, such as a sunset or beach. This allows you to search, filter, and curate large image libraries.

Facial Comparison

Rekognition Image lets you measure the likelihood that faces in two images are of the same person. With Rekognition, you can use the similarity score to verify a user against a reference photo in near real time. 

Rekognition Image enables you to find similar faces in a large collection of images. You can create an index of faces detected in your images. Rekognition Image’s search returns faces that best match your reference face. 

Facial Analysis

Rekognition Image is designed to locate faces within images and analyze face attributes, such as whether or not the face is smiling or the eyes are open. When analyzing an image, Rekognition Image will return the position and a rectangular frame for each detected face.

Unsafe Image Detection

Rekognition Image is designed to detect explicit and suggestive content so that you can filter images based on your application requirements. Rekognition provides a hierarchical list of labels with confidence scores to enable fine-grained control over what images you want to allow.

Celebrity Rekognition

Rekognition Image is designed to detect and recognize thousands of individuals who are famous, noteworthy, or prominent in their field. This allows you to index and search digital image libraries for celebrities based on your marketing and media needs.

Text in Image

Rekognition Image is designed to locate and extract text within images, including text in natural scenes such as road signs or license plates, text over objects such as t-shirts or mugs, and text on screen such as captions or news. When analyzing an image, Text in Image will return the detected text label, a rectangular frame, along with a confidence score, for each detected words and lines.

Personal Protective Equipment Detection

Rekognition Image is designed to detect if persons in images are wearing PPE such as face covers, hand covers, and head covers and whether the protective equipment covers the corresponding body part (nose for face covers, head for head covers, and hands for hand covers). 

Amazon Rekognition Custom Labels

With Amazon Rekognition Custom Labels, you can identify the objects and scenes in images that are specific to your business needs. For example, you can find your logo in social media posts, identify your products on store shelves, classify machine parts in an assembly line, distinguish healthy and infected plants, or detect animated characters in videos.

Developing a custom model to analyze images is a significant undertaking that requires time expertise, and resources, often taking months to complete. Additionally, it often requires thousands or tens-of-thousands of hand-labeled images to provide the model with enough data to accurately make decisions. Generating this data can take months to gather and require large teams of labelers to prepare it for use in machine learning.

With Amazon Rekognition Custom Labels, we take care of the heavy lifting for you. Rekognition Custom Labels builds off of Rekognition’s existing capabilities, which are already trained on tens of millions of images across many categories. Instead of thousands of images, you simply need to upload a small set of training images (typically a few hundred images or less) that are specific to your use case into our console. If your images are already labeled, Rekognition can begin training in just a few clicks. If not, you can label them directly within Rekognition’s labeling interface, or use Amazon SageMaker Ground Truth to label them for you. Once Rekognition begins training from your image set, it can produce a custom image analysis model for you in just a few hours. Behind the scenes, Rekognition Custom Labels automatically loads and inspects the training data, selects the right machine learning algorithms, trains a model, and provides model performance metrics. You can then use your custom model via the Rekognition Custom Labels API and integrate it into your applications.

Key features

Simplify Data Labeling

The Rekognition Custom Labels console provides a visual interface to make labeling your images faster and simpler. The interface allows you to apply a label to the entire image or to identify and label specific objects in images using bounding boxes with a simple click-and-drag interface.

Alternately, if you have a large data set, you can use Amazon SageMaker Ground Truth to label your images at scale.

Automated Machine Learning

Rekognition Custom Labels includes AutoML capabilities that take care of the machine learning for you. Once the training images are provided, Rekognition Custom Labels can automatically load and inspect the data, select the right machine learning algorithms, train a model, and provide model performance metrics.

Simplified Model Evaluation, Inference and Feedback

Evaluate your custom model’s performance on your test set. For every image in the test set, you can see the side by side comparison of the model’s prediction vs. the label assigned. You can also review detailed performance metrics such as precision/recall metrics, f-score, and confidence scores. You can start using your model immediately for image analysis, or iterate and re-train new versions with more images to improve performance. After you start using your model, you can track your predictions, correct any mistakes and use the feedback data to retrain new model versions and improve performance.

Amazon Rekognition General Information

Administration via API, Console, or Command Line

Amazon Rekognition can be accessed using the Amazon Rekognition API, AWS Management Console, and the AWS command-line interface (CLI). The console, API, and CLI provide the ability to use the Rekognition APIs to detect labels, analyze faces, compare faces, and find a face. AWS Lambda has blueprints for Rekognition that make it easy to initiate image analysis based on events in your AWS data stores such as Amazon S3 and Amazon DynamoDB.

Administrative Security

Amazon Rekognition is integrated with AWS Identity and Access Management (IAM). IAM policies can be used to control access to the Amazon Rekognition API as well as manage resource-level permissions for your account.

Human Review

Amazon Rekognition is directly integrated with Amazon Augmented AI (Amazon A2I) so you can implement human review for unsafe image detection. Amazon A2I provides built-in human review workflow for image moderation, which allows predictions from Amazon Rekognition to be reviewed and validated. With Amazon A2I, you can use a pool of reviewers within your own organization, or you can access the workforce of over 500,000 independent contractors who are already performing machine learning tasks through Amazon Mechanical Turk. You can also make use of workforce vendors that are pre-screened by AWS for quality and adherence to security procedures.

Additional Information

For additional information about service controls, security features and functionalities, including, as applicable, information about storing, retrieving, modifying, restricting, and deleting data, please see https://docs.aws.amazon.com/index.html. This additional information does not form part of the Documentation for purposes of the AWS Customer Agreement available at http://aws.amazon.com/agreement, or other agreement between you and AWS governing your use of AWS’s services.