Receive notifications for image analysis with Amazon Rekognition Custom Labels and analyze predictions

Amazon Rekognition Custom Labels is a fully managed computer vision service that allows developers to build custom models to classify and identify objects in images that are specific and unique to your business.

Rekognition Custom Labels doesn’t require you to have any prior computer vision expertise. You can get started by simply uploading tens of images instead of thousands. If the images are already labeled, you can begin training a model in just a few clicks. If not, you can label them directly within the Rekognition Custom Labels console, or use Amazon SageMaker Ground Truth to label them. Rekognition Custom Labels uses transfer learning to automatically inspect the training data, select the right model framework and algorithm, optimize the hyperparameters, and train the model. When you’re satisfied with the model accuracy, you can start hosting the trained model with just one click.

However, if you’re a business user looking to solve a computer vision problem, visualize inference results of the custom model, and receive notifications when such inference results are available, you have to rely on your engineering team to build such an application. For example, an agricultural operations manager can be notified when a crop is found to have a disease, a winemaker can be notified when the grapes are ripe for harvesting, or a store manager can be notified when it’s time to restock inventories such as soft drinks in a vertical refrigerator.

In this post, we walk you through the process of building a solution that allows you to visualize the inference result and send notifications to subscribed users when specific labels are identified in images that are processed using models built by Rekognition Custom Labels.

Solution overview

The following diagram illustrates our solution architecture.

This solution uses the following AWS services to implement a scalable and cost-effective architecture:

• Amazon Athena – A serverless interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL.
• AWS Lambda – A serverless compute service that lets you run code in response to triggers such as changes in data, shifts in system state, or user actions. Because Amazon S3 can directly trigger a Lambda function, you can build a variety of real-time serverless data-processing systems.
• Amazon QuickSight – A very fast, easy-to-use, cloud-powered business analytics service that makes it easy to build visualizations, perform ad hoc analysis, and quickly get business insights from the data.
• Amazon Rekognition Custom Labels – Allows you to train a custom computer vision model to identify the objects and scenes in images that are specific to your business needs.
• Amazon Simple Notification Service – Amazon SNS is a fully managed messaging service for both application-to-application (A2A) and application-to-person (A2P) communication.
• Amazon Simple Queue Service – Amazon SQS is a fully managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications.
• Amazon Simple Storage Service – Amazon S3 serves as an object store for your documents and allows for central management with fine-tuned access controls.

The solution utilizes a serverless workflow that gets triggered when an image is uploaded to the input S3 bucket. An SQS queue receives an event notification for object creation. The solution also creates dead-letter queues (DLQs) to set aside and isolate messages that can’t be processed correctly. A Lambda function feeds off of the SQS queue and makes the DetectLabels API call to detect all labels in the image. To scale this solution and make it a loosely coupled design, the Lambda function sends the prediction results to another SQS queue. This SQS queue triggers another Lambda function, which analyzes all the labels found in the predictions. Based on the user preference (configured during solution deployment), the function publishes a message to an SNS topic. The SNS topic is configured to deliver email notifications to the user. You can configure the Lambda function to add a URL to the message sent to Amazon SNS to access the image (using an Amazon S3 presigned URL). Finally, the Lambda function uploads a prediction result and image metadata to an S3 bucket. You can then use Athena and QuickSight to analyze and visualize the results from the S3 bucket.

Prerequisites

You need to have a model trained and running with Rekognition Custom Labels.

Rekognition Custom Labels lets you manage the machine learning model training process on the Amazon Rekognition console, which simplifies the end-to-end model development process. For this post, we use a classification model trained to detect plant leaf disease.

Deploy the solution

You deploy an AWS CloudFormation template to provision the necessary resources, including S3 buckets, SQS queues, SNS topic, Lambda functions, and AWS Identity and Access Management (IAM) roles. The template creates the stack the us-east-1 Region, but you can use the template to create your stack in any Region where the above AWS services are available.

1. Launch the following CloudFormation template in the Region and AWS account where you deployed the Rekognition Custom Labels model:

2. For Stack name, enter a stack name, such as rekognition-customlabels-analytics-and-notification.
3. For CustomModelARN, enter the ARN of the Amazon Rekognition Custom Labels model that you want to use.

The Rekognition Custom Labels model needs to be deployed in the same AWS account.

4. For EmailNotification, enter an email address where you want to receive notifications.
5. For InputBucketName, enter a unique name for the S3 bucket the stack creates; for example, plant-leaf-disease-data-input.

This is where the incoming plant leaf images are stored.

6. For LabelsofInterest, you can enter up to 10 different labels you want to be notified of, in comma-separated format. For our plant disease example, enter bacterial-leaf-blight,leaf-smut.
7. For MinConfidence, enter the minimum confidence threshold to receive notification. Labels detected with a confidence below the value of MinConfidence aren’t returned in the response and will not generate notification.
8. For OutputBucketName, enter a unique name for the S3 bucket the stack creates; for example, plant-leaf-disease-data-output.

The output bucket contains JSON files with image metadata (labels found and confidence score).

9. Choose Next.

10. On the Configure stack options page, set any additional parameters for the stack, including tags.
11. Choose Next.
12. In the Capabilities and transforms section, select the check box to acknowledge that AWS CloudFormation might create IAM resources.
13. Choose Create stack.

The stack details page should show the status of the stack as CREATE_IN_PROGRESS. It can take up to 5 minutes for the status to change to CREATE_COMPLETE.

Amazon SNS will send a subscription confirmation message to the email address. You need to confirm the subscription.

Test the solution

Now that we have deployed the resources, we’re ready to test the solution. Make sure you start the model.

1. On the Amazon S3 console, choose Buckets.
2. Choose the input S3 bucket.

3. Upload test images to the bucket.

In production, you can set up automated processes to deliver images to this bucket.

These images trigger the workflow. If the label confidence exceeds the specified threshold, you receive an email notification like the following.

You can also configure the SNS topic to deliver these notifications to any destinations supported by the service.

Analyze the prediction results

After you test the solution, you can extend the solution to create a visual analysis for the predictions of processed images. For this purpose, we use Athena, an interactive query service that makes it easy to analyze data directly from Amazon S3 using standard SQL, and QuickSight to visualize the data.

Configure Athena

If you are not familiar with Amazon Athena, see this tutorial. On the Athena console, create a table in the Athena data catalog with the following code:

CREATE EXTERNAL TABLE IF NOT EXISTS `default`.`rekognition_customlabels_analytics` (
`Image` string,
`Label` string,
`Confidence` string
)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
WITH SERDEPROPERTIES (
'serialization.format' = '1'
) LOCATION 's3://<<OUTPUT BUCKET NAME>>/'
TBLPROPERTIES ('has_encrypted_data'='false');

Populate the Location field in the preceding query with your output bucket name, such as plant-leaf-disease-data-output.

This code tells Athena how to interpret each row of the text in the S3 bucket.

You can now query the data:

SELECT * FROM "default"."rekognition_customlabels_analytics" limit 10;

Configure QuickSight

To configure QuickSight, complete the following steps:

1. Open the QuickSight console.
2. If you’re not signed up for QuickSight, you’re prompted with the option to sign up. Follow the steps to sign up to use QuickSight.
3. After you log in to QuickSight, choose Manage QuickSight under your account.

4. In the navigation pane, choose Security & permissions.
5. Under QuickSight access to AWS services, choose Add or remove.

A page appears for enabling QuickSight access to AWS services.

6. Select Amazon Athena.

7. In the pop-up window, choose Next.

8. On the S3 tab, select the necessary S3 buckets. For this post, I select the bucket that stores my Athena query results.
9. For each bucket, also select Write permission for Athena Workgroup.
10. Choose Finish.
11. Choose Update.
12. On the QuickSight console, choose New analysis.
13. Choose New dataset.
14. For Datasets, choose Athena.
15. For Data source name, enter Athena-CustomLabels-analysis.
16. For Athena workgroup, choose primary.
17. Choose Create data source.

18. For Database, choose default on the drop-down menu.
19. For Tables, select the table rekognition_customlabels_analytics.
20. Choose Select.

21. Choose Visualize.

22. On the Visualize page, under the Fields list, choose label and select the pie chart from Visual types.

You can add more visualizations in the dashboard. When your analysis is ready, you can choose Share to create a dashboard and share it within your organization.

Summary

In this post, we showed how you can create a solution to receive notifications for specific labels (such as bacterial leaf blight or leaf smut) found in processed images using Rekognition Custom Labels. In addition, we showed how you can create dashboards to visualize the results using Athena and QuickSight.

You can now easily share such visualization dashboards with business users and allow them to subscribe to notifications instead of having to rely on your engineering teams to build such an application.

About the Authors

Jay Rao is a Principal Solutions Architect at AWS. He enjoys providing technical and strategic guidance to customers and helping them design and implement solutions on AWS.

Pashmeen Mistry is the Senior Product Manager for Amazon Rekognition Custom Labels. Outside of work, Pashmeen enjoys adventurous hikes, photography, and spending time with his family.