Using Shutterstock’s image datasets to train your computer vision models

Image classification and object detection technology allows you to build scalable artificial intelligence models for business cases like visual search, product recommendations, autonomous vehicle object recognition, content moderation, and more. Today, services like Amazon Rekognition offer APIs to perform image analysis and object recognition. However, if your use case requires a more custom image classification model, you must have access to an image dataset that not only fits your custom use case, but also contains enough image samples to accurately train your machine learning model.

In this blog post, I show you how to gain access to and train computer vision models on custom image datasets using Shutterstock Image Datasets in AWS Marketplace. These datasets are collections of curated images from Shutterstock’s library of more than 370 million images. You can subscribe to one of the preexisting collections, such as Food & Beverage, Clothing, or Hospitality, or you can work with the Shutterstock Data Exchange team to request a custom collection of images for your use case. Each image includes a descriptive title with up to 200 characters and an optimal 7-50 keywords.

For this post, I use the Free Sample: Images & Metadata of “Whole Foods” Shoppers dataset from Shutterstock’s Image Datasets to demonstrate how to train a multi-label image classification model using Shutterstock’s pre-labeled image assets. This dataset contains images of Whole Foods shoppers. Each image is tagged with 7-50 keywords describing what is seen in the image. The first image shows a male store employee behind a deli counter selecting an item for a female customer holding a shopping basket. The second image shows a couple choosing fresh produce. The third image shows a woman with a shopping cart holding a head of lettuce.

Photo by George Rudy | Shutterstock

Photo by LightField Studios | Shutterstock

Photo by PR Image Factory | Shutterstock

I will build my image classification model by training the Amazon SageMaker (SageMaker) image classification algorithm. This algorithm is a supervised learning algorithm that supports multi-label classification. It takes an image as input and outputs one or more labels assigned to that image. It uses a convolutional neural network (ResNet) that can be trained from scratch or trained using transfer learning when a large number of training images are not available.

Solution overview

The following architecture diagram outlines the components of this solution. It consists of these components:

• The Shutterstock Free Sample: Images & Metadata of “Whole Foods” Shoppers dataset to demonstrate an example of multi-label image classification using free content available from the Shutterstock Image Datasets.
• An Amazon S3 bucket to store our image training dataset.
• A SageMaker notebook to write code that will help build, train, and evaluate our machine learning model.
• A SageMaker model endpoint, where our machine learning model is available to provide inferences. The persistent model endpoint provides an HTTPS endpoint to get one prediction at a time from our model using SageMaker hosting services. Alternatively, you can use SageMaker batch transforms to get predictions for an entire dataset.

In the following architecture diagram, I export the images from AWS Data Exchange to an S3 bucket. Then I use a SageMaker notebook to prepare my image data, train my image classification model, and then deploy a model endpoint. Once this is complete, I evaluate our model on a test image to retrieve the recommended image labels and associated confidence scores for each label.

Prerequisites

To run this solution, you must have the following prerequisites:

• An AWS account – if you don’t already have one, create one.

Solution walkthrough

Step 1: Export the Shutterstock Whole Foods image dataset to an S3 bucket

To start working with your dataset, you must subscribe to the dataset and then export the data to an S3 bucket.

1. If you don’t already have an S3 bucket to hold your image dataset, navigate to the S3 console and select Create bucket. Take note that the Shutterstock Image Datasets exist in US East (Ohio). To avoid cross-Region data transfer charges, we recommend choosing US East (Ohio) us-east-2 as the region for your bucket and SageMaker environment.
2. Subscribe to Shutterstock’s Free Sample: Images & Metadata of “Whole Foods” Shoppers dataset by subscribing in AWS Marketplace. To do that, follow this link and choose Continue to subscribe. Follow the subscription wizard.
3. Navigate to the AWS Data Exchange Console. In the navigation pane under the My subscriptions dropdown, select Entitled Data.
4. From your Entitled data, expand the Free Data Set: ‘Whole Foods’ Shoppers by choosing the arrow to the left of it. If you are unable to see this dataset, make sure that you are in the Ohio (us-east-2) Region.
5. Scroll down to view the Revisions of this dataset. There should be two revisions: one with a comment “Metadata” and another with a comment “Images.” Select each revision and select Export to Amazon S3. In the pop-up window, enter the name of the S3 bucket that you created for this dataset, and select Export.

The progress of your export job appears in the Jobs table list at the bottom of the page. When the job state appears as Completed, you’re ready to move on to the next step.

Step 2: Train, test, and export your model by using SageMaker notebooks

I am using an Amazon SageMaker notebook instance to train and test my model. SageMaker manages the creation of the instance and related resources. To train and test your model, do the following:

1. Open the Amazon SageMaker console.
2. In the navigation pane, under Notebook, choose Notebook instances and select Create notebook instance.
3. Assign a notebook instance name, and under Notebook instance type, select t2.medium.
4. In the Permissions and encryption section, select the dropdown under IAM role and select Create a new role. In the Create an IAM role pop-up, under S3 buckets you specify – optional, select Specific S3 buckets and enter the name of the S3 bucket that is being used to store your image dataset.
5. In the Git repositories section, set your Repository as Clone a public Git repository to this notebook instance only and enter the following URL into Git repository URL: https://github.com/aws-samples/aws-data-exchange-shutterstock-image-datasets.git. Select Create notebook instance.
6. The notebook will take a few minutes to start. When your notebook status shows InService, select Open Jupyter from the notebook actions. When the Jupyter environment loads, the notebook file image-classification-with-shutterstock-datasets.ipynb that contains the classification code appears. To open the notebook, choose the file.
7. Follow the steps in the notebook to complete the creation of your image classification model. In Step 1 of the notebook, update the images_bucket and images_bucket_prefix variables to point to the location of your Shutterstock image dataset.

The SageMaker notebook walks you through how to ingest the Whole Foods Shutterstock image dataset and metadata, use these components to train a model, and evaluate the performance of your model. On the last step, you can test on various images in the dataset to reveal the probabilities that each label is associated with an image. At the end of the notebook, you’ll test your model on the following sample image depicting a figure in a mask standing next to a shopping cart. Following is an example of the returned label probabilities that you will see: “adult” (32.44%), “beautiful” (32.84%), “buying” (65.97%), “cart” (93.60%), and “casual” (43.79%).

Photo (used to test image classification model) By Thomas Andreas | Shutterstock

These steps provide an example of how to get started building a computer vision model using Shutterstock Image Datasets. When you have the results of the first version of your image classification model, you can improve the accuracy of our model by training on additional labels, increasing the size of the training set, or modifying the model’s hyperparameters.

Cleaning up

After you have completed these steps and are no longer using the resources created, stop and clean up the resources in this blog post so as to not incur any additional charges. To remove these resources:

• Delete the SageMaker Model Endpoint and Notebook Instance.
• Delete the S3 buckets and S3 bucket contents, where applicable. This includes the bucket that held the sample image dataset as well as the contents of the default SageMaker bucket used for the training and model data.

Conclusion

In this post, I showed how to use the Shutterstock Whole Foods sample dataset from AWS Data Exchange to train a multi-label image classification model in Amazon SageMaker.

To get started using Shutterstock Image Datasets to train your own models for object detection, product categorization, visual search, and more, subscribe to this AWS Data Exchange offering on AWS Marketplace.

For support with building more complex models or creating custom datasets using the Shutterstock 370+ million image library, contact the Shutterstock team directly at SalesADX@shutterstock.com.

About the author

Katie Williams is an AWS Solutions Architect based in New York City. She works with Media & Entertainment customers in the Digital Native Business segment to help them design, deploy, and scale out their applications in the cloud.