Front-End Web & Mobile

Using Amazon Rekognition to detect celebrities in an iOS app

This article was written by Prashant Pawan Pisipati, Sr. Product Manager, Alexa.

Picture yourself watching a movie or TV series. You see an actor on screen that looks very familiar, but you just can’t seem to remember their name or where you’ve seen them before. Today, if you’re watching an Amazon Prime Video, you get a neat feature called X-Ray, which identifies the actors on screen and provides handy links to their IMDb profiles. What if you could build similar functionality in your own mobile app and point your phone at whatever you’re watching to detect celebrities?

In this walkthrough, we build a sample iOS Swift app that lets you snap a photo with your phone camera, and uses Amazon Rekognition to identify faces and celebrities in the picture, with links to additional information about the identified celebrity. Amazon Rekognition is an AWS service that enables you to easily add intelligent image and video analysis to your applications.

The app we’re building in this walkthrough showcases features of the Amazon Rekognition API—such as celebrity detection, result confidence, face detection, and face boundaries. This solution is an iOS app that’s written in Swift, but you can easily extend the underlying concepts to other mobile platforms. The final solution app code (san your credentials) is available from this GitHub repository.

Here’s what the completed iOS app looks like in action.

Step 1: Set up permissions for calling the Amazon Rekognition API

Create an Amazon Cognito identity pool

  1. Go to the Amazon Cognito console, go to Manage Identity Pools, and then choose Create new Identity Pool.
  2. Enter a name that’s recognizable later, such as “My Rekognition App”.
  3. Select the check box for Enable access to unauthenticated identities, so that you don’t need to sign in for this demo app.
  4. Choose Create Pool, which takes you to the Your Cognito identities require access to your resources page for assigning a role to your identity pool.

Set up an IAM role and policy

The next step is to create a role and attach an IAM policy to this role that gives access to the Amazon Rekognition API.

To do this, you edit the IAM policy on the Your Cognito identities require access to your resources page. Choose View Details, choose View Policy Document, and then choose Edit. You do this for the unauth role, but you can also do the same thing for the auth role if you’re going to use the app as an authenticated user.

You can also use a readily available policy template for access to the Amazon Rekognition API.

  1. To do this, choose Allow on the Your Cognito identities require access to your resources page and head over to the IAM console.
  2. In the IAM console, go to the Roles section, and find your unauth role from the previous step.
  3. Choose the role link to go to the role’s page.
  4. Under the Permissions tab, choose Attach policies, and search for a policy titled AmazonRekognitionReadOnly. Select the check box next to this policy, and choose Attach policy.

Step 2: Download the starter app (iOS client)

The starter app comes with the basic code for capturing a photo and displaying an image. This allows you to focus on the components that are related to Amazon Rekognition straight away.

To begin, download or clone the iOS celebrity app from this GitHub repository.

Quick overview of the starter iOS app

The starter app includes the basic plumbing for capturing a picture using the device camera, or selecting a photo from the device photo library. It also includes components that can load a webpage inside a Safari-based webview.

Also, the starter app includes a Celebrity.swift class that’s used to create a Celebrity object for each ‘face’ that’s identified in the provided image. This object uses results from the Amazon Rekognition API to populate its properties and identify the location of each identified ‘face’ in the image. In particular, it uses the bounding box data for each face that’s identified in the image, and draws a button at the position of the face. Choosing the button takes you to a webpage for the celebrity that’s loaded within a Safari web view. 

Step 3: Set up the AWS Mobile SDK

1. We use CocoaPods to set up the AWS Mobile SDK for iOS dependencies. Go to the pod file in the project directory and enter the following under #Pods for <project name>: pod ‘AWSRekognition’

pod 'AWSRekognition'

2. Open the Terminal app, and then change directory to the location of the unpacked files from the previous step.

3. Run the following:

pod install --repo-update

4. If you go to the Pods folder in your project, you should now see the pods AWSCore and AWSRekognition there. These are the AWS Mobile SDK components that enable you to talk directly to Amazon Rekognition. You can also see that a new workspace file was created for this project.

5. Open the project workspace in Xcode and Build the project.

Step 4: Set up credentials for the AWS Mobile SDK

Having set up the Amazon Cognito identity pool with the required role and policy to access the Amazon Rekognition API, we now provide these credentials to the AWS Mobile SDK for iOS in the app.

  1. Open the sample app in Xcode.
  2. Go to AppDelegate.swift.
  3. Add the following  line at the top:

import AWSCore

     4. In the app’s didFinishLaunchingWithOptions function, add the following lines:

In the previous steps, we created a credentialsProvider object and assigned the Amazon Cognito identity pool that we created. Then we created an AWSServiceConfiguration object with these credentials. Finally, we set the default AWS configuration for the AWS Mobile SDK in this app to the AWSServiceConfiguration object. This allows the AWS Mobile SDK to talk to AWS for all service resources that are included in the IAM policy that’s attached to the identity pool.

Step 5: Ask Amazon Rekognition to identify faces and celebrities

This section is at the core of what we’re trying to accomplish—talking to the Amazon Rekognition API to detect faces and celebrities in images that are captured by the app.

1. At the top of the ViewController.swift file, add the import line…

import AWSRekognition

2. Create a class variable of type AWSRekognition. We use this to talk to the Amazon Rekognition API.

var rekognitionObject:AWSRekognition?

3. Go to the sendImageToRekognition function, and add the following lines after the snippet to ‘Delete older labels or buttons’:

In this code, we first initialize the rekognitionObject with a default configuration. We then create an instance of AWSRekognitionImage and assign the bytes property, which is the image data that’s captured by the camera or chosen from the photo library. Next, we create an AWSRekognitionRecognizeCelebritiesRequest, and assign the chosen image to the image property of the request.

It’s now time to send this image data to Amazon Rekognition to detect faces and identify celebrities.

1.  Add the following lines to the sendImageToRekognition function after the lines in the previous step:

Here, we passed the AWSRekognitionRecognizeCelebritiesRequest from the previous step to the Rekognition recognizeCelebrities API method. For more details about the Amazon Rekognition response, see RecognizeCelebrities in the Amazon Rekognition Developer Guide.

2.  Inside the if statement for result != nil, add the following lines to extract celebrities and faces from the Amazon Rekognition response. Review the code comments to get a better understanding of each task.

Click on image to see code

Let’s quickly walk through what we did:

  • The response has three components to it: celebrities, faces, and nothing found. We start by extracting celebrities.
  • Photos can contain multiple faces, so we need to iterate through all of the identified celebrities.
  • We use the bounding box information that’s returned by the Amazon Rekognition response to determine where each celebrity’s face is in the image, and place a label with the returned name in the center of the celebrity’s face. We then create a button that serves as a label and a clickable link to the celebrity’s IMDb page.
  • After we’re done with celebrities, we identify other unknown faces in the image and point them out as well.
  • Note that the API also has information about face landmarks like eyes, nose, and the pose (3D orientation of the face) conveyed via pitch, roll, and yaw. We’re not using these in this app, but feel free to explore enhancements using this information.

Step 6: Test your app

That’s it! The hard work is done and it’s now time to try out the application. Build the app to your device and confirm that the app is working and making calls to Amazon Rekognition. If you have a clickable “Jeff Bezos” label for the celebrity in the image, the app is working! Now go ahead and snap a pic of a celebrity on your TV screen, in a magazine, or in person. If everything works, you should see a label on the face in the image that Amazon Rekognition has identified as a celebrity. Clicking on the identified celebrity takes you to the celebrity’s IMDb profile page. Note that the app supports a camera image or an image from the phone’s photo library.

As always, we’d love to hear from you. Please file issues in the GitHub repository for this sample.