AWS AI Blog

Customize and Display AWS DeepLens Project Output on Your Laptop

by Guy Ernest and Sunil Mallya | on | in AWS DeepLens | Permalink | Comments |  Share

AWS DeepLens is a deep-learning-enabled developer toolkit with a video camera. It enables you to develop machine learning skills using hands-on computer vision tutorials, pre-built models and allows you to extend them. Examples of pre-built models include object detection to recognize and detect different objects in your room such as TV monitors, people and bottles, and action recognition to recognize different kinds of actions such as brushing teeth, applying lipstick, drumming, playing a violin, and playing basketball.

AWS DeepLens allows you to display the stream from the device’s camera as well the output of the models on the IoT console as well as the local device. To learn more on how to do this, you can refer to the documentation. In this blog post, we’ll discuss how you can customize and display project output from AWS DeepLens on an HTML page.

We will use:

  • Amazon Cognito to enable the HTML page to access AWS DeepLens MQTT messages via IoT WebSockets
  • AWS IoT to handle data subscription and publishing
  • Amazon S3 to store the HTML file used to display the output

You can either use AWS CLI or the AWS Management Console to customize your AWS DeepLens project output. Steps for using both the CLI as well as the console are elaborated below.

Prerequisites

To follow the steps below to customize your AWS DeepLens output, you need to

  • Own an AWS DeepLens device
  • Register the device
  • Create a project and deploy it to the device

For more information, you can refer to the documentation. If you do not own an AWS DeepLens device, you can sign up to join our waitlist.

Steps to get started using AWS CLI

To get started, using AWS CLI, follow the steps below.

STEP 1: Create a Cognito Identity Pool

Every time you register an AWS DeepLens device, the service creates a unique MQTT topic. This topic showcases your deployed model’s output on the AWS IoT console. For example, if the project you’ve deployed to AWS DeepLens detects objects in your surroundings, the MQTT topic can be used to emit detected object names to the IoT console. For this blog post, we want to be able to display the MQTT messages on the customized HTML page. We will use Amazon Cognito for this.

Amazon Cognito makes it easy to access AWS resources in a secure way using temporary credentials, avoiding the need to put AWS credentials in source code or web pages.

For this, we first need to create an identity pool. We will later supply this IdentityPoolId to the HTML page, to enable it to subscribe to the MQTT topic using temporary credentials.

On AWS CLI, paste the below command. Before you execute the step, ensure that you are creating the identity in the desired AWS Region.

> aws cognito-identity create-identity-pool \ 
  --identity-pool-name DeepLens_Console  \
  --allow-unauthenticated-identities --region us-east-1

The response will in the following format:

{
    "IdentityPoolId": "us-east-1:XXXXXXX-1cd1-4c3a-9a39-dac267545277",
    "AllowUnauthenticatedIdentities": true,
    "IdentityPoolName": "DeepLensConsole"
}

Take note of the IdentityPoolId (highlighted in yellow). You will use it later as a parameter to the HTML page.

Note: This identity pool will allow unauthenticated users to get some permissions to access your AWS resources (which we will define in the next steps). This has been done to simplify the interaction with the HTML page. If you want to tighten security on the page, you can remove support for unauthenticated users altogether, or restrict page access to limited functionality. You could also add sign-up functionality to the page to authenticate users using various login providers. 

STEP 2: Create an IAM role to allow AWS IoT MQTT subscription

The AWS DeepLens device communicates to the cloud over IoT subscriptions using MQTT topics and messages. These channels are optimized for secure and reliable communication in IoT environments. To be able to receive these messages and display them on the HTML page, we need to define the following IAM role and permissions.

First, create a policy document for the IAM role. For this, save the following content to a file called DeepLensConsoleUnauthName.json, in the same directory where you are running the CLI commands. Please note to replace the IdentityPoolId with the one you retrieved in the previous step.

In this step, we define the trust policy for unauthenticated users to access AWS resources. For now, the role has no permission, but we allow Amazon Cognito to assume it.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "cognito-identity.amazonaws.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "cognito-identity.amazonaws.com:aud": "us-east-1:XXXXX-1cd1-4c3a-9a39-dac267545277"
        },
        "ForAnyValue:StringLike": {
          "cognito-identity.amazonaws.com:amr": "unauthenticated"
        }
      }
    }
  ]
}

Next, create an IAM role and attach the policy that you just created to it. Note that you need to use the same path and file name that you created in the previous step. Remember the role name, as you will need it for subsequent steps.

> aws iam create-role --role-name DeepLensConsoleUnauthName \  
  --assume-role-policy-document file://DeepLensConsoleUnauthName.json

Next, attach the AWSIoTDataAccess policy to the role that you just created. This attaches a managed policy that allows this role to read and write messages to all topics of your IoT endpoint. You can create a more restrictive policy to limit the topics and commands allowed, or you can add more permissions to the role. For example, you could allow Amazon Polly to read the labels out loud using the following policy-arn: arn:aws:iam::aws:policy/AmazonPollyReadOnlyAccess.

> attach-role-policy \
  --role-name DeepLensConsoleUnauthName \
  --policy-arn arn:aws:iam::aws:policy/AWSIoTDataAccess

STEP 3: Connect the identity pool and the IAM role

Now you need to connect the Cognito identity pool to the IAM role. Use the following command to make this association.

> aws cognito-identity set-identity-pool-roles \ 
--identity-pool-id "us-east-1:XXXXX-1cd1-4c3a-9a39-dac267545277" \ 
--roles unauthenticated=DeepLensConsoleUnauthName

STEP 4: Find the IoT endpoint of your account

Using AWS CLI, follow these steps:

Type the command.

aws iot describe-endpoint

It will return your IoT endpoint.

{
       "endpointAddress": "XXXXXXXXXXXXX.iot.us-east-1.amazonaws.com"
}

STEP 5: Open the HTML page to view your project output

Now you can open the static HTML page in Amazon S3 by passing the parameters you defined earlier:

https://s3-us-west-2.amazonaws.com/im-examples/console/index.html?region=us-east-1&host=IoTEndpoint&id-pool=IdentityPoolId

For example, here is the link with the values used in this guide:

https://s3-us-west-2.amazonaws.com/im-examples/console/index.html?region=us-east-1&host=XXXXXXXXX.iot.us-east-1.amazonaws.com&id-pool=us-east-1:XXXXXXXX-XXXX-4a79-8dd4-568590df9298

Depending on what your project displays as a result, you can view MQTT messages, images, and a video stream with bounding boxes on this HTML page.

Steps to get started using the AWS Management Console

To get started using the AWS Management Console, follow these steps.

STEP 1: Create a Cognito Identity Pool

Every time you register an AWS DeepLens device, the service creates a unique MQTT topic. This topic showcases your deployed model’s output on the AWS IoT console. For example, if the project you’ve deployed to AWS DeepLens detects objects in your surroundings, the MQTT topic can be used to emit detected object names to the IoT console. For this blog post, we want to be able to display the MQTT messages on the customized HTML page. We will use Amazon Cognito for this.

Amazon Cognito makes it easy to access AWS resources in a secure way using temporary credentials, avoiding the need to put AWS credentials in source code or web pages.

For this, we first need to create an identity pool. We will later supply this IdentityPoolId to the HTML page, to enable it to subscribe to the MQTT topic using temporary credentials.

In the Amazon Cognito console choose “Manage Federated Identities” and “Create new identity pool”. Give it a name (for example, “DeepLens_Console”) and check the option to “Enable access to unauthenticated identities.”

Note: This identity pool will allow unauthenticated users to get some permissions to access your AWS resources (which we will define in the next steps). This has been done to simplify the interaction with the HTML page. If you want to tighten security on the page, you can remove support for unauthenticated users altogether, or restrict page access to limited functionality. You could also add sign-up functionality to the page to authenticate users using various login providers. 

Allow Amazon Cognito to assume roles on your behalf.

On the next screen, you will see instructions and code samples on how to use the IdentityPoolId in mobile and web interfaces. For now, choose the “Edit identity pool” button, and note the IdentityPoolId and the role name for the unauthenticated identities.

STEP 2: Edit IAM role to allow AWS IoT MQTT subscription

The AWS DeepLens device communicates to the cloud over IoT subscriptions using MQTT topics and messages. These channels are optimized for secure and reliable communication in IoT environments. To be able to receive these messages and display them on the local HTMP page, we need to define a specific IAM role with corresponding permissions to access the IoT resources.

Go to the IAM Console and search for the name of the unauthenticated role. Select the role with the Unauth_Role suffix (the second on the list).

Now, click “Attach policy” to the role, search for the IoT managed policy, and choose “AWSIoTDataAccess”.

Choose “Attach policy.”

This attaches a managed policy that allows this role to read and write messages to all topics of your IoT endpoint. You can create a more restrictive policy to limit the topics and commands allowed, or add more permissions to the role (for example, you could allow Amazon Polly to read the labels out loud using the following policy:arn: arn:aws:iam::aws:policy/AmazonPollyReadOnlyAccess ).

STEP 3: Find the IoT endpoint of your account

Using the AWS IoT console, choose the Settings option:

STEP 4: Open the HTML page to view your project output

Now you can open the static HTML page in Amazon S3 by passing the parameters you defined earlier.

https://s3-us-west-2.amazonaws.com/im-examples/console/index.html?region=us-east-1&host=IoTEndpoint&id-pool=IdentityPoolId

For example, here is the link with the values used in this blog post:

https://s3-us-west-2.amazonaws.com/im-examples/console/index.html?region=us-east-1&host=XXXXXXXXX.iot.us-east-1.amazonaws.com&id-pool=us-east-1:XXXXXXXX-XXXX-4a79-8dd4-568590df9298

Depending on what your project displays as a result, you can view MQTT messages, images, and a video stream with bounding boxes on this HTML page.

Sending Images from the device to show on the project output page

The simple functionality of the edge AWS Lambda functions is to publish textual messages to the IoT topic. You will be able to see these messages on the project output page (or on the “test” tab in the AWS IoT console).

The first extension is to show images before or after model inference (with bounding boxes). In this example, you will upload the images to Amazon S3 and send the image URL to the IoT topic.

Upload Images to S3

First, you will add a Python function to the edge Lambda function to upload the images to Amazon S3:

# Function to write to S3
# The function is creating an S3 client every time to use temporary credentials
# from the GG session over TES 
def write_image_to_s3(img):
    session = Session()
    s3 = session.create_client('s3')
    file_name = 'DeepLens/image-'+time.strftime("%Y%m%d-%H%M%S")+'.jpg'
    # You can contorl the size and quality of the image
    encode_param=[int(cv2.IMWRITE_JPEG_QUALITY),90]
    _, jpg_data = cv2.imencode('.jpg', img, encode_param)
    response = s3.put_object(ACL='public-read', Body=jpg_data.tostring(),Bucket='<BUCKET_NAME>',Key=file_name)

    image_url = 'https://s3.amazonaws.com/<BUCKET_NAME>/'+file_name
    return image_url

Next, you will call the function from the inference Python function:

    ...
    # Upload to S3 to allow viewing the image in the browser
    image_url = write_image_to_s3(frame_with_bb)

And add it to the output message:

    client.publish(topic=iotTopic, payload='{{"img":"{}"}}'.format(image_url))

Another addition to the images is showing bounding boxes, when the model output it (‘ssd’ type):

# Adding top n bounding boxes results
def apply_bounding_box(img, topn_results):
        '''
        cv2.rectangle(img, (x1, y1), (x2, y2), RGB(255,0,0), 2)
        x1,y1 ------
        |          |
        --------x2,y2
        '''
        for obj in topn_results:
            class_id = obj['label']
            class_prob = obj['prob']
            img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
            cv2.rectangle(img, (int(obj['xmin']), int(obj['ymin'])), (int(obj['xmax']), int(obj['ymax'])), (255,0,0), 2)
        return img

And use the following code to get top n (for example, 5) results and apply them to the image:

     ...
     results = model.parseResult('ssd',results)
     tops = results['ssd'][0:5]
     frame_with_bb = apply_bounding_box(frame_resize,tops)

Conclusion

There are many other additions to such an output page, which some will be covered in future blog posts. Some examples are to stream the video over MQTT messages and rebuild the stream on the output page, or embed Amazon CloudWatch graphs or dashboards, etc.

AWS DeepLens is designed as an open platform to build educational and production projects alike. We expect you to extend the built-in projects by building new deep learning models, new Lambda edge functions, new functionalities with cloud logic, and new human interfaces to control and view the output of your projects.
In this blog post, you saw a simple HTML-based page that is able to connect securely to your device and projects running on it, and show the output of the inference of projects. I hope that it gives you a good framework and foundation to build more advanced interfaces for more advanced use cases. I would love to hear your ideas and share them with the Community around AWS DeepLens.


Additional Reading

Learn how to extend AWS DeepLens to send SMS Notifications with AWS Lambda.


About the Authors

Guy Ernest is a principal solutions architect in Amazon AI. He has the exciting opportunity to help shape and deliver on a strategy to build mind share and broad use of Amazon’s cloud computing platform for AI, machine learning and deep learning use cases. In his spare time, he enjoys spending time with his wife and family, gathering embarrassing stories, to share in talks about Amazon and the future of AI.

 

Sunil Mallya is a Senior Solutions Architect in the AWS Deep Learning team. He helps our customers build machine learning and deep learning solutions to advance their businesses. In his spare time, he enjoys cooking, sailing and building self driving RC autonomous cars.