The Internet of Things on AWS – Official Blog
How to Install a Face Recognition Model at the Edge with AWS IoT Greengrass
Editor Note: This post is co-authored by Gong Xinyue, one of Global Accounts Solution Architects.
You might already know how to use AWS IoT Core and AWS IoT Greengrass for remote device communication and control. With AWS IoT Greengrass Machine Learning (ML) Inference, you can run machine learning models on your local devices without any transmission delay. In this blog post, I will show you how to use AWS IoT Greengrass ML Inference on a Raspberry Pi to perform local facial recognition for home surveillance.
Using an Amazon Echo Dot, which is connected to the Alexa Voice Service, as the control device for the Raspberry Pi’s camera, you’ll be able to take a photo of people outside your door and, using the photo, perform facial detection and comparison with a local dataset using the pretrained ML model deployed to the Raspberry Pi. Although the comparison results can also be used with your door lock or other smart devices, these use cases are not covered in this post.
Prerequisites
Install the AWS IoT Greengrass Core software on your Raspberry Pi device. Create an IoT Greengrass group and core. For instructions, see Getting Started with AWS IoT Greengrass in the AWS IoT Greengrass Developer Guide. The group and core are used in deployments.
In this post, I use a pretrained face-detection model, train it with TensorFlow, and then deploy it to the Raspberry Pi with AWS IoT Greengrass. You can use the same model, or you can use Amazon SageMaker to train one of your own. For information about how to prepare a model with Amazon SageMaker, see Get Started in the Amazon SageMaker Developer Guide.
Install the following dependencies.
- OpenCV 3.3
- TensorFlow
- Numpy
- Scipy
- Scikit-learn
- Dlib
- All image processing libraries
Use the following commands to install the image processing dependencies.
sudo apt-get install build-essential sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev sudo apt-get install python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev pkg-config graphicsmagick
Use the following commands to install OpenCV 3.3.
git clone https://github.com/Itseez/opencv.git git clone https://github.com/Itseez/opencv_contrib.git cd opencv git checkout 3.1.0 cd ../opencv_contrib/ git checkout 3.1.0 cd .. cd opencv mkdir release cd release cmake -D CMAKE_BUILD_TYPE=RELEASE \ -D CMAKE_INSTALL_PREFIX=/usr/local \ -D INSTALL_PYTHON_EXAMPLES=ON \ -D OPENCV_EXTRA_MODULES_PATH=~/opencv_contrib/modules \ -D BUILD_EXAMPLES=ON .. make -j4 sudo make install sudo ldconfig
The ML model used in this post is TensorFlow. Use the following commands to install TensorFlow on your Raspberry Pi.
# Install Wheel wget https://github.com/samjabrahams/tensorflow-on-raspberry-pi/releases/download/v1.0.1/tensorflow-1.0.1-cp34-cp34m-linux_armv7l.whl sudo pip install tensorflow-1.0.1-cp34-cp34m-linux_armv7l.whl sudo pip install mock #Install Pip3 sudo apt install libatlas-base-dev pip install tensorflow
Use the following commands to install Numpy, Scipy, and Scikit-learn.
pip install numpy scipy scikit-learn
Use following commands to install Dlib.
#Modify your swapfile sudo nano /etc/dphys-swapfile change CONF_SWAPSIZE = 100 to CONF_SWAPSIZE=1024 change virtual memory from 100M to 1G sudo /etc/init.d/dphys-swapfile restart sudo raspi-config #Change you boot Options 1)Boot Options => Desktop / CLI => Console Autologin 2)Advanced Options => Memory Split => change GPU to 16 #Doanload your Dlib download Dlib to your device : http://dlib.net/ sudo python setup.py install
ML models and AWS Lambda functions
You will find two ML models on GitHub.
- Face Detection Model
|—-haarcascade_frontalface_default.xml - Face Recognition Model
|—–train_faces.model-500.data-00000-of-00001
|—–train_faces.model-500.index
|—–train_faces.model-500.meta
|—–checkpoint
You will also find notes to that describe the Lambda functions used in this post.
FaceDetection Lambda function on AWS IoT Greengrass
#open camera cam = cv2.VideoCapture(0) while True: # capture an image from video stream _, img = cam.read() # convert RBG image to grayscale image gray_image = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) # detect face dets = detector(gray_image, 1) if not len(dets): # detect face failed key = cv2.waitKey(30) & 0xff if key == 27: sys.exit(0) # start recognition for i, d in enumerate(dets): x1 = d.top() if d.top() > 0 else 0 y1 = d.bottom() if d.bottom() > 0 else 0 x2 = d.left() if d.left() > 0 else 0 y2 = d.right() if d.right() > 0 else 0 face = img[x1:y1, x2:y2] face = cv2.resize(face, (size, size)) # compare image with dataset and compose JSON format message record_data = {} record_data['result'] = is_my_face(face) prepared_record_data = json.dumps(record_data) # send MQTT message to “record/webcam” topic client.publish(topic = 'record/webcam', payload = prepared_record_data) break;
Alexa trigger Lambda function
def get_welcome_response(): # public a message to IoT core client.publish( topic='listening/record', payload="Hi", qos=1 ) # return message of this Alexa Skill session_attributes = {} card_title = "Welcome" speech_output = "Sure. I will check the camera now." # If the user either does not reply to the welcome message or says something # that is not understood, they will be prompted again with this text. reprompt_text = "Sure. I will check the camera now." should_end_session = True return build_response(session_attributes, build_speechlet_response( card_title, speech_output, reprompt_text, should_end_session))
Get started
The architecture of the example described in this post is shown here. The facial recognition model and datasets, which are used to create AWS Lambda function for recognition, have been uploaded to an Amazon S3 bucket. AWS IoT Greengrass synchronizes the required files to the Raspberry Pi. Echo Dot runs as a trigger. When Echo Dot listens to a command such as,“Alexa, open Monitor,” it calls an Alexa skill to send a message to AWS IoT Core. AWS IoT Core invokes the recognition Lambda function, which is deployed on Raspberry Pi local storage, and if the Lambda function recognizes the identity of the guest, the door opens.
Let’s prepare the resources you will deploy on the Raspberry Pi.
First, create an AWS Lambda function that will capture the frame from the Raspberry Pi camera, convert it to a JPG file, and then invoke the local ML model for analysis. The AWS Lambda function will send the analysis result in JSON format to AWS IoT Core.
In the AWS Lambda console, choose Create Function, and then choose Author from scratch. Give your AWS Lambda function a name like greengrassFaceRecognization. For Runtime, choose Python 2.7. Because this Lambda function will be deployed on edge devices, you can choose any role without impacting its function on the Raspberry Pi.
In the function code area, choose the option to upload a ZIP file, and then upload the ZIP package provided for this Lambda function from GitHub. From Actions, publish a new version. AWS IoT Greengrass doesn’t support $LATEST as the version for Lambda aliases, so make sure you assign a version number (for example, version 1) to your Lambda function.
When the Lambda function runs, it will call the local ML model for facial recognition. Use the following values to configure the Lambda function:
Attribute | Configuration |
Memory Limit | 96 |
Timeout | 8 |
Lambda Lifecycle | Make this function long-lived and keep it running indefinitely |
Read access to /sys directory | Enable |
Input payload data type | JSON |
The Lambda function needs to invoke some local devices on your Raspberry Pi. Add those devices to your AWS IoT Greengrass resources.
In the AWS IoT Core console, choose AWS IoT Greengrass. Choose Groups, and then choose your Greengrass group (for example, greengrassFaceReco). In the left navigation pane, choose Resources. On the Local tab, choose Add a local resource.
Because the example in this post uses the Raspberry Pi camera, add two devices to the Greengrass group:
- videoCoreSharedMemory
- videoCoreInterface
Complete the fields as follows:
When you’re done, your device configuration should look like this:
Add your ML model to this Greengrass group. In the left navigation pane, choose Resources, choose Machine Learning, and then choose Add machine learning resource.
In this example, the model is stored in an S3 bucket. If you use Amazon SageMaker to train your model, choose Amazon SageMaker as your model source. The local path is the directory on your edge device where you will store the model. Configure this path carefully. Otherwise, your model cannot be used on your edge device. You can see your model has been added to your Greengrass group:
Now create a subscription so that the local analysis result can be sent to the AWS IoT Cloud for processing, such as storing the result in Amazon DynamoDB and performing result analytics in Amazon EMR. If you have another local device connected to Raspberry Pi, you can use the AWS IoT Greengrass core to control it through AWS Lambda based on the analysis result. In this example, the result is sent to the AWS IoT Cloud through an MQTT message.
You can now deploy the Greengrass group. From Actions, choose Deploy. Deployment times vary, depending on the size of the model. When the deployment is complete, the results appear in the console, as shown here.
Your Raspberry Pi should now be able to recognize the face photo captured by the Pi camera. Use Echo Dot to voice control the Pi camera to get the image.
Create another Lambda function to trigger the AWS IoT Greengrass local face detection Lambda function through an MQTT message. The trigger Lambda function will send an MQTT message to the Greengrass core through AWS IoT Core. AWS IoT Greengrass will process this message through the local face detection Lambda function and then trigger the photo analysis. You can find this Lambda code based on Python 2.7 on GitHub. Make a note of the ARN for this Lambda function. You will use it when you configure Alexa skills.
Open the Amazon Developer Portal. Choose Alexa, choose Your Alexa Consoles, choose Skills, and then choose Create Skill. For information about creating your own Alexa skills, see Build Skills with the Alexa Skills Kit.
Finally, build a smart home surveillance system by using your Echo Dot to send a voice control message to your Raspberry Pi to start the local face recognition process. The result will be sent back to the AWS IoT Cloud through an MQTT message.
Conclusion
By using the example in this post, you can build a small home surveillance system on your Raspberry Pi device with AWS IoT Greengrass. Then, you can explore further by training other ML models and deploying them to your AWS IoT Greengrass devices.