Learn about SafeHaven: The third place winner of the AWS DeepLens Challenge Hackathon

Nathan Stone (NS) and Peter McLean (PM) are a team both professionally at Haven Power, a business energy supplier in Ipswich, UK, and also off the clock when they recently collaborated to create SafeHaven, the third place winner in the AWS DeepLens Challenge.

SafeHaven was designed to protect vulnerable people, by enabling them to identify “who is at the door?” using an Alexa Skill. Unknown visitors trigger SMS or email alerts to relatives or carers, via an SNS subscription.

Nathan, a BI Developer and Pete, a Data Architect, have been using AWS services to design and build the BI platform at Haven Power.  However, prior to using AWS DeepLens they had no machine learning (ML) experience and didn’t know where to begin. They have now started their ML journey by building SafeHaven using Amazon Rekognition, AWS Lambda, Amazon Dynamo DB, Amazon Alexa, Amazon SageMaker, Amazon S3, Amazon SNS, AWS IoT, and AWS DeepLens.

We interviewed Nathan and Pete about their experience with AWS DeepLens and asked them to tell us a bit more about how they created their winning entry.

The McLean sisters reading in safety thanks to SafeHaven 

Getting started with machine learning

Nathan and Pete were already thinking about ML opportunities, inspired by the work they had been doing with AWS:

NS: “As we began building up our AWS Data Lake architecture (with lots of support from the Solutions Architects at AWS, by the way), Pete and I recognised that there were lots of optimisation and ML problems to solve at work.”  

But, similar to many developers they hadn’t yet found an easy way to get started:

PM: “I had heard lots about Machine Learning over the past few years, and read numerous articles. However not having a PhD in statistical analysis felt that it was a bit beyond my skills. How re:Invent changed this!”

NS: “DeepLens was our first opportunity to get hands-on with ML. The integration with SageMaker was a real eye-opener – we could see that AWS had democratised ML, abstracting away the low-level statistical machinery, freeing us to think about problems at a business level.”

Nathan and Pete watched Andy Jassy announce AWS DeepLens from their respective hotels rooms at AWS re:Invent:

NS: “As Andy announced SageMaker, we guessed that AWS might add some last-minute sessions into the schedule. We were on the phone to each other as we scrambled to book places. We were really excited that we’d secured two spots. When Andy announced that attendees would also receive the DeepLens device, we realised that we’d secured the ultimate in AWS swag!”

They were able to attend the workshop and get their hands on AWS DeepLens and of course the famous workshop hotdogs:

NS: “The workshop was incredible – there was a real buzz around the whole event.”

PM: “The session was great, especially when the hotdogs came round. I didn’t quite manage to get through the entire session without eating though. “

Inspiration for SafeHaven

Nathan and Pete had talked about the entering the hackathon at re:Invent and had a few ideas, but came up with SafeHaven after being inspired by an article that appeared in WIRED in February 2018:

NS: “A few weeks before we had the idea, we read Steven Levy’s WIRED article (https://www.wired.com/story/amazon-artificial-intelligence-flywheel/), which describes the rise of Deep Learning and how it spread throughout the AWS ecosystem. Levy talks about the Jeff Bezos “Six Pager,” in which proposals for new products and services should come with a speculative press release to demonstrate the benefit to the customer. We started out with the idea of “vulnerable people and the family around them.” We had this in our minds throughout, and I think you can see that in the video Pete created showing SafeHaven in action.”

Their prior experience with AWS services helped them get their design in place:

NS: “We’ve done a lot of work with AWS serverless architecture so, once the idea was there, the high-level design fell into place within a couple of days. Obviously, the devil is in the detail, and that’s where most of our weekends went until the submission date. We drank a lot of coffee!”

Building with AWS DeepLens

Nathan and Pete got started by using the AWS DeepLens prebuilt models:

PM: “We started with the object detection model, mainly to test that we could get output from the device. Once this was working we changed to the facial detection model. Getting the device to output the image to S3 took a bit of research, but thanks to the DeepLens Slack Channel managed to get this working.”

As you will see in the architecture diagram (PDF), Nathan and Pete brought multiple pieces of technology together to create SafeHaven. They extended the functionality of AWS DeepLens, integrating it not only with Amazon Echo and the Amazon Alexa Skills Kit, but also with Lambda functions to integrate with Amazon Rekognition and Amazon DynamoDB for the processing and storing of the images. In addition it was integrated with Amazon SNS to trigger SMS or email notifications when an “unknown” person is detected.

SafeHaven triggers a notification to alert an unknown person is at the door

Ultimately for Nathan and Pete, they were able to get hands-on with machine learning and relished the challenge of being amongst the first to have the AWS DeepLens developer kit to help them on their way:

NS: “DeepLens is a new piece of kit and there were a few teething problems, as you would expect. I think lots of people faced the same issues – e.g., Wi-Fi resets and model deployment. But that’s where the fun lies – how often do you get to play with a piece of tech that no-one else can buy yet? “

What’s next for SafeHaven

Nathan and Pete have several ideas to take SafeHaven to the next level. The current iteration shows the example of a child as the vulnerable party, this could also be extended to other use cases, such as the elderly or other vulnerable persons. They are also thinking about technology extensions like using the Amazon Echo Show, which could display the visitor’s face along with a caption to describe who they are.

PM: “We can see the system scaling up to be used in multi-tenant assisted care facilities, where the door can only be opened to recognised visitors, even when vulnerable or elderly residents may have been persuaded to permit entry. In this case, on-site supervisors will receive alerts so they can challenge unknown visitors.”

They are also keen to continue their innovation journey with Amazon:

NS: “We’ve been racking our brains for an idea for the latest Alexa “Life Hacks” competition. Who needs weekends, anyway?”

Winners’ celebration

Nathan and Pete are not short on ideas of what to do with their winnings:

NS: We’re coming back to Vegas! The day after the winners were announced, we saw that Jeff Bezos bought “Ring” for $1Billion. We really wish he’d spoken to us first – he could have bought SafeHaven for half of that!”

PM: “I’d quite like to put it towards a Nord Stage 3 keyboard, but I think it will go towards something for the kids (not any more toys though, as you may have noticed from the video I think they’ve already got enough!).

In conclusion

Similar to the journey our prior winners with their projects ReadToMe and Dee, Nathan and Pete have gone from having no machine learning experience to building a project that children and other vulnerable people can now benefit from.

Congratulations to Nathan Stone and Pete McLean and their families on this well-deserved win! A special shout out to Pete’s wife and daughters—the stars of the SafeHaven video! Nice job!

Hopefully, Nathan and Pete’s story has inspired you to want to learn more about AWS DeepLens. You can view all of the projects from the AWS DeepLens Challenge on the DeepLens Community Projects webpage. For more general information, take a look at the AWS DeepLens website or browse AWS DeepLens posts on the AWS Machine Learning blog.

The AWS DeepLens Challenge was a virtual hackathon brought to you by AWS and Intel to encourage developers to get creative with their AWS DeepLens. To learn more about the contest, check out the AWS DeepLens Challenge website. Entries are now closed.

About the Author

Sally Revell is a Principal Product Marketing Manager for AWS DeepLens. She loves to work on innovative products that have the potential to impact people’s lives in a positive way. In her spare time, she loves to do yoga, horseback riding and being outdoors in the beauty of the Pacific Northwest.