Second Place Winner:
Dee - DeepLens Educating Entertainer
My initial hope was that, rather than pictures, the child could show toys to Dee. Picking up, for example, a toy plane or cuddly sheep, would be more exciting than a piece of paper. But in testing, the object detection model did not see toys as being the same as their real counterparts. A toy plane is just too different from a real plane, it seems. Training a model to work on toys would fix this, of course, but I was unable to find a good and large enough training data set. This is something to work on.
Accomplishments that I'm proud of
I'm impressed with how this form of interaction really works. As you'll see from the YouTube video, we tried Dee with my three-year-old son, and he loved it. He's asking to play with it again. This may be a prototype but it's good enough for him to use.
I'm also excited with how the positive reinforcement aspects can help children with autism or Asperger's.
What I learned
This project has brought me up to speed with deep learning concepts, and AWS's approach to managing and running them (through SageMaker and GreenGrass).
Away from the tech, I've also learned the potential of technology increasingly understanding the human world. Intelligent video recognition allows all kinds of new ways to play games and learn new things.
What's next for Dee
The potential for Dee is huge. If she could recognise a wider range of things, a much more varied set of questions could be asked. Consider:
"Can you hold up three fingers?" (to test counting skills)
"Show me your biggest smile!"
"Can you do a star jump?"
"Which one is the letter A?"
"Can you show me your favourite toy?"
Of course, training new models will be a key part of this. And with services such as SageMaker making training more straightforward, the possibility emerges for end-users being able to train their own models. A teacher could, for example, train Dee to recognise certain objects in the classroom. Or a carer could train Dee to respond to specific objects that are important to someone with autism.
Finally, there's plenty more improvements in the logic that could be done too. Could Dee track your progress over time, and report on how well you're learning, say, the alphabet? Could it recognize different people and set them different challenges? The possibilities are endless.