Using Amazon Rekognition to enhance MacOS Finder Tags

Sunday morning I was looking at a large folder on my laptop containing hundreds of images. Thumbnails are wonderful, but what I really wanted was an easy way to search my folder quickly for photos that contained picture of cliffs.

Starting in OS X Mavericks, you can use the Tags feature to find tagged files in the Finder window. I wanted to know how difficult it would be to have my laptop send photos to Amazon Rekognition, have each photo analyzed using the deep visual learning of Amazon Rekognition , and then apply these identified objects as tags to my files that I could then open in Finder.

This would give me the ability to search in Finder or Spotlight (a MacOS search feature) by using Tag:<term>. Want to find all of your photos of cats? Tag:Cat would instantly return these results to you.

After finding a snippet of code for the writexattrs function online, it was just a matter of passing the image to Amazon Rekognition, then looping the Tag results and writing them to the file. In about 30 minutes I had 50 lines of code and a working prototype.

The code is available here to play with: https://github.com/mkosut/rekognition_osx_tagfile

That worked fine for processing a large folder of images. To improve performance, a team member submitted a pull request which resized the images prior to uploading and runs the process across multiple threads. What I really want is a way to have these images get auto-tagged when they are added to the folder.

Enter MacOS Automator. Automator provides an easy interface to watch for folder activity and run an action when a new file is written. It’s similar to how AWS Lambda can run any time a file is modified in Amazon S3.

This workflow waits for new files to write into the “TagMe” folder, and passes them to the rek_osx_tagfile.py script with the filename as a parameter.

Now for the final test:

Success!

The one big realization I had while playing with this hack is that AWS can be used to extend the capabilities of virtually anything. Here I have a simple underpowered laptop, yet I’m able to augment the capabilities of it by tapping into the enormous deep learning of Amazon Rekognition to visually inspect my images. All possible with a minimal amount of code!