Amazon Mechanical Turk and Image Processing

Image processing (and its cousin, machine vision) both entail designing algorithms to process pixel-based images and to extract recognizable information from them.

Earlier today I saw the following request on the Ask Metafilter site:

i need an algorithm to extract statistics about the diameters (in pixels) of some roughly-circular objects from an image file.

Having done several presentations on the Amazon Mechanical Turk in the last couple of days, it sometimes feels like I am holding an all-purpose hammer in my hand!

Without knowing how many images the requester needs to process, I could see several interesting ways to get this job done using one or more types of Turk HITs (Human Intelligence Tasks).

First, the work could simply be converted into HITs. The worker would be presented with an enlarged image of the circular objects with a grid superimposed, and then asked to count the pixels, something like this:

Of course, some Qualifications would be used to ensure high quality results, and the same HIT can be sent to more than one worker and plurality used to select the result chosen by the majority of the workers.

Second, the Mechanical Turk could be used to run a quality check on the results of an actual image processing algorithm.

Third, the Mechanical Turk could  be used to handle only certain images, perhaps  those that don’t clearly separate the image from its background, or those for which the algorithm returns an indeterminate result.

Fourth, the Mechanical Turk could be used  to provide the feedback needed to train some type of learning algorithm, perhaps one that uses a neural net or something even more advanced. The machine would, in effect, ask the worker “is there a circle of size N pixels here?”

I’ll be the first to admit that this isn’t real Computer Science, but it is problem solving, and in the end that’s what is really needed.

It is worthwhile to take a look at the economics here as well. How many hours is it going to take to program and test a suitable recognition and measuring algorithm, and how much would it cost to do all of the processing using HITs? How much testing will be needed to see how well the algorithm works, and what if the problem statement changes and we also need to measure squares or octagons?

Let’s say that we pay workers 1 cent for each such HIT, and that we send each one to 5 workers to get a high quality result, accepting only those answers that are within 10% of the acceptable size. We end up paying around 4 cents per image.

Sometimes it makes sense to put a real human being in the loop, and this is quite possibly one of them.

— Jeff;