Mechanical Turk Case Study: Channel Intelligence

However, technology alone has not been able to completely fill the gap when it comes to subjective intelligence. Answering questions such as “Is this a picture of a printer?” or “What are the attributes of this product?” still require a higher level of intelligence; typically human intelligence. Unfortunately, human intelligence is a costly resource that has significant limitations when it comes to scalability.

At the beginning of 2008, CI was introduced to Amazon Mechanical Turk as a possible solution for filling the human intelligence gap while ensuring high quality results. CI started with a simple test. They took 73,000 products that its automated optimization systems had a hard time categorizing and asked the workers on Mechanical Turk (also called “Turkers”) to categorize them.

If CI used internal resources they determined it would take approximately 600 valuable man hours or 75 days to complete. The Turkers finished categorizing all 73,000 products in 4 days. After analyzing the results, CI found the items were categorized correctly 88.3% of the time. And, the work was done at 22.1% of the normal cost.

Since the initial test, CI has utilized Turkers for nine different task types ranging from categorizing products and harvesting product attributes to identifying retail store locations and validating product URLs. By integrating Mechanical Turk into its operational processes, CI has been able to leverage the intelligence of thousands of humans worldwide. This change allowed CI to increase productivity and the velocity in which the work is done, while increasing overall quality.

Tim Tryzbiak, Vice President of Operations shares some best practices:

1. Continually communicate to the Mechanical Turk community

“The diligence, work ethic, and effort put forward by the Turkers exceeded our expectations. We found that most of the Turkers truly cared about their work and were very thorough in their replies.”

“Turkers are an active community that use blogs and forums to discuss work and offer advice back to the requesters. CI spends roughly 15 hours a week reviewing the community forums and responding to Turker questions. This time investment pays off because the Turkers are more involved in the projects and seem to be more focused when there is support.”

2. Leverage consensus to manage quality
“Mechanical Turk gave us the ability to have a single piece of work done multiple times by different people around the world. This enabled us to create algorithms that process the results and determine the final answer based on consensus. In tasks where higher quality was needed, we simply increased the consensus constraint.”

“By keeping track of the Turkers’ results we were able to determine which Turker should be paid, used going forward, or blocked from future projects. Amazon also does a great job of managing the Turker base. Of the roughly 6,000 Turkers that have worked on CI tasks, we had to block less than 4% of Turkers because for poor quality performance.”

3. Integrate into existing systems for maximum efficiency
“Having thousands of workers around the world working on your tasks is powerful. However, having to deal with the results can be challenging. Initially, we treated Mechanical Turk as a side solution to our normal processes. Results were then merged together with production data and put back into the production systems.”

“With CI’s larger projects we learned to trust the Mechanical Turk results the same way we trust our other technologies. Once quality has been confirmed, data is injected directly into the production systems and managed like any other data source. This allowed us to leverage the Amazon Mechanical Turk workforce, using a robust API, as a core technology and not just a side process.”

In commercial applications, automated subjective intelligence is neither cheap nor easy to implement. When looking at the changes the Internet has brought over the last several years, it’s hard to ignore its next evolution: access to a global workforce. Using Amazon Mechanical Turk, CI was able to leverage human intelligence around the globe and decrease task specific costs by 85%.

To read the full interview with Tim Tryzbiak, VP of Operations at Channel Intelligence, visit Channel Intelligence Interview

Learn more about CI at channelintelligence.com or for questions about CI’s Mechanical Turk implementation, contact amazonturk@channelintelligence.com.