Mechanical Turk Case Study: CastingWords

Why did you start your business?

We created this business because we were having a very hard time with the searchability of audio. We were big into podcasts, which were impossible to find. So we started transcribing podcasts to increase searchability. That eventually led directly to a business, doing that for money. There aren’t any companies who do good voice recognition of untrained audio so there is a need in this market.

How long have you used Mechanical Turk?

Our first batch of HITs was about two weeks after the Mechanical Turk launch. It was a set of audio transcriptions of our favorite podcasts and that would let other people search our favorite podcasts, and that was very quick. It was easy to get up and running with MTurk, even back then.

What do you think the most common misconception is about Mechanical Turk?

I think the biggest misconceptions about MTurk is the quality of work that gets returned. A lot of people recognize that the quality is good; but I think a lot of people expect the quality to be very, very poor. And it just isn’t. The quality is amazing. The quality is astounding. You get things done that you didn’t even know you could get done, much less returning back good results.

What work do you get done with Mechanical Turk?

We were always big on MTurk. So most things were natural, and we went straight for a MTurk solution for almost everything, including editing our home page and all kinds of stuff including marketing. Over time we eventually started using the MTurk workers to proof the work. Before we were taking the work ourselves and saying about each transcription: “Yeah, this is good”, “Yeah, this isn’t good”. We started using MTurk workers for that and have never turned back. This grading of the work using Mechanical Turk is something that we view as one of our core competencies: it’s actually what we sell most to our consulting clients. This gives our customers the ability to not only get their work done and back but also to pay the right amount for the highest-quality work.

How do you use workers to grade the completed work?

We use the workers to grade all of the transcribed work and we use a number of different methods. The basic idea is that Turkers are very good at “Yes”, “No”, or numerical grades. So, after we get a completed task back, we send that task back out to Turkers to say “Yes, this is good” or “No, this isn’t good” or, more frequently, grade it from 1 to 10. When a number of workers agree that a completed assignment was a grade 8, then we pay a bonus to the original worker, based off the score being an 8. We use a large group for consensus on the grades. This has led to our having metrics on over 8,000 Mechanical Turk workers in a database, including their testing skill method. This is quite amazing for a company of our size of infrastructure, to have that amount of workers at our fingertips. We’re just a couple people and so this is a big thing. Flexibility with MTurk allows you to gather good quality metrics on all of your workers and pick and choose workers based on what you need. Using numbered metrics makes this possible and much faster than traditional methods.

What type of workforce have you established and how does this compare to traditional methods?

The real impact in our cost saving has been the flexibility of getting work done. This is a superior alternative to going the traditional route: hiring people, having a big infrastructure, computers, a space and all of that stuff. With a hiring or outsourcing method, it would be expensive to even just get them in for an hour at a time; we’d have to pay them while we sorted their work and identify where they would contribute most in our work process. I’d have to do that, and I wouldn’t be able to generate code to decide whether this worker was good or this worker was bad. With MTurk, the work comes in and other Turkers grade. The Turkers tell me whether the work is good or bad, and piece-by-piece, we build out a whole infrastructure of trust. We use good workers more, and we don’t use bad workers at all. We have the metrics on many Turkers and they’re basically what makes the company tick. What makes any good MTurk project tick is to know your workers and be able to leverage the best workers for their unique contribution.

How has Mechanical Turk helped CastingWords grow and what are your plans for future growth?

We’ve grown the business hugely using MTurk. We started out self-funded, both of us had other jobs, and today we both work full-time on this company. We’ve been up to five people internally to the company, although we aren’t any longer because we’ve outsourced a lot of that work back out to MTurk. We’ve got 10s of thousands of orders, and those orders come in from little teeny people straight up to Fortune 500 companies. We’ve been more than doubling every year for annual revenue for the past three years.

We have a lot of future plans to grow the business. We are constantly expanding our product line. We just introduced a taped transcription product, which will feed those results directly into Mechanical Turk, and we have added video services. We will be transcribing streaming video including YouTube. We’re also adding captioning services. Those things will all flow through Mechanical Turk. We’re looking to sort of expand the kinds of ways we do things. We’re adding an audio engineering task sometime in the near future. We will have Turkers take audio and improve the quality so that our transcribers can do a better job.

What advice would you give others who want to build businesses around Mechanical Turk?

We feel specialization is how you build a good Mechanical Turk business. Turkers seem to like specialization. They like to do one task they do very well and just keep doing it. If they find a HIT they like, they will come back day after day.

To make a good MTurk product built on top of Mechanical Turk you break down all your tasks into subcategories. Select and categorize workers in various categories. For example, those good at grading should grade and good at audio engineering do audio engineering. And if the audio engineers suck at transcription, that shouldn’t be part of their job. By breaking everything down, you let workers excel at things they’re really good at.

To find and label what workers specialize in, you should advertise tasks in categories and let people come in and do the non-time critical tasks. Then, the results come back and are graded by the Turkers who are your graders. Those Turkers will tell you whether those people are good in various categories. The system just keeps track of itself. The system knows who’s good and who’s bad by the work they’ve already submitted done.

Any last words?

Amazon Mechanical Turk is the core of our business. We are in a people-hungry business. All of our work is done by people. It’s not a mass-production thing; it is individual hand-crafted work. Our entire workforce is Mechanical Turk. There are vast amounts of service-oriented businesses on this planet, and all of these businesses could leverage the sort of hand work that is doable on Mechanical Turk.

You can reach Nathan McFarland at CastingWords by emailing nathan@castingwords.com, or visiting http://castingwords.com .