Review of Amazon Mechanical Turk

April, 2013

Rating: 5/5

I recommend Amazon Mechanical Turk for business problems that can be broken up into a large number of very small tasks. Amazon Mechanical Turk is the leading crowdsourcing marketplace for small tasks.

Amazon Mechanical Turk is a market for “Human Intelligence Tasks” or HITs. These are very small tasks which are usually valued at cents each - although some are a few dollars, some are also less than one cent.

You break your work down into 100’s or 1,000’s of HITs, and post them on Amazon Mechanical Turk. Workers will choose to perform your tasks (assuming the rates are competitive), then you review and approve their work. Many different workers can work on your HITs at the same time.

What sort of work?

I recommend using Amazon Mechanical Turk for:

  • Simple tasks
  • Gathering data
  • De-duplication or data cleansing
  • Classifying and categorising - e.g. images, products, services
  • Data entry
  • Transcription of audio to text
  • Simple or short writing tasks
What not to use it for

I don’t recommend Amazon Mechanical Turk for:

  • Any confidential tasks or data
  • More complex or skilled work - instead have a look at some of the freelance marketplaces like Freelancer or oDesk
  • Larger tasks which can't be broken down into tiny pieces. You should be able to complete a HIT in seconds or minutes - not days
My recommendations

First, if you plan to use Amazon Mechanical Turk, remember that you are dealing with real people - many who use Mechanical Turk as their primary source of income. It’s important to communicate well, be fair and honest, and make sure that you respond quickly.

Second, don’t limit your HITs for Mechanical Turk Masters, unless you have a good reason. Amazon offers a Mechanical Turk Master qualification, which is awarded to the top Mechanical Turk workers (based on Amazon’s statistical monitoring). While it might be appealing to offer work to only those with Master status, in my opinion the standard and benefit of Masters qualifications isn’t clear. There are a huge number of excellent workers who aren’t Masters.

Finally, it’s important to start out well, and get a good reputation. Try to break down work into very small tasks - you may find that the some workers are unlikely to accept larger tasks unless you have a good track-record. While you can reject work that doesn’t meet your standard - make sure you only reject work if you’ve got a good reason to do so.

Summary

I recommend Amazon Mechanical Turk - it’s a great crowdsourcing marketplace. Before spending too much time or money, I recommend you put a small limit (say $5 or $10) and use some low risk tasks to trial the service.

The Sales Pitch

Mechanical Turk is a marketplace for work.

We give businesses and developers access to an on-demand, scalable workforce. Workers select from thousands of tasks and work whenever it’s convenient.

As a Mechanical Turk Worker you: * Can work from home * Choose your own work hours * Get paid for doing good work

As a Mechanical Turk Requester you: * Have access to a global, on-demand, 24 x 7 workforce * Get thousands of HITs completed in minutes * Pay only when you’re satisfied with the results

Features

On-demand scalable workforce, multiple workers on the same tasks simultaneously, Mechanical Turk Masters (workers with high approval ratings), Marketplace for Human Intelligence Tasks

Customers

Acxiom Corporation, AOL, CastingWords, Channel Intelligence, DARPA, SnapMyLife, Statera, Tagasaurius, US Army Research Laboratory

Tech Information
Launch Date
2005
Total Payments
Large - (total unknown)
Number of Providers
Large - More than 500,000 workers from 190 countries
Specialities
Small Human Intelligence Tasks, Data cleansing (verification, deduplication, data entry, collection, algorithm training), Categorisation, Get feedback (usability, search relevancy), Create and moderate content (photos, transcription)