Exploring Automated Adjudication Workflow in Amazon Mechanical Turk Java-Based Code Sample

items>Exploring Automated Adjudication Workflow in Amazon Mechanical Turk Java Based Code Sample
Community Contributed Software

  • Amazon Web Services provides links to these packages as a convenience for our customers, but software not authored by an "@AWS" account has not been reviewed or screened by AWS.
  • Please review this software to ensure it meets your needs before using it.


Submitted By: CJ@AWS
AWS Products Used: Amazon Mechanical Turk
Language(s): Java
License: Apache License 2.0
Created On: March 2, 2010 2:09 AM GMT
Last Updated: March 2, 2010 6:38 PM GMT

About This Code Sample

This code sample creates an "Adjudication Engine" which allows a Requester to use plurality to automatically approve or reject work.

  • Specifically, this code sample will publish a HIT with 2 Assignments and compare the Worker results. If the results don't match the Adjudication Engine publishes a 3rd Assignment and compares the 3 results to determine which results to accept and which to reject
  • Requesters using this code sample may want to change the number of Assignments compared, which answer(s) are compared or which Qualifications a Worker must possess to complete the Assignments
  • This code sample includes sample HIT contents for testing. Requesters should modify the sample HIT contents
  • This code sample uses the Amazon Mechanical Turk Java SDK found here: http://developer.amazonwebservices.com/connect/entry.jspa?externalID=695&categoryID=193

Features and Benefits

Automated HIT Creation from input file
  • Simplifies integration into existing system and processes
  • Allows upstream processes to request HITs be created by placing the HIT data into an input directory
Answer Comparison Engine
  • Eliminates the need to manually review HIT answers by asking multiple Workers to complete the HIT and comparing their answers
Automated HIT Extension creates an additional Assignment for the HIT when Workers disagree
  • Improves accuracy by automatically asking an additional Worker to complete the HIT to resolve disagreement between original Workers
Automated Adjudication accepts matching answer and rejects others
  • Lowers costs by only paying for answers that match
  • Improves accuracy by rejecting work from Workers whose answers don't match
  • Eliminates the need for manual adjudication
Intelligent Type-Ahead in HIT
  • Allows Workers to select from list of previously provided keywords matching their free-format text input
  • Normalizes Worker free-form typed responses


Note:  These instructions assume a *nix system.  For Windows, change paths as necessary to adhere to Windows file and directory naming standards.

  • You will need an Amazon Web Services account. You can sign up at the AWS web site: http://aws.amazon.com
  • You will need an Amazon Mechanical Turk Requester account. Be sure to use the same e-mail address and password as you used when creating your Amazon Web Services account. You can sign up at the Requester web site: http://requester.mturk.com
  • This code sample was tested with Sun's Java6 SDK, Maven 2, and MySQL 5.

Using the Code Sample — Basic Setup

Redirect a public IP address, port 80, to local port 8080 on your web application server hosting this application.  This is required for the system to receive assignment update notifications from Amazon Mechanical Turk.

  • Create a new MySQL database named mturk2p1
  • Adjust the MySQL root password to match the server root in src/main/webapp/WEB-INF/spring.xml
  • This project requires artifacts that are not currently hosted in public Maven repositories. After installing Maven 2, install these artifacts, by going into src/main/local and run the "install.sh" script. This should install the specific dependencies into your local repository.
  • Edit the contents of src/main/resources/mturk2p1.properties. Set your AWS KEYS and the public IP you configured in the first step.
  • Build the project and run the unit tests with "mvn test"
  • The test has run successfully when you see data in your Categories table in the MySQL DB.  These categories are required reference data for the HITs that you are about to create by running this code sample.
  • Launch the system via embedded tomcat.  Type "mvn tomcat:run".  This will start tomcat on port 8080.  You can verify that everything is working by pointing your browser at http://localhost:8080/mturk2p1 (To check your IP redirect configuration, browse to http://[YOURPUBLICIP]:80/mturk2p1). the redirect must function properly inorder for the Adjudication Engine to receive and adjudicate the HIT results.
  • Launch test HITs. Copy the test file src/test/data/test.csv to the directory /opt/mturk2p1/in/Appliances.  This directory and 17 others were created on your system when the initial categories were loaded from categories.csv.  These directories are constantly polled for new input files.  When a new file is dropped in one of these directories, the code sample will generate the corresponding HITs, log the data to the db and move the file to the corresponding "processed" directory.  In this case, that is /opt/mturk2p1/processed/Appliances.
  • Login to the Amazon Mechanical Turk Sandbox, as a Worker, and search for HITs created by your Requester account.  You should see the HIT you just created.  The HIT asks you to categorize a product.  Accept the HIT and answer the question. The answer you provide will be stored in the database.
  • Log out of Amazon Mechanical Turk and log back in as a different Worker.  Find the other Assignment for the same HIT.  Answer the question with a different answer from what you picked in the previous step. The Adjudication Engine will store the answer in the database, compare the answers and create an additional Assignment for this HIT.
  • Once you have submitted this additional Assignment from another Worker account, the Adjudication Engine will again store and compare the answers.  Assuming 2 answers match, the Adjudication Engine will accept the matching Assignments and reject the other.   

Additional Documentation

  • Additional documentation is included in the sample package.  Please review for additional details of installation, configuration and testing.

Legal Notice

Copyright 2010 Amazon.com, Inc. or its affiliates. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at  http://aws.amazon.com/apache2.0/ or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Portions developed for Amazon.com, Inc. or its affiliates by David Knoernschild at Statera Inc., 2229 112th Ave NE, Suite 100, Bellevue, Washington 98004-2936. For customization or consulting on this code sample, contact Statera at codesupport@statera.com.

©2017, Amazon Web Services, Inc. or its affiliates. All rights reserved.