About This Code Sample
This code sample creates an "Adjudication Engine" which allows a Requester to use plurality to automatically approve or reject work.
- Specifically, this code sample will publish a HIT with 2 Assignments and compare the Worker results. If the results don't match the Adjudication Engine publishes a 3rd Assignment and compares the 3 results to determine which results to accept and which to reject
- Requesters using this code sample may want to change the number of Assignments compared, which answer(s) are compared or which Qualifications a Worker must possess to complete the Assignments
- This code sample includes sample HIT contents for testing. Requesters should modify the sample HIT contents
- This code sample uses the Amazon Mechanical Turk Java SDK found here: http://developer.amazonwebservices.com/connect/entry.jspa?externalID=695&categoryID=193
Features and Benefits
|Automated HIT Creation from
|Answer Comparison Engine
|Automated HIT Extension creates
an additional Assignment for the HIT when Workers disagree
|Automated Adjudication accepts
matching answer and rejects others
|Intelligent Type-Ahead in HIT||
Note: These instructions assume a *nix system. For Windows, change paths as necessary to adhere to Windows file and directory naming standards.
- You will need an Amazon Web Services account. You can sign up at the AWS web site: http://aws.amazon.com
- You will need an Amazon Mechanical Turk Requester account. Be sure to use the same e-mail address and password as you used when creating your Amazon Web Services account. You can sign up at the Requester web site: http://requester.mturk.com
- This code sample was tested with Sun's Java6 SDK, Maven 2, and MySQL 5.
Using the Code Sample Basic Setup
Redirect a public IP address, port 80, to local port 8080 on your web application server hosting this application. This is required for the system to receive assignment update notifications from Amazon Mechanical Turk.
- Create a new MySQL database named mturk2p1
- Adjust the MySQL root password to match the server root in src/main/webapp/WEB-INF/spring.xml
- This project requires artifacts that are not currently hosted in public Maven repositories. After installing Maven 2, install these artifacts, by going into src/main/local and run the "install.sh" script. This should install the specific dependencies into your local repository.
- Edit the contents of src/main/resources/mturk2p1.properties. Set your AWS KEYS and the public IP you configured in the first step.
- Build the project and run the unit tests with "mvn test"
- The test has run successfully when you see data in your Categories table in the MySQL DB. These categories are required reference data for the HITs that you are about to create by running this code sample.
- Launch the system via embedded tomcat. Type "mvn tomcat:run". This will start tomcat on port 8080. You can verify that everything is working by pointing your browser at http://localhost:8080/mturk2p1 (To check your IP redirect configuration, browse to http://[YOURPUBLICIP]:80/mturk2p1). the redirect must function properly inorder for the Adjudication Engine to receive and adjudicate the HIT results.
- Launch test HITs. Copy the test file src/test/data/test.csv to the directory /opt/mturk2p1/in/Appliances. This directory and 17 others were created on your system when the initial categories were loaded from categories.csv. These directories are constantly polled for new input files. When a new file is dropped in one of these directories, the code sample will generate the corresponding HITs, log the data to the db and move the file to the corresponding "processed" directory. In this case, that is /opt/mturk2p1/processed/Appliances.
- Login to the Amazon Mechanical Turk Sandbox, as a Worker, and
search for HITs created by your Requester account. You should see
the HIT you just created. The HIT asks you to categorize a
product. Accept the HIT and answer the question. The answer you
provide will be stored in the database.
- Log out of Amazon Mechanical Turk and log back in as a different Worker. Find the other Assignment for the same HIT. Answer the question with a different answer from what you picked in the previous step. The Adjudication Engine will store the answer in the database, compare the answers and create an additional Assignment for this HIT.
- Once you have submitted this additional Assignment from another Worker account, the Adjudication Engine will again store and compare the answers. Assuming 2 answers match, the Adjudication Engine will accept the matching Assignments and reject the other.
- Additional documentation is included in the sample package. Please review for additional details of installation, configuration and testing.
Copyright 2010 Amazon.com, Inc. or
its affiliates. All Rights Reserved.
Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Portions developed for Amazon.com, Inc. or its affiliates by David Knoernschild at Statera Inc., 2229 112th Ave NE, Suite 100, Bellevue, Washington 98004-2936. For customization or consulting on this code sample, contact Statera at firstname.lastname@example.org.