Amazon Mechanical Turk (MTurk) provides an on-demand, scalable, human workforce to complete jobs that humans can do better than computers. MTurk formalizes job offers to the thousands of Workers willing to do piecemeal work at their convenience.—MTurk User Guide
More specifically, MTurk is an online platform where you can run your own surveys (or experiments).
From: Baese-Berk & Morrill (2016)
From: Gao & Weinberger (2018)
1. Replicability of Lab-setting experiments
MTurk experiments replicated results of the same linguistic/psycholinguistic experiments conducted in Lab-settings. (Enochson & Culbertson, 2015).
We do however need more participants if the experiment is conducted online.
2. Workers
MTurk termed survey/experiment participants as “Workers” (Bewared of legal implications). About 2,000 participants are active on MTurk at any given time. 51% of them are female, 49% of them are male. About 75% of the participants are from the United States. Indian participants represent 16% of the population. The rest are from Canada, Great Britain, Philippines and Germany. (Difallah et al., 2018)
Hardly any non-English speaking monolinguals are on the MTurk platform.
Sign in as a Requester
Sign in : https://requester.mturk.com/
Let’s take a look at the existing templates. Click the “New Project”" tag.
Problems with the project templates: 1. cannot randomize stimuli (that’s a big problem to any psycholinguistic studies) 2. customization issues (e.g., ISI, Reaction Time, coloring etc.)
Sign in as a Worker
Sign in https://worker.mturk.com
Now we are logged in as a worker, which means we can participate in the experiments. Let’s take a look some real-time MTurk HITs (i.e., experiments).
S3 Storage
This service provides storage for hosting ‘static’ websites, including those using JavaScript.
Activate the S3 storage service on AWS. It is Amazon’s Dropbox service. The difference is that the S3 storage also proivde some basic web-hosting functions. We are going to upload our experiments there and link the experiments to MTurk.
MTurk experiments are conducted online. Perception studies are the most suitable. Although it is possible to collect speech samples from participants, the quality of the recordings cannot be guaranteed. This tutorial only introduces a perception experiment.
Program the experiments in HTML (CSS+Javascript), becuase HTML is universally supported by all browsers. Flash files are not.
Here’s an example
The basic layout of a perception experiment includes
an introduction page, telling potentially participants what the experiment is about, how long it takes, and how much you are going to pay them.
general instruction of the experiment
the experiment
demographic survey.
Check BeginExperiment.js
Detailed description of the task. Avoid jargons!
Check the jQuery package for possible inputs (e.g., rating scale, box input, buttons etc.)
Age, Gender, L1/L2, Speech/hearing disorder, etc.
Include:
Click HERE for methods of installing
Once you got it installed, you need to configure it using your AWS security credentials, which means you need to register an AWS account on Amazon (Have your credit card ready). Once that’s done, you will be able to get the security credentials.
type the credentials in your aws shell
$ aws-shell
aws> configure
AWS Access Key ID [None]: your-access-key-id
AWS Secret Access Key [None]: your-secret-access-key
Default region name [None]: region-to-use (e.g us-west-2, us-west-1, etc).
Default output format [None]:
aws>mturk create-hit --endpoint-url https://mturk-requester-sandbox.us-east-1.amazonaws.com \
--max-assignments 9 \
--auto-approval-delay-in-seconds 86400 \
--lifetime-in-seconds 7200 \
--assignment-duration-in-seconds 3600 \
--reward 1.00 \
--title "GMU Demo zgao" \
--description "Demo Experiment ZGao." \
--question '
<ExternalQuestion
xmlns="
http://mechanicalturk.amazonaws.com/AWSMechanicalTurkDataSchemas/2006-07-14/ExternalQuestion.xsd">
<ExternalURL>
https://s3.amazonaws.com/fap2018/demo/mturk.html?condition=${condition}&list=${list
}</ExternalURL>
<FrameHeight>600</FrameHeight>
</ExternalQuestion>
'
The “–endpoint-url https://mturk-requester-sandbox.us-east-1.amazonaws.com " means we are testing the experiment in the sandbox.
Remove this line if you want to do it for real.
--qualification-requirements
"QualificationTypeId=00000000000000000071,
Comparator=EqualTo,LocaleValues=[{Country=US,Subdivision=VA}]"
The AWS shell also allows
check HERE for details
References:
Baese-Berk, M. M., Morrill, T. H., & Bradlow, A. R. (2016). Intelligibility, fluency, and variability in non-native speech. The Journal of the Acoustical Society of America, 139(4), 2162-2162.
Culbertson, J., & Adger, D. (2014). Language learners privilege structured meaning over surface frequency. Proceedings of the National Academy of Sciences, 111(16), 5842-5847.
Difallah, Djellel, Elena Filatova, and Panos Ipeirotis (2018). Demographics and Dynamics of Mechanical Turk Workers. Proceedings of the 18th ACM International Conference on Web Search and Data Mining (WSDM), 135–143
Enochson, K., & Culbertson, J. (2015). Collecting psycholinguistic response time data using Amazon Mechanical Turk. PloS one, 10(3), e0116946.
Gao, Z., & Weinberger, S. (2018). Which phonetic features should pronunciation Instructions focus on? An evaluation on the accentedness of segmental/syllable errors in L2 speech. Research in Language, 16(2), 135-154.