Running the Evaluation on Amazon’s Mechanical Turk¶

In order to collect data quickly, we recommend recruiting participants for your audio evaluation by posting HITs (Human Information Tasks) to Amazon’s Mechanical Turk (MTurk), which is a microtask labor market. On this platform it is recommended that you pay workers at least $6.00 / hr (see MTurk Guidelines for Academic Requesters). On top of that Amazon also takes a fee. This works out to be comparable to what you would pay participants in a lab study, but you can obtain your data much more quickly and with less effort.

Recruiting Participants¶

To recruit participants to evaluate our audio, we post assignments on Mechanical Turk. For each assignment, a participant will perform a single trial (the evaluations associated with one condition as defined in the config, e.g. one MUSHRA)

Create an account on MTurk
Create an Amazon AWS IAM user with AmazonMechanicalTurkFullAccess and put your this user’s credentials into your secret_keys.py file (under CAQE/src/caqe).
1. Log into Amazon AWS
2. Click on Identity & Access Management under Amazon Web Services.
3. Click on Users on the side-panel menu.
4. Click on Create New Users
5. Enter a name for your CAQE application and press Create.
6. Click Show User Security Credentials and copy those keys into the secret_keys.py file that you generated earlier using generate_key_file.py.
7. Click on Policies on the side-panel menu.
8. Check AmazonMechanicalTurkFullAccess, click the Policy Actions button at the top, and click Attach.
9. Attach the policy to the user you just created.
Configure the MTurk variables in your config file. These are all variables that have the MTURK prefix. Use these variables to set your HIT’s title, description, reward, etc. See caqe.configuration module for more information.
Set APP_MODE to PRODUCTION:
```
$ export APP_MODE=PRODUCTION
```
Set the SERVER_ADDRESS environment variable to your production server, e.g.
```
$ export SERVER_ADDRESS=<your-caqe-app>.herokuapp.com
```
After your variables are configured, post your HIT to the MTurk sandbox for testing. With the --debug flag HITs are created without the worker qualification requirements so that you can test them in the sandbox.
```
$ python turk_admin_cli.py --debug create-hits <num_hits>
```
Your HITs will now be viewable on the Mechanical Turk sandbox site. Visit http://workersandbox.mturk.com and search for the title you provided in your configuration. You can expire them with turk_admin_cli.py:
```
$ python turk_admin_cli.py expire-all-hits
```
Once you are ready to post your HITs to production Mechanical Turk Site, clear out your database. E.g., for Heroku:
```
$ heroku run python src/create_db.py
```
set the MTURK_HOST environment variable:
```
$ export MTURK_HOST='mechanicalturk.amazonaws.com'
```
and then again use turk_admin_cli.py to create HITs:
```
$ python turk_admin_cli.py create-hits <num_hits>
```
You can view the progress of your evaluation at http://your-caqe-app.com/admin/stats

Note

If you need to end the HIT early (e.g. you made a mistake or you have enough data), you can expire the hits:

$ python turk_admin_cli.py expire-all-hits

Paying Participants¶

After a HIT is complete, you must pay participants in a timely manner. The MTURK_AUTO_APPROVAL_DELAY_IN_SECONDS config variable sets the auto-approval window. MTURK_AUTO_APPROVAL_DELAY_IN_SECONDS seconds after a participant submits a HIT, they will automatically be paid the reward of MTURK_REWARD in USD.

If you need to approve HIT assignments and pay participants before this amount of time passes, you can issue the following command:

$ python turk_admin_cli.py approve-all-hits

After HIT assignment have been approved you can bonus participants. If you are asking the participant to perform a hearing screening or survey on their first HIT assignment, you should pay them for that extra time in the form of a bonus. To assign a bonus for all completed first assignments. (Note the bonus amount is set by default by the MTURK_FIRST_HIT_BONUS config variable):

$ python turk_admin_cli.py give-first-trial-bonus <bonus_amt>

Note

If you are using the general_mushra.cfg, sourceseparation_mushra.cfg, general_pairwise.cfg, or sourceseparation_pairwise.cfg configurations, workers are told in the description that they will receive this bonus. Therefore, make sure to either pay the bonus or change description and instructions in the configuration.

If you are performing a pairwise-comparison listening test, you can bonus participants based on their pairwise consistency. This is a good incentive for workers. (see python turk_admin_clip.y give-pairwise-consistency-bonus --help and the MTURK_MAX_CONSISTENCY_BONUS, MTURK_MIN_CONSISTENCY_THRESHOLD_FOR_BONUS for more details):

$ python turk_admin_cli.py give-pairwise-consistency-bonus

Note

If you are using the general_pairwise.cfg or sourceseparation_pairwise.cfg configurations, workers are told in the description that they will receive this bonus. Therefore, make sure to either pay the bonus or change description and instructions in the configuration.

Once you have paid the workers and downloaded the data (see Downloading and Analyzing Data), you can dispose of the HITs:

$ python turk_admin_cli.py dispose-all-hits