caqe.turk_admin module¶

Amazon Mechanical Turk administration. Use this module to post, approve, expire, and bonus HITs.

class caqe.turk_admin.TurkAdmin(debug=False)[source]¶

Bases: object

Instantiate this class to connect to MTurk and perform administrative tasks.

approve_all(hit_types=None)[source]¶

Approve all ‘Submitted’ assignments

Parameters:	hit_types (list of str, optional) –
Returns:
Return type:	None

create_hits(num_hits, configuration=None, hit_type_id=None)[source]¶

Create num_audio_hits according to the parameters specified in hit_params

Parameters:	num_hits (int) – configuration (dict) – hit_type_id (int, optional) –
Returns:
Return type:	None

dispose_all_hits()[source]¶

Dispose all hits

Returns:
Return type:	None

dispose_hits(hit_types=None)[source]¶

Dispose all hits whose HITTypeId is in hit_types (Disposing a HIT removes it from the system).

Parameters:	hit_types (list of str, optional) –
Returns:
Return type:	None

expire_all_hits()[source]¶

Expire all hits

Returns:
Return type:	None

expire_hits(hit_types=None)[source]¶

Expire all hits whose HITTypeId is in hit_types

Parameters:	hit_types (list of str, optional) –
Returns:
Return type:	None

filter_hits(hits, hit_types=None)[source]¶

Return the hits whose type is in hit_types

Parameters:	hits (list of boto.HIT) – hit_types (list of str) –
Returns:
Return type:	list of boto.HIT

force_approve_all()[source]¶

Approve all ‘Submitted’ assignments

Returns:
Return type:	None

get_all_assignments()[source]¶

Get all assignments regardless of HIT type

Returns:	assignments
Return type:	list of boto.Assignment

get_all_assignments_to_review(hit_type, status=('Submitted', 'Approved', 'Rejected'))[source]¶

Get all the assignments to review for the specified HIT type

Parameters:	hit_type (str) – status (str, optional) – Assignment status. Valid values are ‘Submitted’, ‘Approved’, ‘Rejected’.
Returns:	assignments
Return type:	list of boto.Assignment

get_assignments_to_review(hit_type, page_size=100, page_number=1)[source]¶

Get assignments to review for the specified HIT type given page size and page number

Parameters:	hit_type (str) – HITTypeId page_size (int) – How many assignments to return page_number (int) – What page of assignments to return
Returns:	assignments
Return type:	list of boto.Assignment

get_completion_times(assignments=None)[source]¶

Compute completion time of assignments. The completion time is the time between when the HIT was accepted and submitted.

Parameters:	assignments (list of Assignment, optional) – The list of assignments to compute average completion time. If None, then compute on all assignments. Default is None.
Returns:	times – The list of completion times in seconds.
Return type:	list of float

give_bonus_to_all_first_completed_trials(price=0.3, calculate_amt_only=False, reason=None, already_bonused_ids=set([]))[source]¶

Grant bonuses for the first completed trial for each participant.

Parameters:	price (float, optional) – The bonus amount to grant in dollars. Default is defined by the CAQE configuration. calculate_amt_only (bool, optional) – Only calculate the amount of the bonus, do not actual pay out the bonus. reason (str, optional) – The message to send the workers when they receive the bonus already_bonused_ids (set, optional) – Set of participant ids that have already been bonused
Returns:	total_bonus (float) – The total amount paid participants_wo_valid_asgnmts (list of caqe.models.Participant) – The participants who did not have valid assignments in their trial data (e.g. there must have been an error when submitting the assignment)

give_consistency_bonus(max_price=0.25, threshold=0.7, calculate_amt_only=False, reason=None, already_bonused_ids=set([]))[source]¶

Grant bonuses based on ratings consistency. Bonus calculated by

\[((consistency - threshold) / (1.0 - threshold)) * max\_price * (consistency > threshold))\]

Parameters:

max_price (float, optional) – The maximum bonus amount to grant in dollars. Default is defined by the CAQE configuration. threshold (bool) – Consistency must exceed this value before a bonus is paid out. Default is defined by the CAQE configuration. calculate_amt_only (bool, optional) – Only calculate the amount of the bonus, do not actual pay out the bonus. reason (str, optional) – The message to send the workers when they receive the bonus already_bonused_ids (set, optional) – Set of participant ids that have already been bonused

Returns:

total_bonus (float) – The total amount paid participants_wo_valid_asgnmts (list of caqe.models.Participant) – The participants who did not have valid assignments in their trial data (e.g. there must have been an error when submitting the assignment)

hit_type_id¶

register_hit(configuration=None)[source]¶

Register a hit on Mechanical Turk according to hit_params. This will provide you with a HITTypeId.

Parameters:	configuration (dict) –
Returns:	The HITTypeId which is how you refer to your newly registered hit with Amazon
Return type:	str

caqe.turk_admin.calculate_tsr(ratings, stimuli=('S1', 'S2', 'S3', 'S4', 'S5', 'S6', 'S7', 'S8'))[source]¶

Calculate the Transitivity Satisfaction Rate (TSR) for a group of ratings.

Parameters:	ratings (dict) – Ratings dictionary stimuli (tuple of str) – Tuple of stimulus identifiers in order.
Returns:	The TSR.
Return type:	float

caqe.turk_admin.turk_connect()[source]¶

Connect to Mechanical Turk and return a connection. This uses AWS_ACCESS_KEY_ID and AWS_SECRET_KEY from secret_keys.py (you must put these in yourself).

Returns:
Return type:	boto.MTurkConnection