caqe.configuration module¶
A app configuration defines the user-tunable parameters of the application and also the quality evaluation such as the:
- Amazon Mechanical Turk HIT description, pricing, keywords, etc.
- The description and instructions of the task
- The configuration of the type of test (e.g ‘mushra’ or ‘pairwise’)
- The definition of the quality scales
- The paths to the audio stimuli
- Which components of the evaluation are active (e.g. pre-test survey, post-test survey, hearing screening, etc.)
This subpackage contains a base configuration which contains overridable defaults, as well as pre-defined testing
configurations for common audio quality evaluation scenarios. Make sure that before you run a test that you at least
change the stimuli and the SERVER_ADDRESS
variable.
See also
-
class
caqe.configuration.
BaseConfig
[source]¶ Bases:
object
The base application configuration.
-
DEBUG
¶ bool – Enable/disable debug mode (see Flask docs) (default is False)
-
TESTING
¶ bool – Enable/disable testing mode (see Flask docs) (default is False)
-
SECRET_KEY
¶ str – If a secret key is set, cryptographic components can use this to sign cookies and other things. Set this to a complex random value when you want to use the secure cookie for instance. Set via generate_key_file.py or using environment variable ‘SECRET_KEY’. (see Flask Docs)
-
CSRF_SESSION_KEY
¶ str – A Cross-site Request Forgery (CSRF) secret key for signing data. Set via generate_key_file.py or using environment variable ‘CSRF_SESSION_KEY’. (see Flask docs)
-
CSRF_ENABLED
¶ bool – Enable/disable protection against Cross-site Request Forgery (CSRF) (see Flask docs) (default is True)
-
SERVER_ADDRESS
¶ str – The name and port number of the server. Do not include ‘http’. (e.g.: ‘caqe.local:5000’) (see Flask docs) Can be set via environment variable ‘SERVER_ADDRESS’. (default is ‘caqe.local:5000’)
-
SQLALCHEMY_DATABASE_URI
¶ str – The database URI that should be used for the connection (see Flask-SQLAlchemy docs). Examples: * sqlite:////tmp/test.db * mysql://username:password@server/db Can be set via environment variable ‘DATABASE_URL’. (default is’sqlite:////~/caqe.db’)
-
PREFERRED_URL_SCHEME
¶ str – The URL scheme that should be used for URL generation if no URL scheme is available. ‘http’ or ‘https’ (default is ‘https’)
-
AUDIO_FILE_DIRECTORY
¶ str – Relative directory path to testing audio stimuli. (default is ‘static/audio’)
-
ENCRYPT_AUDIO_STIMULI_URLS
¶ bool – Enable/disable encryption of the URLs so that users can’t game consistency. (default is True)
-
TEST_TYPE
¶ str – The test type (limited to ‘pairwise’ or ‘mushra’ for now). (default is None)
-
ANONYMOUS_PARTICIPANTS_ENABLED
¶ bool – Enable/disable participants to enter through ‘/anonymous’ entry point. (default is False)
-
IP_COLLECTION_ENABLED
¶ bool – Enable/disable collection participants’ IP addresses. (default is True)
-
OBTAIN_CONSENT
¶ bool – If True, obtain consent from each participant (see consent.html) (default is True)
-
PRE_TEST_SURVEY_ENABLED
¶ bool – If True, ask participants a survey before evaluation (see pre_test_survey.html). (default is True)
-
PRE_TEST_SURVEY_INCLUSION_CRITERIA
¶ list of str – Pre-test survey inclusion criteria. (default is [“int(survey[‘age’]) >= 18”, “survey[‘hearing_disorder’] == ‘No’”])
-
POST_TEST_SURVEY_ENABLED
¶ bool – If True, ask participants a survey after evaluation (see post_test_survey.html) (default is True)
-
HEARING_RESPONSE_ESTIMATION_ENABLED
¶ bool – If enabled, ask participants to complete the in-situ hearing response estimation. (default is True)
-
CONDITIONS_PER_EVALUATION
¶ int – The number of conditions to present to a participant in a single visit to ‘/evaluate’. Note that currently evaluation is limited to one condition group. So if this value is more than 1, there must be at least as many conditions per group as there are conditions per evaluation for this to have an effect. It is also recommended that an integer multiple of CONDITIONS_PER_EVALUATION comprise the number of conditions per group. For example, if there are 28 conditions in a group, set the number of CONDITIONS_PER_EVALUATION to 14 or 7. (default is 1)
-
TRIALS_PER_CONDITION
¶ int – The number of trials we should collect per condition (with distinct participants). (default is 20)
-
LIMIT_SUBJECT_TO_ONE_TASK_TYPE
¶ bool – If True, each subject is limited to one type of Test. (default is True)
-
TEST_CONDITION_ORDER_RANDOMIZED
¶ bool – Randomize the condition order per test for each participant. (default is True)
-
STIMULUS_ORDER_RANDOMIZED
¶ bool – Randomize the stimulus order per for each condition. (default is True)
-
HEARING_SCREENING_TEST_ENABLED
¶ bool – Set to True if you want the participants to be required to take a hearing screening test. (default is True)
-
HEARING_TEST_EXPIRATION_HOURS
¶ int – The number of hours their hearing test is valid for (they must retake after this time has passed) (default is 24)
-
MAX_HEARING_TEST_ATTEMPTS
¶ int – The number of attempts one has before they are sent away (they must wait hearing_test_expiration_hours to take it again) (default is 2)
-
HEARING_TEST_REJECTION_ENABLED
¶ bool – If this is set to True, then we still test the users, but we don’t reject them. (default is True)
-
HEARING_RESPONSE_NOPTIONS
¶ int – Max number of frequencies for user to respond with in hearing response estimation. (default is 20)
-
MTURK_HOST
¶ str – Amazon Mechanical Turk host location. By default set it to the sandbox, and configure it via an environment variable (so, it can be easily modified when deploying and testing using Heroku). Can be set via environment variable ‘MTURK_HOST’. (default is ‘mechanicalturk.sandbox.amazonaws.com’)
-
MTURK_QUESTION_URL
¶ str – Entry point URL. (default is ‘https://%s/mturk‘ % SERVER_ADDRESS)
-
MTURK_REWARD
¶ float – This is the reward given to each worker for an approved assignment (in USD) (note that Amazon takes their Mechanical Turk Fee on top of this. See https://requester.mturk.com/pricing) (default is 0.50)
-
MTURK_FIRST_HIT_BONUS
¶ float – The default bonus reward in USD that is optionally given (using
turk_admin_cli.py
) to participants that completed the first assignment, which may have additional testing (e.g. survey, hearing tests, etc.) (default is 0.30)
-
MTURK_MAX_CONSISTENCY_BONUS
¶ float – The defualt maximum bonus reward in USD for pairwise consistency. This optional bonus is given using
turk_admin_cli.py
. (default is 0.25)
-
MTURK_MIN_CONSISTENCY_THRESHOLD_FOR_BONUS
¶ float – The minimum pairwise consistency required to receive the optional bonus (given through
turk_admin_cli.py
.) (default is 0.7)
-
MTURK_NUMBER_HITS_APPROVED_REQUIREMENT
¶ int – MTurk worker must have this many approved HITs to accept task. (default is 1000)
-
MTURK_PERCENT_ASSIGNMENTS_APPROVED_REQUIREMENT
¶ int – MTurk worker must have this percentage of approved assignments to accept task. (default is 97)
-
MTURK_TITLE
¶ str – Title of MTurk HIT (default is ‘Critical audio listening task. Listen to audio recordings and rate them on various scales of quality.’)
-
MTURK_DESCRIPTION
¶ str – Description of MTurk HIT. (default is ‘This listening test aims to rate the quality of a set of signals in comparison to a reference signal. Note that while the maximum number assignments a worker can do is 10, it’s possible that fewer than 10 may be available to you. **CHROME ONLY** **BONUS AVAILABLE**’)
-
MTURK_KEYWORDS
¶ str – Keywords for MTurk HIT. (default is ‘audio, sound, music, listening, research’)
-
MTURK_ASSIGNMENT_DURATION_IN_SECONDS
¶ int – Accepted MTurk assignments must be completed within this duration or they will be released to other workers (default is 60 * 30, i.e. 30 minutes)
-
MTURK_LIFETIME_IN_SECONDS
¶ int – HITs expire (no one can accept them) after this duration since being posted. (default is 60 * 60 * 24 * 7, i.e 1 week)
-
MTURK_FRAME_HEIGHT
¶ int – The size of the Mechanical Turk browser frame (default is 1200)
-
ACCEPTABLE_BROWSERS
¶ list of str – The set of acceptable browsers. set as None to disable browser rejection. (default is [‘chrome’,])
-
BEGIN_BUTTON_ENABLED
¶ bool – If true, participants will have to click a button that launches a new window. This is useful in order to delay condition assignment until the user is engaged in the task, and allows a new window to be launched that is bigger than the Mechanical Turk frame for instance. (default is True)
-
POPUP_WIDTH
¶ int – The width of the window launched when participants press the “begin button” the task. (default is 1200)
-
POPUP_HEIGHT
¶ int – The height of the window launched when participants press the “begin button” the task. (default is 1200)
-
TEST_TIMEOUT_SEC
¶ float – The participant must spend at least this amount of time on the evaluation task before submission. (default is 60.)
-
REQUIRE_LISTENING_TO_ALL_TRAINING_SOUNDS
¶ bool – If True, the participant must listen to all of the training sounds before proceeding to the evaluation task. (default is True)
-
PREVIEW_HTML
¶ str – The HTML content of the preview page. This will be the same for all conditions, regardless of test since conditions are assigned on the fly (so we can have complete control over condition assignment). (default is None)
-
MIN_RATING_VALUE
¶ int – The minimum rating value on the MUSHRA slider. (default is 0)
-
MAX_RATING_VALUE
¶ int – The maximum rating value on the MUSHRA slider. (default is 99)
-
DEFAULT_RATING_VALUE
¶ int – The default rating value on the MUSHRA slider. (default is 50)
-
TESTS
¶ list of dict – The test and condition-specific configuration variables. Note that if ‘evaluation_instructions_html’ is not None in the condition, it will override the instructions defined in the test. Note also that reference keys must be alphanumeric and stimulus keys must begin with ‘S’ followed by a number, e.g. ‘S29’.
The dicts are of the form:
{'test_config_variables': {'test_title': '...', # The test title that is displayed on the evaluation page 'first_task_introduction_html': '...', # Content of the intro page the first time they do a task 'introduction_html': '...', # Content of the intro page (after the first time they perform the task) 'training_instructions_html': '...', # The HTML content of the training instructions 'evaluation_instructions_html': '...'}, # The HTML content of the evaluation instructions 'references' : (('<reference_name>', '<reference_description>'),), # Reference names and descriptions 'reference_example_dict': {'<reference_name}': url_for('static', filename='audio/<reference_filename>.wav'), ... }, 'quality_example_dict': {'<example_type0>': [url_for('static', filename='audio/<example0_filename>.wav'), url_for('static', filename='audio/<example1_filename>.wav'),], '<example_type1>': [url_for('static', filename='audio/<example3_filename>),]}}, 'condition_groups' : [{'reference_files': {<reference_name>: '<reference_filename>.wav',}, {'stimulus_files': {'S1': '<S1_filename>.wav', 'S2': '<S2_filename>,wav',}}, {'conditions': [{'reference_keys': [<reference_name>,], 'stimulus_keys': ['S1','S2','S7', ... ], 'evaluation_instructions_html': <condition_specific_evaluation_instructions>},]},]}
(default is [])
Note
For testing, add:
0.0.0.0 caqe.local
to /etc/hosts We need to set the SERVER_ADDRESS to resolve
url_for
definitions when constructing the database, but we can’t simply use localhost because the secure sessions are not compatible with that.-
-
class
caqe.configuration.
TestingOverrideConfig
[source]¶ Bases:
object
Override config for testing.
Note
To enable these parameters set environment variable
APP_MODE
to ‘TESTING’. In Linux:$ export APP_MODE=TESTING
-
DEBUG
= True¶
-
MTURK_QUESTION_URL
= 'https://caqe.local:5000/mturk'¶
-
PREFERRED_URL_SCHEME
= 'http'¶
-
SERVER_ADDRESS
= 'caqe.local:5000'¶
-
SQLALCHEMY_DATABASE_URI
= 'sqlite://'¶
-
TESTING
= True¶
-
-
class
caqe.configuration.
DevelopmentOverrideConfig
[source]¶ Bases:
object
Override config for development.
Note
To enable these parameters set environment variable
APP_MODE
to ‘DEVELOPMENT’. In Linux:$ export APP_MODE=DEVELOPMENT
-
DEBUG
= True¶
-
HEARING_TEST_REJECTION_ENABLED
= False¶
-
MTURK_QUESTION_URL
= 'https://caqe.local:5000/mturk'¶
-
PREFERRED_URL_SCHEME
= 'http'¶
-
REQUIRE_LISTENING_TO_ALL_TRAINING_SOUNDS
= False¶
-
SERVER_ADDRESS
= 'caqe.local:5000'¶
-