The Experiment Class

Experiments are designed in Dallinger by creating a custom subclass of the base Experiment class. The code for the Experiment class is in experiments.py. Unlike the other classes, each experiment involves only a single Experiment object and it is not stored as an entry in a corresponding table, rather each Experiment is a set of instructions that tell the server what to do with the database when the server receives requests from outside.

class dallinger.experiment.Experiment(session=None)

Define the structure of an experiment.

verbose

Boolean, determines whether the experiment logs output when running. Default is True.

task

String, the name of the experiment. Default is “Experiment title”.

session

session, the experiment’s connection to the database.

practice_repeats

int, the number of practice networks (see role). Default is 0.

experiment_repeats

int, the number of non practice networks (see role). Default is 0.

experiment_routes

Flask Blueprint for experiment. Functions and methods on the class should be registered as Flask routes using the experiment_route() decorator. Route functions can not be instance methods and should either be plain functions or classmethods. You can also register route functions at the module level using the standard route decorator on this Blueprint.

recruiter

Reference to a Recruiter, the Dallinger class that recruits participants.

initial_recruitment_size

int, the number of participants requested when the experiment first starts. Default is 1.

known_classes

dictionary, the classes Dallinger can make in response to front-end requests. Experiments can add new classes to this dictionary.

participant_constructor

Constructor for Participant objects. Callable returning an instance of Participant or a sub-class. Used by create_participant().

hidden_dashboards

Sequence of dashboard route/function names that should be excluded from rendering as tabs in the dashboard view.

public_properties

dictionary, the properties of this experiment that are exposed to the public over an AJAX call

__init__(session=None)

Create the experiment class. Sets the default value of attributes.

add_node_to_network(node, network)

Add a node to a network.

This passes node to add_node().

assignment_abandoned(participant)

What to do if a participant abandons the hit.

This runs when a notification from AWS is received indicating that participant has run out of time. Calls fail_participant().

assignment_reassigned(participant)

What to do if the assignment assigned to a participant is reassigned to another participant while the first participant is still working.

This runs when a participant is created with the same assignment_id as another participant if the earlier participant still has the status “working”. Calls fail_participant().

assignment_returned(participant)

What to do if a participant returns the hit.

This runs when a notification from AWS is received indicating that participant has returned the experiment assignment. Calls fail_participant().

attention_check(participant)

Check if participant performed adequately.

Return a boolean value indicating whether the participant’s data is acceptable. This is mean to check the participant’s data to determine that they paid attention. This check will run once the participant completes the experiment. By default performs no checks and returns True. See also data_check().

attention_check_failed(participant)

What to do if a participant fails the attention check.

Runs when participant has failed the attention_check(). By default calls fail_participant().

bonus(participant)

The bonus to be awarded to the given participant.

Return the value of the bonus to be paid to participant. By default returns 0.

bonus_reason()

The reason offered to the participant for giving the bonus.

Return a string that will be included in an email sent to the participant receiving a bonus. By default it is “Thank you for participating! Here is your bonus.”

calculate_qualifications(participant)

All the qualifications we want to assign to a worker.

This default implementation produces qualifications compatible with Dallinger’s standard recruiters, and the MTurkRecruiter in particular.

Workers will always be assigned one qualification specific to the experiment run. If a “group_name” config value is set, this will be parsed for additional qualifications to grant.

Return type is a list of dictionaries with “name”, “description”, and optionally “score” (an integer), or an empty list.

collect(app_id, exp_config=None, bot=False, **kwargs)

Collect data for the provided experiment id.

The app_id parameter must be a valid UUID. If an existing data file is found for the UUID it will be returned, otherwise - if the UUID is not already registered - the experiment will be run and data collected.

See run() method for other parameters.

create_network()

Return a new network.

create_node(participant, network)

Create a node for a participant.

create_participant(worker_id, hit_id, assignment_id, mode, recruiter_name=None, fingerprint_hash=None, entry_information=None)

Creates and returns a new participant object. Uses participant_constructor as the constructor.

Parameters:

• worker_id (str) – the recruiter Worker Id

• hit_id (str) – the recruiter HIT Id

• assignment_id (str) – the recruiter Assignment Id

• mode (str) – the application mode

• recruiter_name (str) – the recruiter name

• fingerprint_hash (str) – the user’s fingerprint

• entry_information – a JSON serializable data structure containing additional participant entry information

Returns:

A Participant instance

dashboard_database_actions()

Returns a sequence of custom actions for the database dashboard. Each action must have a title and a name corresponding to a method on the experiment class.

The named methods should take a single data argument which will be a list of dicts representing the datatables rendering of a Dallinger model object. The named methods should return a dict containing a "message" which will be displayed in the dashboard.

Returns a single action referencing the dashboard_fail() method by default.

dashboard_fail(data)

Marks matching non-failed items as failed. Items are looked up by id and object_type (e.g. "Participant").

Parameters:

data – A list of dicts representing model items to be marked as failed. Each must have an id and an object_type

Returns:

Returns a dict with a "message" string indicating how many items were successfully marked as failed.

data_check(participant)

Check that the data are acceptable.

Return a boolean value indicating whether the participant’s data is acceptable. This is meant to check for missing or invalid data. This check will be run once the participant completes the experiment. By default performs no checks and returns True. See also, attention_check().

data_check_failed(participant)

What to do if a participant fails the data check.

Runs when participant has failed data_check(). By default calls fail_participant().

events_for_replay(session=None, target=None)

Returns an ordered list of “events” for replaying. Experiments may override this method to provide custom replay logic. The “events” returned by this method will be passed to replay_event(). The default implementation simply returns all Info objects in the order they were created.

classmethod extra_parameters()

Override this classmethod to register new config variables. It is called during config load. See Extra Configuration for an example.

exit_info_for(participant)

An experiment can return a dictionary of infomation that will be shown to the participant at the very last point in their lifecycle, if the HIT is not submitted to an external recruitment service for submission.

For complete control over the exit page, a customized version of the exit_recruiter.html template can be included in the experient directory, and this will override the default provided by Dallinger.

Parameters:

participant – the Participant instance for which to calculate an exit value

Returns:

dict which may be rendered to the worker as an HTML table when they submit their assigment.

fail_participant(participant)

Fail all the nodes of a participant.

get_network_for_participant(participant)

Find a network for a participant.

If no networks are available, None will be returned. By default participants can participate only once in each network and participants first complete networks with role=”practice” before doing all other networks in a random order.

info_get_request(node, infos)

Run when a request to get infos is complete.

info_post_request(node, info)

Run when a request to create an info is complete.

is_complete()

Method for custom determination of experiment completion. Experiments should override this to provide custom experiment completion logic. Returns None to use the experiment server default logic, otherwise should return True or False.

is_overrecruited(waiting_count)

Returns True if the number of people waiting is in excess of the total number expected, indicating that this and subsequent users should skip the experiment. A quorum value of 0 means we don’t limit recruitment, and always return False.

load_participant(assignment_id)

Returns a participant object looked up by assignment_id.

Intended to allow a user to resume a session in a running experiment.

Parameters:

assignment_id (str) – the recruiter Assignment Id

Returns:

A Participant instance or None if there is not a single matching participant.

log(text, key='?????', force=False)

Print a string to the logs.

log_summary()

Log a summary of all the participants’ status codes.

classmethod make_uuid(app_id=None)

Generates a new UUID. This is a class method and can be called as Experiment.make_uuid(). Takes an optional app_id which is converted to a string and, if it is a valid UUID, returned.

networks(role='all', full='all')

All the networks in the experiment.

node_get_request(node=None, nodes=None)

Run when a request to get nodes is complete.

node_post_request(participant, node)

Run when a request to make a node is complete.

normalize_entry_information(entry_information)

Accepts a dictionary with information about a recruited user. Returns a dictionary containing data the needed to create or load a Dallinger Participant. The returned data should include valid assignment_id, worker_id, and hit_id values. It may also include an entry_information key which should contain a transformed entry_information dict which will be stored for newly created participants.

By default, the extraction of these values is delegated to the recruiter’s normalize_entry_information method.

Returning a dictionary without valid hit_id, assignment_id, or worker_id will generally result in an exception.

participant_task_completed(participant)

Called when an experiment task is finished and submitted, and prior to data and attendance checks.

Assigns the qualifications to the Participant, via their recruiter. These will include one Qualification for the experiment ID, and others for the configured group_name, if it’s been set.

Overrecruited participants don’t receive qualifications, since they haven’t actually completed the experiment. This allows them to remain eligible for future runs.

Parameters:

participant – the Participant instance

recruit()

Recruit participants to the experiment as needed.

This method runs whenever a participant successfully completes the experiment (participants who fail to finish successfully are automatically replaced). By default it recruits 1 participant at a time until all networks are full.

replay_event(event)

Stub method to replay an event returned by events_for_replay(). Experiments must override this method to provide replay support.

replay_start()

Stub method for starting an experiment replay. Experiments must override this method to provide replay support.

replay_finish()

Stub method for ending an experiment replay. Experiments must override this method to provide replay support.

replay_started()

Returns True if an experiment replay has started.

run(exp_config=None, app_id=None, bot=False, **kwargs)

Deploy and run an experiment.

The exp_config object is either a dictionary or a localconfig.LocalConfig object with parameters specific to the experiment run grouped by section.

save(*objects)

Add all the objects to the session and commit them.

This only needs to be done for networks and participants.

setup()

Create the networks if they don’t already exist.

submission_successful(participant)

Run when a participant’s experiment submission passes data and attendence checks.

Parameters:

participant – the Participant instance

transformation_get_request(node, transformations)

Run when a request to get transformations is complete.

transformation_post_request(node, transformation)

Run when a request to transform an info is complete.

transmission_get_request(node, transmissions)

Run when a request to get transmissions is complete.

transmission_post_request(node, transmissions)

Run when a request to transmit is complete.

vector_get_request(node, vectors)

Run when a request to get vectors is complete.

vector_post_request(node, vectors)

Run when a request to connect is complete.

dallinger.experiment.experiment_route(rule, **kwargs)

Creates a decorator to register experiment functions or classmethods as routes on the Experiment.experiment_routes blueprint. Accepts all standard flask route arguments. The registration is deferred until experiment server setup to allow routes to be overridden.

Returns:

A decorator to register methods from a class as experiment routes.