Recruitment

A recruiter is a program that takes charge of recruiting participants for an experiment. Dallinger’s design supports the option to run the same experiment with different recruitment systems, generally requiring only changes to configuration parameters in your experiment’s config.txt file, but no changes to your experiment code.

General Considerations

Some facets of recruiting are common across all recruitment systems, so we’ll describe these first.

Recruitment Planning

An experimenter needs to consider recruitment from the initial stages of planning an experiment. How many participants are needed? Do they need to interact with each other? Is the interaction synchronous or asynchronous? What happens when we over-recruit participants? Dallinger allows a good deal of flexibility to tweak participant recruitment, but it needs to be well planned in advance.

The experimenter also has to take into account the time and effort required of participants to participate in research. If signing up the correct number of participants requires some of them to wait for a long time, for instance, they might not stay around to finish, or may do so one time, then opt out of any further experiments by the same experimenter.

Configuration

For a specific experiment, the experimenter will want to specify a number of participants that can be trusted as much as possible to follow the instructions and complete the experiment. Dallinger’s recruiters each support various configuration parameters to let the experimenter achieve this.

For example, the following configuration is defined by GridUniverse, a parameterized space of games for the study of human social behavior:

[HIT Configuration]
title = Griduniverse
description = Play a game
keywords = Psychology, game, play
base_payment = 1.00
lifetime = 24
duration = 0.1
us_only = true
approve_requirement = 95
group_name = Griduniverse,Survival

The title and description (and in the case of the MTurk recruiter, keywords) are important, because these are what a potential participant will see when deciding whether to participate in an experiment or not.

Other values control compensation (base_payment), how long to display the HIT in worker feeds (lifetime), filters on which workers will see the HIT in their feed (us_only, approval_requirement), and so on.

Some of these values will be common across recruiters, but many will be specific to the recruiter you choose.

Amazon Mechanical Turk (“MTurk”) Recruiting

Configuration Parameters

base_payment is how much a participant will be paid for their participation. This depends more on the experimenter’s organization and policies than on the experiment itself, although an exceptionally hard to complete experiment might benefit from a higher payment figure.

lifetime is how many hours to keep the experiment “open” for MTurk users. An experiment with many participants that are recruited sequentially or are not required to interact with each other, might benefit from a larger window.

Once a participant is looking at your experiment sign on page, the duration parameter controls how long (in hours) it will wait for participation confirmation before timing out. This prevents undecided or forgetful users from causing recruitment problems.

Dallinger is being developed in the US, and for the time being most users are located there. Many experiments can be run without taking into account the participant’s nationality, but in some cases, experimenters may need to restrict participation to US-only participants, The us_only parameter allows this.

A remote experiment obviously would benefit from having very trustworthy participants, so that experimenters can be reasonably sure that the experiment will be completed and the instructions are followed to the best of the participant’s ability. MTurk keeps track of how many experiments a participant has been in, and what percentage of those are approved by the experimenter. The approve_requirement parameter takes a number from 1 to 100, representing the percentage of approved experiments that a participant must have to be able to participate in the experiment.

The group_name parameter is used to assign named qualifications to participants that complete an experiment. You can use this later to find out if a possible participant has already completed the experiment under the same group name. This can be a single value, or a comma-separated list of values can be provided, and a qualification will be assigned for each. Note that it’s not enough to set this parameter to have the qualification saved. It’s necessary to also set the assign_qualifications parameter to true as well. As an alternative to (or in combination with) using these configuration values, you can also override properties in your experiment class. See especially the group_qualifications property, which is responsible for providing name->value qualification definitions in the form of a python dictionary.

Finally, the qualification_blacklist parameter can be used to filter out potential participants and prevent them from even viewing the experiment sign-on page. It takes a comma-separated list of qualification names to avoid. In order to prevent participant from repeating an experiment or group, you can set this parameter to an experiment ID or group name, and set assign_qualifications to true.

Prolific Recruitment

In many respects, recruiting with Prolific is very similar to recruiting with MTurk, however, there are some differences.

Feature Differences

Unlike MTurk, Prolific does not provide a system for assigning custom qualifications to workers. When testing, it is possible to explicitly reveal your experiment to only a select group of workers by providing an “allow list” in your experiment configuration. This is most easily done by including a separate file in JSON format, to be read in as part of your config.txt by providing the file name of the JSON file.

Example JSON file contents (say it’s been named “prolific.json”):

{
  "eligibility_requirements": [
      {
        "attributes": [
          {
            "name": "white_list",
            "value": [
              # worker ID one,
              # worker ID two,
              # etc.
            ]
          }
        ],
        "_cls": "web.eligibility.models.CustomWhitelistEligibilityRequirement"
      }
    ]
}

Inclusion in config.txt:

prolific_recruitment_config = file:prolific.json

Configuration Differences

Because some of the configuration options for Prolific do not perfectly match the options of MTurk, different configuration keys are used to avoid ambiguous meaning. Details of the keys used for Prolific recruitment are described in detail in the Prolific Recruitment section of the Configuration documentation.

Currencies

Prolific will use the currency of your researcher account, and convert automatically to the participant’s currency when calculating base pay and bonuses.

Advanced Features

Waiting Rooms

One other thing that affects recruitment is the use of a waiting room. Waiting rooms are used when an experiment requires participants to be synchronized. Participants are kept in the “room” until enough of them have signed up and are ready to start. Experimenters can set the quorum in the experiment code.

Recruitment Handling in Experiment Code

In addition to the previously mentioned configuration parameters, Dallinger experiment creators can use their experiment code to further affect recruitment. There are a number of basic recruitment attributes that can be set on experiment initialization, and recruitment can be further affected by calling specific methods during experiment runtime.

There are specific points in an experiment code where recruitment is usually affected. To show how you can set up recruitment for your experiment, we will use GridUniverse code as a guide. The methods discussed here are part of the experiment base class, so it is not required to implement them in your experiment, but most experiments need at least the configure and create_network methods.

def configure(self):
    super(Griduniverse, self).configure()
    self.num_participants = config.get('max_participants', 3)
    self.quorum = self.num_participants
    self.initial_recruitment_size = config.get('num_recruits',
                                               self.num_participants)

The configure method is called during experiment initialization, and is where experiment specific configuration takes place. Many times, configuration parameters from the experiment config.txt file are used here.

GridUniverse defines max_participants and num_recruits parameters. They are used in this method to set experiment.num_participants, experiment.quorum and experiment.initial_recruitment_size. The first of these is only used in GridUniverse code, so we can ignore it.

In its configure method, GridUniverse sets experiment_quorum to be the same as the configured number of participants. This means that the participants will be held in the waiting room until all participants have been recruited. Other experiment designs might not need all of the participants to be ready at the same time, but only a fraction of them. This attribute only applies to experiments that use a waiting room. The default value for experiment.quorum is zero (no waiting room).

experiment.initial_recruitment_size is the number of participants required at the beginning of the experiment. This is used during the experiment’s launch phase to start the recruitment process.

def create_network(self):
    """Create a new network by reading the configuration file."""
    class_ = getattr(
        dallinger.networks,
        self.network_factory
    )
    return class_(max_size=self.num_participants + 1)

The create_network method is where the experiment network is created, usually setting the initial number of users to the number defined in experiment.initial_recruitment_size. Most experiments will have a specific network defined in their code, and call that network explicitly. In the case of GridUniverse, the experiment allows the use of any network defined by Dallinger, which is passed in as a configuration parameter. Regardless of the selected network class, it’s called with max_size set to the number of participants configured, plus one.

A simpler experiment might use something like this instead:

def create_network(self):
    return Chain(max_size=self.initial_recruitment_size)

Over-recruitment

It’s common for recruited participants to join and leave an experiment before it starts. This is difficult in experiments where multiple participants are needed in order to start the experiment. To prevent this from disrupting an experiment, experimenters can over-recruit participants to ensure that they have the correct amount of participants at the start of the experiment. The participants who are over-recruited, but not needed for the experiment, still receive a base payout and are sent to the end of the experiment.

Over-recruitment occurs when an experiment has a quorum other than zero and the number of participants in the waiting room is larger than the quorum. As mentioned above, because users in the waiting room have already been recruited, Dallinger has to treat them as having completed the experiment, and they have to be paid.

There are a couple of strategies that can be used to limit over-recruitment. It is best for an experiment to close recruitment as soon as possible after the intended quorum is full. GridUinverse overrides the experiment’s create_node method to do this.

def create_node(self, participant, network):
    try:
        return dallinger.models.Node(
            network=network, participant=participant
        )
    finally:
        if not self.networks(full=False):
            # If there are no spaces left in our networks we can close
            # recruitment, to alleviate problems of over-recruitment
            self.recruiter().close_recruitment()

This method is called when a participant is added, so GridUniverse uses it to try to detect as soon as possible if the experiment networks are full (all participants are in). It does this by getting all networks that are not full. If there are none, it calls its recruiter’s close_recruitment method.

GridUniverse also overrides the experiment’s recruit method to unconditionally close recruitment if it is called. This method is called whenever a participant successfully completes an experiment. Since GridUniverse uses a quorum and never requires adding new participants after experiment start, it’s safe to just go ahead and close recruitment here.

def recruit(self):
    self.recruiter().close_recruitment()