Using EC2 for Docker Based Deployments

The dallinger commandline tool has some helpers for managing EC2 instances to help facilitate using docker to deploy experiments on those instances (see Docker support).

Initial Setup

Route53 DNS

If your lab is doing this for the first time, you probably need to acquire a domain name for your experiment server. This is the parent URL that will be used to host your experiments. If your lab already has a domain name, you can skip this step. On the AWS online console, navigate to the Route 53 service. On the Dashboard you can register a domain name. Note that different domain names come with different costs, and that registering a domain name can take from a few minutes to several hours. Before proceeding with the next steps, please wait until the AWS console tells you that the registration is complete.

The examples below assume that you have setup a domain my-experiments.org in Route53.

EC2 Security Group

By default the dallinger ec2 commands will provision instances with the security group dallinger. If that group does not exist, one will be created with appropriate ingress rules. You can use an existing security group by specifying the --security_group_name option on the commandline, or by setting the ec2_default_security_group value in your ~/.dallingerconfig file.

EC2 SSH Key Pair (PEM File)

By default the dallinger ec2 commands will use an SSH key pair named dallinger to create instances. If that key pair does not exist, one will be created and added to your SSH keychain. You can use an existing SSH key pair by specifying the --pem option on the commandline, or by setting the ec2_default_pem value in your ~/.dallingerconfig file.

AWS Region

Except where noted, when --region is not specified then the config value set in aws_region in e.g. ~/.dallingerconfig will be used.

Provisioning an EC2 server instance

Note: If you decide to deploy to an EC2 instance, you need to “hire” the server (Provisioning). Once you do that – the clock is ticking and you will be charged hourly until you release it (Teardown).

To provision an on-demand EC2 instance:

dallinger ec2 provision --name <server_name> --region <region> --dns-host <subdomain>.my-experiments.org --type <type> --pem <pem> --security_group_name <security_group>

Pick an instance name which is easy to recognize, for example ‘tapping_deployment_batch_2’ is good but ‘melody123’ would be bad:

dallinger ec2 provision --name tapping_deployment_batch_2 --region <region> --dns-host <subdomain>.my-experiments.org --type <type>

For example, if you want to collect data in Paris your command will include the region name for Paris, like this:

dallinger ec2 provision --name tapping_deployment_batch_2 --region eu-west-3 --dns-host <subdomain>.my-experiments.org --type <type>

Subdomain name should target your identity so it could be your own name. For example:

dallinger ec2 provision --name tapping_deployment_batch_2 --region eu-west-3 --dns-host elif.my-experiments.org --type <type>

You should use a different instance type according to your need. m7i.large is recommended for debugging and m7i.xlarge is for deploying. For example:

dallinger ec2 provision --name tapping_deployment_batch_2 --region eu-west-3 --dns-host elif.my-experiments.org -

You can configure different amounts of storage or different instance types:

dallinger ec2 provision --name <server_name> --region <region> --storage 100 --type m5.2xlarge

Increasing Storage on An Instance

You can increase the storage on a running instance by server name:

dallinger ec2 increase-storage --name <server_name> --region <region> --storage 200

Or by DNS name:

dallinger ec2 increase-storage --dns <subdomain>.my-experiments.org --region <region> --storage 200

Stopping and Starting

You can temporarily stop a running instance to prevent it from incurring hourly costs (while retaining its stored data):

dallinger ec2 stop --name <server_name> --region <region>
dallinger ec2 stop --dns <subdomain>.my-experiments.org --region <region>

Similarly you can start a stopped instance:

dallinger ec2 start --name <server_name> --region <region>
dallinger ec2 start --dns <subdomain>.my-experiments.org --region <region>

Or restart a running instance:

dallinger ec2 restart --name <server_name> --region <region>
dallinger ec2 restart --dns <subdomain>.my-experiments.org --region <region>

Teardown an EC2 Instance

Important: don’t forget to export your data before you tear down the server. If you don’t all data is lost and there is NO way to retrieve them. Before you teardown the instance make sure:

The experiment is stopped on the recruiter, e.g. in Prolific the experiment should be STOPPED and thus not active

Make sure you exported your data and run export.py to make sure your data is not faulty

To teardown an on-demand EC2 instance ny server name:

dallinger ec2 teardown --name <server_name> --region <region>

Or by DNS name:

dallinger ec2 teardown --dns <subdomain>.my-experiments.org --region <region>

Listing Available Regions and Instance Types

You can list the available EC2 regions using:

dallinger ec2 list regions

Different instance types may be available in different regions, you can list the available instance types for a region using:

dallinger ec2 list instance_types --region <region>

Listing Existing Instances

Dallinger provides some tools for introspecting your current EC2 resources. You can list all instances:

dallinger ec2 list instances --region <region>

Or filter based on instance state:

dallinger ec2 list instances --region <region> --running
dallinger ec2 list instances --region <region> --stopped --terminated

Additionally you can filter based on instance PEM key name:

dallinger ec2 list instances --region <region> --running --pem my-pem

Note: If --region is not explicitly specified instances in all regions will be listed.

Connecting to a Container Running an Experiment

You can make an SSH connection to the docker container running the a specific experiment using the server DNS name and the experiment app name with the following command:

dallinger ssh web --dns <subdomain>.my-experiments.org --app <subdomain>.my-experiments.org