Monitoring a Live Experiment

There are a number of ways that you can monitor a live experiment:

Command line tools

dallinger summary --app {#id}, where {#id} is the id (w...) of the application.

This will print a summary showing the number of participants with each status code, as well as the overall yield:

status  | count
----------------
1   | 26
101 | 80
103 | 43
104 | 2

Yield: 64.00%

The Dashboard

The Dallinger experiment server provides a dashboard view for experiment administrators to monitor running experiments. The dasboard can be found at /dashboard, and requires login credentials that are provided by the commandline output when launching an experiment using dallinger debug, dallinger sandbox, or dallinger deploy.

When running under dallinger debug a browser window should open with the dashboard already logged in. The dashboard username and password can also be found in the dashboard_user and dashboard_password configuration parameters in the deployed config.txt configuration file. By default the user is named admin and the password is generated randomly, but the user name and password can be specified using configuration files.

Customizing the Dashboard

You can add custom tabs to the Dallinger Dashboard by registering new new Flask routes on the dashboard using the dashboard_tab decorator:

dallinger.experiment_server.dashboard.dashboard_tab(title, **kwargs)[source]

Creates a decorator to register experiment functions or classmethods as dashboard tabs. Adds a tab with a title at the path /dashboard/function_name and accepts any other flask route keyword arguments. Registers the decorated method as a route on the dallinger.experiment_server.dashboard.dashboard Blueprint. The registration is deferred until experiment server setup to allow routes to be overridden. Optionally accepts after_route and before_route arguments to specify tab ordering relative to other named routes.

Parameters:

title (str) – The dashboard tab title
after_route (str) – Optional name of a tab after which to insert this tab
before_route (str) – Optional name of a tab before which to insert this tab
tab (DashboardTab) – Optional DashboardTab instance if you need to provide nested dashboard menus, or other tab features.

Returns:

Returns a decorator to register methods from a class as dashboard routes.

For example in your custom Experiment class could add the following code to add a “My Experiment” tab to the dashboard:

from dallinger.experiment import Experiment
from dallinger.experiment_server.dashboard import dashboard_tab

  class MyExperimentClass(Experiment

      @dashboard_tab("My Experiment")
      def my_experiment():
          return "Hello, World. This is some information about My Experiment"

This will regsiter the flask route on the dashboard as /dashboard/my_experiment under a tab named “My Experiment”.

The dashboard also supports nested tab/menus using the DashboardTab object:

from dallinger.experiment_server.dashboard import dashboard_tabs, DashboardTab

def child_tabs():
    return [DashboardTab('Child1', 'child1'), DashboardTab('Child2', 'child2')]

complex_tab = DashboardTab('Title', 'route_name', child_tabs)
dashboard_tabs.insert_tab(complex_tab)

The dashboard_tabs object supports the following methods for managing the available tabs on your experiment’s dashboard:

class dallinger.experiment_server.dashboard.DashboardTabs(tabs)[source]

insert(title, route_name, position=None)[source]

Creates a new dashboard tab and inserts it (optionally at a specific position)

Parameters:

title (str) – Title string to appear in the dashboard HTML
route_name (str) – The registered route name (optionally prefixed with dashboard.)
position (int, optional) – The 0-based index where the tab should be inserted. By default tabs will be appended to the end.

insert_tab(tab, position=None)[source]

Insert a new dashboard tab (optionally at a specific position)

Parameters:

tab (DashboardTab) – DashboardTab instance
position (int, optional) – The 0-based index where the tab should be inserted. By default tabs will be appended to the end.

insert_before_route(title, route_name, before_route)[source]

Creates a new dashboard tab and inserts it before an existing tab by route name

Parameters:

title (str) – Title string to appear in the dashboard HTML
route_name (str) – The registered route name (optionally prefixed with dashboard.)
before_route (str) – The route name to insert this tab before.

Raises:

ValueError – When before_route is not found in registered tabs

insert_tab_before_route(tab, before_route)[source]

Insert a new dashboard tab before an existing tab by route name

Parameters:

tab (DashboardTab) – DashboardTab instance
before_route (str) – The route name to insert this tab before.

Raises:

ValueError – When before_route is not found in registered tabs

insert_after_route(title, route_name, after_route)[source]

Creates a new dashboard tab and inserts it after an existing tab by route name

Parameters:

title (str) – Title string to appear in the dashboard HTML
route_name (str) – The registered route name (optionally prefixed with dashboard.)
after_route (str) – The route name to insert this tab after.

Raises:

ValueError – When after_route is not found in registered tabs

insert_tab_after_route(tab, after_route)[source]

Insert a new dashboard tab after an existing tab by route name

Parameters:

tab (DashboardTab) – DashboardTab instance
after_route (str) – The route name to insert this tab after.

Raises:

ValueError – When after_route is not found in registered tabs

remove(route_name)[source]

Remove a tab by route name

Parameters:: route_name (str) – The registered route name (optionally prefixed with dashboard.)

The DashboardTab object used by the various insert_tab* methods provide the following API:

class dallinger.experiment_server.dashboard.DashboardTab(title, route_name, children_function=None, params=None)[source]

__init__(title, route_name, children_function=None, params=None)[source]

Creates a new dashboard tab

Parameters:

title (str) – Title string to appear in the dashboard HTML
route_name (str) – The registered route name (optionally prefixed with dashboard.)
children_function – A callable that returns an iterable of DashboardTab to be displayed as children of this tab
params – A mapping of url query string parameters used when generating the route url.

The dashboard monitoring view can be extended by adding panes to the sidebar or extending the existing panes. This can be done customizing the monitoring_panels and/or monitoring_statistics methods of your experiment class. Additionally, you can customize the display of the selected nodes customizing the node_visualization_html method, or the visualization_html property on your model class. Finally, the layout of the visualization can be configured by customizing the node_visualization_options method to return a dictionary of vis.js configuration options.

The dashboard database view can be customized by customizing the json_data method on your model classes to add/modify data provided by each model to the dashboard views, or by modifying the DataTables data returned by the table_data method in your Experiment class.

class dallinger.experiment.Experiment(session=None)[source]

Define the structure of an experiment.

monitoring_panels(**kw)[source]

Provides monitoring dashboard sidebar panels.

Parameters:

**kw – arguments passed in from the request

Returns:

An OrderedDict() mapping panel titles to HTML strings to render in the dashboard sidebar.

monitoring_statistics(**kw)[source]

The default data used for the monitoring panels

Parameters:

**kw – arguments passed in from the request

Returns:

An OrderedDict() mapping panel titles to data structures describing the experiment state.

node_visualization_html(object_type, obj_id)[source]

Returns a string with custom HTML visualization for a given object referenced by the object base type and id.

Parameters:

object_type (str) – The base object class name, e.g. Network, Node, Info, Participant, etc.

id (int) – The id of the object

Returns:

A valid HTML string to be inserted into the monitoring dashboard

node_visualization_options()[source]

Provides custom vis.js configuration options for the Network Monitoring Dashboard.

Returns:

A dict with vis.js option values

table_data(table: str = 'participant', polymorphic_identity: str | None = None)[source]

Generates DataTablesJS data and configuration for the experiment. The data is compiled from the models’ __json__ methods, and can be customized by either overriding this method or using the json_data method on the model to return additional serializable data.

Parameters:

table – table to query

polymorphic_identity – optional polymorphic identity (corresponds to the type column)

Returns:

Returns a dict with DataTablesJS data and configuration, filters using arbitrary keyword arguments. Should contain data and columns keys at least, with columns containing data for all fields on all returned objects.

dashboard_database_actions()[source]

Returns a sequence of custom actions for the database dashboard. Each action must have a title and a name corresponding to a method on the experiment class.

The named methods should take a single data argument which will be a list of dicts representing the datatables rendering of a Dallinger model object. The named methods should return a dict containing a "message" which will be displayed in the dashboard.

Returns a single action referencing the dashboard_fail() method by default.

You may also add new actions to the dashboard database view by adding additional title and name pairs to the dashboard_database_actions() output along with corresponding methods that process submitted data. The dashboard_fail() method is an example of such an action.

Papertrail

You can use Papertrail to view and search the live logs of your experiment. You can access the logs either through the Heroku dashboard’s Resources panel (https://dashboard.heroku.com/apps/{#id}/resources), where {#id} is the id of your experiment, or directly through Papertrail.com (https://papertrailapp.com/systems/{#id}/events).

Setting up alerts

You can set up Papertrail to send error notifications to Slack or another communications platform.

Take a deep breath.
Open the Papertrail logs.
Search for the term error.
To the right of the search bar, you will see a button titled “+ Save Search”. Click it. Name the search “Errors”. Then click “Save & Setup an Alert”, which is to the right of “Save Search”.
You will be directed to a page with a list of services that you can use to set up an alert.
Click, e.g., Slack.
Choose the desired frequency of alert. We recommend the minimum, 1 minute.
Under the heading “Slack details”, open (in a new tab or window) the link new Papertrail integration.
This will bring you to a Slack page where you will choose a channel to post to. You may need to log in.
Select the desired channel.
Click “Add Papertrail Integration”.
You will be brought to a page with more information about the integration.
Scroll down to Step 3 to get the Webhook URL. It should look something like https://hooks.slack.com/services/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.
Copy this link to your clipboard.
Change anything else you want and then scroll to the bottom and click “Save integration”.
Go back to Papertrail page that you left in Step 7.
Paste the copied URL into the input text box labeled “Integration’s Webhook URL” under the “Slack Details” heading.
Click “Create Alert” on the same page.
Victory.