Monitoring a Live Experiment¶
There are a number of ways that you can monitor a live experiment:
Command line tools¶
dallinger summary --app {#id}
, where {#id}
is the id (w...
) of
the application.
This will print a summary showing the number of participants with each status code, as well as the overall yield:
status | count
----------------
1 | 26
101 | 80
103 | 43
104 | 2
Yield: 64.00%
The Dashboard¶
The Dallinger experiment server provides a dashboard view for experiment
administrators to monitor running experiments. The dasboard can be found at
/dashboard
, and requires login credentials that are provided by the
commandline output when launching an experiment using dallinger debug
,
dallinger sandbox
, or dallinger deploy
.
When running under dallinger debug
a browser window should open with the
dashboard already logged in. When running on Heroku, the dashboard username
and password can also be found in the heroku configuration parameters
DASHBOARD_USER
and DASHBOARD_PASSWORD
.
Customizing the Dashboard¶
You can add custom tabs to the Dallinger Dashboard by adding and registering
new Flask routes on the dashboard
Blueprint, and resgistering the view as a dashboard_tab
. For example in your experiment.py
you
could add the following code to add a “My Experiment” tab to the dashboard:
from dallinger.experiment_server.dashboard import dashboard, dashboard_tabs
@dashboard.route("my-experiment")
def my_experiment():
return "Hello, World. This is some information about My Experiment"
dashboard_tabs.insert("My Experiment", "my-experiment")
The dashboard also supports nested tab/menus using the DashboardTab
object:
from dallinger.experiment_server.dashboard import dashboard_tabs, DashboardTab
def child_tabs():
return [DashboardTab('Child1', 'child1'), DashboardTab('Child2', 'child2')]
complex_tab = DashboardTab('Title', 'route_name', child_tabs)
dashboard_tabs.insert_tab(complex_tab)
The dashboard_tabs
object supports the following methods for managing the
available tabs on your experiment’s dashboard:
-
class
dallinger.experiment_server.dashboard.
DashboardTabs
(tabs)[source]¶ -
insert
(title, route_name, position=None)[source]¶ Creates a new dashboard tab and inserts it (optionally at a specific position)
Parameters: - title (str) – Title string to appear in the dashboard HTML
- route_name (str) – The registered route name (optionally prefixed with
dashboard.
) - position (int, optional) – The 0-based index where the tab should be inserted. By default tabs will be appended to the end.
-
insert_tab
(tab, position=None)[source]¶ Insert a new dashboard tab (optionally at a specific position)
Parameters: - tab (DashboardTab) – DashboardTab instance
- position (int, optional) – The 0-based index where the tab should be inserted. By default tabs will be appended to the end.
-
insert_before_route
(title, route_name, before_route)[source]¶ Creates a new dashboard tab and inserts it before an existing tab by route name
Parameters: - title (str) – Title string to appear in the dashboard HTML
- route_name (str) – The registered route name (optionally prefixed with
dashboard.
) - before_route (str) – The route name to insert this tab before.
Raises: ValueError – When
before_route
is not found in registered tabs
-
insert_tab_before_route
(tab, before_route)[source]¶ Insert a new dashboard tab before an existing tab by route name
Parameters: - tab (DashboardTab) – DashboardTab instance
- before_route (str) – The route name to insert this tab before.
Raises: ValueError – When
before_route
is not found in registered tabs
-
insert_after_route
(title, route_name, after_route)[source]¶ Creates a new dashboard tab and inserts it after an existing tab by route name
Parameters: - title (str) – Title string to appear in the dashboard HTML
- route_name (str) – The registered route name (optionally prefixed with
dashboard.
) - after_route (str) – The route name to insert this tab after.
Raises: ValueError – When
after_route
is not found in registered tabs
-
insert_tab_after_route
(tab, after_route)[source]¶ Insert a new dashboard tab after an existing tab by route name
Parameters: - tab (DashboardTab) – DashboardTab instance
- after_route (str) – The route name to insert this tab after.
Raises: ValueError – When
after_route
is not found in registered tabs
-
The DashboardTab
object used by the various insert_tab*
methods provide the following API:
-
class
dallinger.experiment_server.dashboard.
DashboardTab
(title, route_name, children_function=None, params=None)[source]¶ -
__init__
(title, route_name, children_function=None, params=None)[source]¶ Creates a new dashboard tab
Parameters: - title (str) – Title string to appear in the dashboard HTML
- route_name (str) – The registered route name (optionally prefixed with
dashboard.
) - children_function – A callable that returns an iterable of
DashboardTab
to be displayed as children of this tab - params – A mapping of url query string parameters used when generating the route url.
-
The dashboard monitoring view can be extended by adding panes to the sidebar or
extending the existing panes. This can be done customizing the
monitoring_panels
and/or
monitoring_statistics
methods of
your experiment class. Additionally, you can customize the display of the selected
nodes customizing the node_visualization_html
method, or the visualization_html
property on your
model class:
-
class
dallinger.experiment.
Experiment
(session=None)[source] Define the structure of an experiment.
-
monitoring_panels
(**kw)[source]¶ Provides monitoring dashboard sidebar panels.
Parameters: **kw – arguments passed in from the request Returns: An OrderedDict()
mapping panel titles to HTML strings to render in the dashboard sidebar.
-
monitoring_statistics
(**kw)[source]¶ The default data used for the monitoring panels
Parameters: **kw – arguments passed in from the request Returns: An OrderedDict()
mapping panel titles to data structures describing the experiment state.
-
node_visualization_html
(object_type, obj_id)[source]¶ Returns a string with custom HTML visualization for a given object referenced by the object base type and id.
Parameters: - object_type (str) – The base object class name, e.g.
Network
,Node
,Info
,Participant
, etc. - id (int) – The
id
of the object
Returns: A valid HTML string to be inserted into the monitoring dashboard
- object_type (str) – The base object class name, e.g.
-
Papertrail¶
You can use Papertrail to view and search the live logs of your experiment. You can access the logs either through the Heroku dashboard’s Resources panel (https://dashboard.heroku.com/apps/{#id}/resources), where {#id} is the id of your experiment, or directly through Papertrail.com (https://papertrailapp.com/systems/{#id}/events).
Setting up alerts¶
You can set up Papertrail to send error notifications to Slack or another communications platform.
- Take a deep breath.
- Open the Papertrail logs.
- Search for the term
error
. - To the right of the search bar, you will see a button titled “+ Save Search”. Click it. Name the search “Errors”. Then click “Save & Setup an Alert”, which is to the right of “Save Search”.
- You will be directed to a page with a list of services that you can use to set up an alert.
- Click, e.g., Slack.
- Choose the desired frequency of alert. We recommend the minimum, 1 minute.
- Under the heading “Slack details”, open (in a new tab or window) the link new Papertrail integration.
- This will bring you to a Slack page where you will choose a channel to post to. You may need to log in.
- Select the desired channel.
- Click “Add Papertrail Integration”.
- You will be brought to a page with more information about the integration.
- Scroll down to Step 3 to get the Webhook URL. It should look
something like
https://hooks.slack.com/services/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
. - Copy this link to your clipboard.
- Change anything else you want and then scroll to the bottom and click “Save integration”.
- Go back to Papertrail page that you left in Step 7.
- Paste the copied URL into the input text box labeled “Integration’s Webhook URL” under the “Slack Details” heading.
- Click “Create Alert” on the same page.
- Victory.