wiki:GeniExperiments

Version 20 (modified by hmussman@bbn.com, 11 years ago) (diff)

--

GENI Experiment Workflows: Overview

1) Range of GENI Experiments (Harry)

There will be a wide range of GENI experiments, with different characteristics, including these:

a) Student solves a class problem

b) Student executes an on-line tutorial, or follows a reference experiment

c) Researcher evaluates an algorithm or concept

d) Researcher evaluates the performance of a configuration

e) Researcher gathers the results of one or more experiments, and writes a paper

f) Researcher gathers the results of many experiments, and writes a thesis

g) Experimenter prototypes a service, without opt-in users

h) Experimenter prototypes a service, with opt-in users

i) Experimenter demonstrates a service to selected opt-in users

j) Experimenter offers a service to many opt-in users for a long period of time

k) Operator evaluates representative GENI slice performance, such as networking performance, to troubleshoot problems

l) Operator evaluates representative GENI slice performance, such as networking performance, over an extended period of time, and shares their data with others

2) Characteristics of GENI Experiment Workflows (Harry)

a) From a simple to a complex experiment

b) By a novice or an expert experimenter

c) From a short-term to a long-term experiment

d) Using tools with graphical or script-driven interfaces

e) Covering actual experiments, tutorials, and "reference experiments"

3) Goals for GENI Experiment Workflows (Jeanne and Harry)

a) Start with a basic GENI experiment/tutorial/test workflow that uses standardized, self-contained steps, and a format optimized for easy understanding

b) Because self-contained, allows for all the variations in flow that occur in a real experiment: repeating steps (e.g., many runs); re-ordering steps (e.g., analyzing much later); skipping steps (e.g., do not archive)

c) Because standardized and self-contained, makes it easier to design tools, by being able to focus on tools for each step, and then connections (interfaces) between steps.

d) Because standardized, with a format optimized for easy understanding, it is easier for the user to understand a tutorial, particularly after completing another GENI tutorial

e) Later, produce variations that cover additional types of experiments

4) High-level look at Basic GENI Experiment/Tutorial/Test Workflow (Jeanne)

figure

5) Steps in Basic GENI Experiment/Tutorial/Test Workflow (Harry)

a) Start with one "Basic GENI Experiment/Tutorial/Test Workflow", and later introduce variations that cover additional types of experiments

b) Use to guide:

  • Experiment management tool design
  • I&M tool sets
  • Contents of GENI tutorials
  • Operational tests, i.e., acceptance tests of GENI racks

c) See basic workflow steps

d) Includes eight steps that are self-contained, which allows for all the variations in flow that occur in a real experiment: repeating steps (e.g., many runs); re-ordering steps (e.g., analyzing much later); skipping steps (e.g., do not archive)

e) Each step typically includes these items:

+ Goal, for this step

+ Configuration, for this step

+ Sub-steps, numbered 1 - 9

+ Optional sub-sub-steps, identified a - z

+ List of experiment management tools utilized

+ List of persistent services utilized

+ List of artifacts involved

+ Status summary, at completion of the step

f) This is consistent with a "best-in-class" example: web page Getting Started with Amazon EC2 Linux Instances

g) The following may also be included in the workflow, at the top level or within a step:

+ FAQs

+ Troubleshooting info

+ References

6) Current Mapping from Basic GENI Experiment/Tutorial Workflow to Experiment Management Tools (Jeanne)

figure

7) A Standardized GENI Experiment/Tutorial Format, Optimized for Ease of Use (Jeanne and Harry)

a) A standardized format, optimized for easy understanding, will make it is easier for the user to understand a tutorial, particularly after completing another GENI tutorial

b) Start with a static document, that indicates how user should modify for their own run

c) Provide as a web page, with an option to print it out

d) Two "best-in-class" examples were found:

+ Web page Getting Started with Amazon EC2 Linux Instances

+ Configuration manuals from Cisco, such as Cisco Smart Install Configuration Guide

e) Include a "navigation feature", to find the individual steps; in the Amazon case, this is done with a table, that includes links to each step.

f) Number the steps for easy reference; follow the approach in the Amazon table: steps 1 - 9; within a step, sub-steps 1 - 9; optionally, within a sub-step, sub-sub-steps a - z

g) Each step should be phrased: "do this.."; consider a controlled vocabulary, such as: use; do; repeat; stop; load; execute; see; expect

h) Each step should be indented, so that the user can check off the steps on the left, when the web page is printed.

i) Additional notes can be included, with a different identation than the steps, to provide notes to the user

j) Notes to the user may include: tips; FAQs; troubleshooting info; references

j) The configuration in a step should include a figure

k) When a selection or command entry is required, a box should be used to show the related command line, GUI entry and/or script

l) When a selection is required by the user, it shoudl be clear what they are to enter

m) Expected results should be presented, typically with a box that shows the related command line, GUI entry and/or script

n) Formatting conventions should be used to aid understanding:

+ See [ conventions in Cisco example]

+ See conventions in Amazon example

+ But, the overall look should not be too "busy"

o) Suggested formatting conventions are:

+ ?

p) Option: utilize a production method for the web page, so that the user can copy the page as a template, include entries for this particular run, and store everything for later reference; this reference page provides an "experiment description" for a particular experiment run

q) The user's entries may include:

+ check-offs for each step

+ selections made

+ results received

+ notes made

8) Example Tutorials (Jeanne and Harry)

a) GIMI tutorial at GEC15

9) Example GENI Acceptance Tests (Luisa)

10) Example Reference Experiments (Jeanne and Harry)

11) Production Methods for Standardized GENI Experiment/Tutorial Format, Optimized for Ease of Use (Jeanne and Harry)

11) Issues

a) Can we agree on these standardized, self-contained steps? what changes or additions?

b) Issue: variations with type of users (novice to expert) and thus interfaces (graphical to script); can these be mixed?

c) Issue: variations between InstaGENI and ExoGENI, between GIMI and GEMINI; is there a common set?

d) Issue: variations from this basic workflow, to more-specialized workflows; which should be considered?

e) Next steps?

References

workflow text

related storyboard

Currently being used by GIMI and GEMINI I&M projects.

See GIMI GEC15 tutorial that follows these steps.

See slides from Experiment Lifecycle Tools session at GEC15.

p4 for configuration

p21 for mapping of tools to steps

Attachments (4)

Download all attachments as: .zip