wiki:GeniExperiments

Version 31 (modified by hmussman@bbn.com, 6 years ago) (diff)

--

GENI Experiment Workflows

Last edited on 032513 by HEM

1) Range of GENI Experiments (Harry)

There will be a wide range of GENI experiments, with different characteristics, including these:

a) Student solves a class problem

b) Student executes an on-line tutorial, or follows a reference experiment

c) Researcher evaluates an algorithm or concept

d) Researcher evaluates the performance of a configuration

e) Researcher gathers the results of one or more experiments, and writes a paper

f) Researcher gathers the results of many experiments, and writes a thesis

g) Experimenter prototypes a service, without opt-in users h) Experimenter prototypes a service, with opt-in users

i) Experimenter demonstrates a service to selected opt-in users

j) Experimenter offers a service to many opt-in users for a long period of time

k) Operator evaluates representative GENI slice performance, such as networking performance, to troubleshoot problems

l) Operator evaluates representative GENI slice performance, such as networking performance, over an extended period of time, and shares their data with others

2) Characteristics of GENI Experiment Workflows (Harry)

a) From a simple to a complex experiment

b) By a novice or an expert experimenter

c) From a short-term to a long-term experiment

d) Using tools with graphical or script-driven interfaces

e) Covering actual experiments, tutorials, and "reference experiments"

3) Goals for GENI Experiment Workflows (Jeanne and Harry)

a) Start with a basic GENI experiment/tutorial/test workflow that uses standardized, self-contained steps, and a format optimized for easy understanding b) Because self-contained, allows for all the variations in flow that occur in a real experiment: repeating steps (e.g., many runs); re-ordering steps (e.g., analyzing much later); skipping steps (e.g., do not archive) c) Because standardized and self-contained, makes it easier to design tools, by being able to focus on tools for each step, and then connections (interfaces) between steps. d) Because standardized, with a format optimized for easy understanding, it is easier for the user to understand a tutorial, particularly after completing another GENI tutorial

e) Later, produce variations that cover additional types of experiments

4) Basic GENI Experiment/Tutorial/Test Workflow (Harry)

Start with one "Basic GENI Experiment/Tutorial/Test Workflow", and later introduce variations that cover additional types of experiments

4.1) Uses

Use to guide:

a) Experiment management tool design

b) I&M tool sets

c) Contents of GENI tutorials

d) Operational tests, i.e., acceptance tests of GENI racks

4.2) Graphical View of Steps

figure

4.3) Detailed Steps

basic workflow steps

4.4) Summary of Steps

a) Includes eight steps that are self-contained, which allows for all the variations in flow that occur in a real experiment: repeating steps (e.g., many runs); re-ordering steps (e.g., analyzing much later); skipping steps (e.g., do not archive)

b) Each step typically includes these items:

+ Goal, for this step

+ Configuration, for this step

+ Sub-steps, numbered 1 - n

+ Optional sub-sub-steps, identified a - z

+ List of experiment management tools utilized

+ List of persistent services utilized

+ List of artifacts involved

+ Status summary, at completion of the step

c) This is consistent with a "best-in-class" example: web page Getting Started with Amazon EC2 Linux Instances

d) The following may also be included in the workflow, at the top level or within a step:

+ FAQs

+ Troubleshooting info

+ References

4.5) Mapping from Steps to Experiment Management Tools (Jeanne)

figure

5) A Standardized GENI Experiment/Tutorial/Test Presentation Format (Jeanne and Harry)

A standardized format, optimized for easy understanding, will make it is easier for the user to understand a tutorial, particularly after completing another GENI tutorial

5.1) Standardized Document

a) Start with a static document.

b) Typically provide as a web page, with an option to print it out

c) Two "best-in-class" examples were found:

+ Web page Getting Started with Amazon EC2 Linux Instances

+ Note also option from Amazon for a comprehensive pdf document

+ Configuration manuals from Cisco, such as Cisco Smart Install Configuration Guide

d) Include a "navigation feature", to find the individual steps; in the Amazon case, this is done with a table, that includes links to each step.

e) Number the steps for easy reference; follow the approach in the Amazon table: steps 1 - n; within a step, sub-steps 1 - n; optionally, within a sub-step, sub-sub-steps a - z

f) Each step should be phrased: "do this.."; consider a controlled vocabulary, such as: use; do; repeat; stop; load; execute; see; expect

g) Each step should be indented, so that the user can check off the steps on the left, when the web page is printed.

h) Additional notes can be included, with a different indentation than the steps, to provide notes to the user

i) Notes to the user may include: tips; cautions; FAQs; troubleshooting info; references

j) The configuration in a step should typically include a figure

k) When a selection or command entry is required, a box should be used to show the command line, GUI entry and/or script

l) When a selection is required by the user, it should be clear what they are to enter

m) Expected results should be presented, typically with a box that shows a command line, GUI entry, script, table or graph

n) Formatting conventions should be used to aid understanding:

+ See pvii of [Cisco Smart Install Configuration Guide for conventions in Cisco example

+ See conventions in Amazon example

+ But, the overall look should not be too "busy"

o) Suggested formatting conventions are:

+ TBD

p) Suggested production methods are:

+ TBD

5.2) Customized Document, Including Parameters, Annotations and/or Results

a) Utilize a production method for the web page, so that the user can:

+ copy the page as a template,

+ include parameters for this particular experiment and run,

+ add annotations

+ include results

+ and store everything for later reference;

b) Then, this reference page provides an "experiment description" for a particular experiment run

c) Of course, the backup option is to print the static web page, and then write on it

d) The user's entries may include:

+ check-offs for each step

+ selections made

+ results received

+ notes made

e) An example is the left window of the Lab wiki (ref)

f) Suggested production methods are:

+ TBD

5.3) Additional One-Page Summary Document

a) To supplement a document on a web page, particularly for an in-person tutorial

b) Provide a one-page summary on paper, that:

+ includes specific information for this user (i.e., their assigned credentials),

+ allows them to easily check off steps, and

+ capture simple notes and results.

c) Suggested production methods are:

+ TBD

5.4) Dual-screen Presentation, of Workflow Steps and Current Console/Browser

a) First screen shows workflow steps, e.g., a browser pointed at wiki holding steps

b) Second screen shows current console/browser, as instructor follows the steps.

c) This avoids having to go back and forth, if there were only one screen, and makes it easy for the student to follow along.

5.5) Additional Video "Document"

a) To supplement a document on a web page, for an in-person tutorial, or for use as an off-line tutorial

b) Provide a short video, that summarizes how to complete one or more steps.

c) An example is the video included in the GIMI tutorial at GEC16 (ref)

d) Suggested production methods are:

+ TBD

6) Towards a GENI Workflow Execution Service

a) It may be possible to work towards a GENI workflow execution service, driven by a script, that executes part or all of the workflow

b) One current example is when OMF is used in Step x, to orchestrate the I&M and/or experiment services.

+ See ref

c) This requires a workflow description language

d) There are examples from the GRID community:

+ Reference workflow language:

+ Reference workflow execution service:

7) Uses of GENI Storage and Archive Service

a) While a user is executing an experiment/tutorial/test, they need to be able to store and archive various experiment artifacts

+ See note from Fraida

b) Required functions:

+ Availability of artifacts in archive from a reference experiment/tutorial/test

+ Scratchpad, for transfer of artifacts between tools, operating in different steps

+ Storage of artifacts after an experiment is finished, for later analysis or repetition

+ Accessible from all services owned (or acting for) the user

c) Design of GENI Storage and Archive Service:

+ Being built to provide a place for all such artifact, including descriptors

+ iRODS ref

+ Descriptor ref

8) Example Experiments/Tutorials/Tests (Jeanne and Harry)

8.1) Example Experiments (Jeanne and Harry)

a) Goal: move towards "real reference experiments", and not just demos.

8.2) Example Tutorials (Jeanne and Harry)

a) GIMI GEC15 tutorial

8.3) Example GENI Acceptance Tests (Luisa)

7) Issues

a) Issue: variations with type of users (novice to expert) and thus interfaces (graphical to script); can these be mixed? b) Issue: variations between InstaGENI and ExoGENI, between GIMI and GEMINI; is there a common set? c) Issue: variations from this basic workflow, to more-specialized workflows; which should be considered?

References

+ See GIMI GEC15 tutorial that follows these steps.

+ See slides from Experiment Lifecycle Tools session at GEC15.

p4 for configuration

p21 for mapping of tools to steps

Attachments (4)

Download all attachments as: .zip