wiki:GeniExperiments

Version 46 (modified by hmussman@bbn.com, 12 years ago) (diff)

--

GENI Experiment Workflows

Last edited on 032613 by HEM

1) Range of GENI Experiments (Harry)

There will be a wide range of GENI experiments, with different characteristics, including these:

a) Student solves a class problem

b) Student executes an on-line tutorial, or follows a reference experiment

c) Researcher evaluates an algorithm or concept

d) Researcher evaluates the performance of a configuration

e) Researcher gathers the results of one or more experiments, and writes a paper

f) Researcher gathers the results of many experiments, and writes a thesis

g) Experimenter prototypes a service, without opt-in users

h) Experimenter prototypes a service, with opt-in users

i) Experimenter demonstrates a service to selected opt-in users

j) Experimenter offers a service to many opt-in users for a long period of time

k) Operator evaluates representative GENI slice performance, such as networking performance, to troubleshoot problems

l) Operator evaluates representative GENI slice performance, such as networking performance, over an extended period of time, and shares their data with others

2) Characteristics of GENI Experiment Workflows (Harry)

a) From a simple to a complex experiment

b) By a novice or an expert experimenter

c) From a short-term to a long-term experiment

d) Using tools with graphical or script-driven interfaces

e) Covering actual experiments, tutorials, and "reference experiments"

3) Goals for GENI Experiment Workflows (Jeanne and Harry)

a) Start with a basic GENI experiment/tutorial/test workflow that uses standardized, self-contained steps, and a format optimized for easy understanding b) Because self-contained, allows for all the variations in flow that occur in a real experiment: repeating steps (e.g., many runs); re-ordering steps (e.g., analyzing much later); skipping steps (e.g., do not archive) c) Because standardized and self-contained, makes it easier to design tools, by being able to focus on tools for each step, and then connections (interfaces) between steps. d) Because standardized, with a format optimized for easy understanding, it is easier for the user to understand a tutorial, particularly after completing another GENI tutorial

e) Later, produce variations that cover additional types of experiments

4) Type 1 (Basic) GENI Experiment/Tutorial/Test Workflow (Harry)

Start with one "Type 1 (Basic) GENI Experiment/Tutorial/Test Workflow", and later introduce variations that cover additional types of experiments

4.1) Uses

Use to guide:

a) Experiment management tool design

b) I&M tool sets

c) Contents of GENI tutorials

d) Operational tests, i.e., acceptance tests of GENI racks

4.2) Graphical View of Steps


4.3) Detailed Steps

Type 1 (basic) experiment/tutorial/test workflow steps

4.4) Summary of Steps

a) Includes eight steps that are self-contained, which allows for all the variations in flow that occur in a real experiment: repeating steps (e.g., many runs); re-ordering steps (e.g., analyzing much later); skipping steps (e.g., do not archive)

+ Step 1: Design experiment, and find or wirte necessary software, rspecs and/or scripts

+ Step 2: Establish experiment management environment

+ Step 3: Obtain slice of GENI resources, and then install I&M and experiment software

+ Step 4: Configure and initialize I&M and experiment services

+ Step 5: Run and orchestrate I&M and experiment services, to complete run of experiment

+ Step 6: Analyze experiment results, and decide on next step(s)

+ Step 7: When no longer needed, release all experiment resources

+ Step 8: Optionally, archive selected measurements and/or experiment artifacts

b) Each step typically includes these items:

+ Goal, for this step

+ Configuration, for this step

+ Sub-steps, numbered 1 - n

+ Optional sub-sub-steps, identified a - z

+ List of experiment management tools utilized

+ List of persistent services utilized

+ List of artifacts involved

+ Status summary, at completion of the step

c) This is consistent with a "best-in-class" example: web page Getting Started with Amazon EC2 Linux Instances

d) The following may also be included in the workflow, at the top level or within a step:

+ FAQs

+ Troubleshooting info

+ References

4.5) Mapping from Steps to Experiment Management Tools (Jeanne)


5) A Standardized GENI Experiment/Tutorial/Test Presentation Format (Jeanne and Harry)

A standardized format, optimized for easy understanding, will make it is easier for the user to understand a tutorial, particularly after completing another GENI tutorial

5.1) Standardized Document

a) Start with a static document.

b) Typically provide as a web page, with an option to print it out

c) Two "best-in-class" examples were found:

+ Web page Getting Started with Amazon EC2 Linux Instances

+ Note also option from Amazon for a comprehensive pdf document

+ Configuration manuals from Cisco, such as Cisco Smart Install Configuration Guide

d) Include a "navigation feature", to find the individual steps; in the Amazon case, this is done with a table, that includes links to each step.

e) Number the steps for easy reference; follow the approach in the Amazon table: steps 1 - n; within a step, sub-steps 1 - n; optionally, within a sub-step, sub-sub-steps a - z

f) Each step should be phrased: "do this.."; consider a controlled vocabulary, such as: use; do; repeat; stop; load; execute; see; expect

g) Each step should be indented, so that the user can check off the steps on the left, when the web page is printed.

h) Additional notes can be included, with a different indentation than the steps, to provide notes to the user

i) Notes to the user may include: tips; cautions; FAQs; troubleshooting info; references

j) The configuration in a step should typically include a figure

k) When a selection or command entry is required, a box should be used to show the command line, GUI entry and/or script

l) When a selection is required by the user, it should be clear what they are to enter

m) Expected results should be presented, typically with a box that shows a command line, GUI entry, script, table or graph

n) Formatting conventions should be used to aid understanding:

+ See pvii of [Cisco Smart Install Configuration Guide for conventions in Cisco example

+ See conventions in Amazon example

+ But, the overall look should not be too "busy"

o) Suggested formatting conventions are:

+ TBD

p) Suggested production methods are:

+ Currently done with wiki, but it would be helpful to find an easier method

5.2) Customized Document, Including Parameters, Annotations and/or Results

a) Utilize a production method for the web page, so that the user can:

+ copy the page as a template,

+ include parameters for this particular experiment and run,

+ add annotations

+ include results

+ and store everything for later reference;

b) Then, this reference page provides an "experiment description" for a particular experiment run

c) Of course, the backup option is to print the static web page, and then write on it

d) The user's entries may include:

+ check-offs for each step

+ selections made

+ results received

+ notes made

e) An example is the left window of the Lab wiki (ref)

f) Suggested production methods are:

+ Could this be based on approach used in Lab wiki?

5.3) One-Page Summary Document

a) To supplement a document on a web page, typically for an in-person tutorial

b) Provide a one-page summary on paper, that:

+ includes specific information for this user (i.e., their assigned credentials),

+ allows them to easily check off steps, and

+ capture simple notes and basic results.

c) Suggested production methods are:

+ Abstract on-lin document, and use pdf to produce one-page document

5.4) Dual-screen Presentation, of Workflow Steps and Current Console/Browser

a) To improve the presentation at an in-person tutorial

b) First screen shows workflow steps, e.g., a browser pointed at wiki holding steps

c) Second screen shows current console/browser, as instructor follows the steps.

d) This avoids having to go back and forth, if there were only one screen, and makes it easy for the student to follow along.

5.5) Short Video that Describes One or More Steps

a) To supplement a document on a web page, for an in-person tutorial, or for use as an off-line tutorial

b) Provide a short video, that describes how to complete one or more steps.

c) An example is the video included in the LabWiki tutorial at GEC16

d) Suggested production methods are:

+ Mike Zink can suggest a method.

5.6) Online Tutorial Template (Jeanne)

a) Guide

b) Template

6) Towards a GENI Workflow Execution Service

a) It may be possible to work towards a GENI workflow execution service, driven by a script, that executes part or all of the workflow

b) One current example is when OMF is used in Step 5, to orchestrate the I&M and/or experiment services.

+ See ref

c) This requires a workflow description language

d) There are examples from the GRID community:

+ Reference workflow language: link

+ Reference workflow execution service: link and tutorial

7) Uses of GENI Storage and Archive Service

a) While a user is executing an experiment/tutorial/test, they need to be able to store and archive various experiment artifacts

+ See note from Fraida (a lot of artifacts...)

b) Required functions:

+ Availability of artifacts from a reference experiment/tutorial/test, that can be copied over for a new experiment/tutorial/test

+ Scratchpad, for transfer of artifacts between tools, operating in different steps

+ Storage of artifacts after an experiment/tutorial/test is finished, for later analysis or to repeat the experiment/tutorial/test

+ Accessible from all services owned (or acting for) the user

c) Design of GENI Storage and Archive Service:

+ Being built to provide a place for all such artifacts

+ Follows a well-defined structure, with well-defined descriptors (metadata)

+ iRODS ref

+ Descriptor ref

8) Example Experiments/Tutorials/Tests (Jeanne and Harry)

8.1) Example Experiments (Jeanne and Harry)

a) Goal: move towards "real reference experiments", and not just demos.

8.2) Example Tutorials (Jeanne and Harry)

a) GIMI tutorial at GEC15 (Mike Zink)

b) Basic InstaGENI and GEMINI tutorial at GEC16, driven by GENI Desktop (Rob Ricci and Jim Griffioen)

c) Custom experiment on InstaGENI with GEMINI tools at GEC16, driven by command line (Ezra Kissel)

d) Experiments on WiMAX sites with GIMI tools at GEC16, driven by OMF EC (Fraida Fund, Ivan Seskar, Manu Gosain)

8.3) Example GENI Acceptance Tests (Luisa)

8.4) Review of Tutorials at GEC16 (Harry and Jeanne)

All tutorials at GEC16 were reviewed, and a spreadsheet summary of their characteristics was prepared.

This spreadsheet includes comments from the GPO and from the attendees.

Next: Review the spreadsheet, and consider recommendations for presentation structure, and presentation format.

9) Issues

a) Issue: variations with type of users (novice to expert) and thus interfaces (graphical to script); can these be mixed?

b) Issue: variations between InstaGENI and ExoGENI, between GIMI and GEMINI; is there a common set?

c) Issue: variations from this basic workflow, to more-specialized workflows; which should be considered?

References

+ See GIMI GEC15 tutorial that follows these steps.

+ See slides from Experiment Lifecycle Tools session at GEC15.

p4 for configuration

p21 for mapping of tools to steps

Attachments (4)

Download all attachments as: .zip