wiki:GEC14Agenda/IMDesignTopics

Version 10 (modified by hmussman@bbn.com, 12 years ago) (diff)

--

Instrumentation and Measurement (I&M) Design Topics

Monday, July 9, 8am - 10am
Room:
Session leader: Harry Mussman (GENI Project Office, Raytheon BBN Technologies)

Description

This I&M session will review progress towards a comprehensive set of I&M tools for GENI experimenters and for infrastructure measurement.

Since the I&M projects continuing from Solicitation 2 are ending, it will focus on the two large Solicitation 3 I&M D&P projects, GIMI and GEMINI, starting with a review their design and status.

Next, we will review the current status of three ongoing I&M design topics, and five operational I&M services that are being introduced by the GIMI and GEMINI projects.

Finally, we will discuss the I&M goals for the next 4 months, particularly how to best support experimenters and those doing infrastructure measurement.

Summary

Agenda

This is a tentative agenda, which may change.

1) Introductions

8:00am

2) Goals for this meeting

8:05am

Harry Mussman (GPO)

  • Review the progress of the GIMI and GEMINI I&M projects towards providing a comprehensive set of I&M tools for GENI experimenters and for infrastructure measurement.
  • Review the current status of three ongoing I&M design topics.
  • Review five operational I&M services that are being introduced by the GIMI and GEMINI projects, to see which can be used by both projects and/or by other parts of GENI.
  • Discuss the I&M goals for the next 4 months, particularly how to best support experimenters and those doing infrastructure measurement.

3) Summary of GIMI design and status

8:10am
PI Michael Zink, University of Massachusetts at Amherst (mailto:zink@ecs.umass.edu)
Co-PI Max Ott, NICTA (mailto:max.ott@nicta.com.au)
Key personnel Ilia Baldine, RENCI (mailto:ibaldin@renci.org)

GIMI wiki
GIMI website at UMass Amherst

Goals:

  • Starts with OML, and provides easy-to-use tools for GENI environment
  • First focus: I&M use cases for experiments, with default OML Client in nodes that gathers passive measurements like INSTOOLS, and provides basic node-to-node connectivity tests using ping and iperf
  • Second focus: I&M use cases for infrastructure measurement
  • Later focuse: add sensor measurement data, carried in data network
  • Spiral 4 supported aggregates: ORCA servers/VMs and ExoGENI (RENCI) racks
  • Later, support: WiMAX sites
  • Spiral 5 supported aggregates: protoGENI/Emulab servers/VMs and InstaGENI (HP) racks
  • Introduces XML messaging service, with pub/sub, in public IP space
  • Introduces iRODS service, for measurement data archive.
  • Introduces IREEL portal service, for measurement data analysys and presentation.

Mike Zink will review:

  • Current architecture and design
  • Current status
  • Plan for GEC-14 tutorial

4) Summary of GEMINI design and plans

8:25am
PI Martin Swany, Indiana University (mailto:swany@iu.edu)
Co-PI Chris Small, Indiana University (mailto:chsmall@indiana.edu)
Co-PI James Griffioen, University of Kentucky (mailto:griff@netlab.uky.edu)
Co-PI Zongming Fei, University of Kentucky (mailto:fei@netlab.uky.edu)

GEMINI wiki

Goals:

Starts with LAMP, which provides flexible tools based on perfSONAR for active infrastructure measurements
++Fold INSTOOLS features into LAMP to form GEMINI tools, and then discontinue INSTOOLS

*First focus: I&M use cases for infrastructure measurement *Include sharing of data with other users (e.g., operators), at an interface with authorization using a GENI credential
*Show how to monitor clusters/racks
*Show how to measure Layer 2 and OpenFlow paths (with input from OnTimeMeasure project Slides)

*Second focus: I&M use cases for experiments
*Consider how to gather basic host measurements with BLiPP service (as was done by INSTOOLS using SNMP)
*Consider how to gather customized data from applications (as provided by OML Client)

++Extending LAMP with INSTOOLS features:
*Consider how to load tools on top of basic image
*Consider how to extend Periscope to include druple content management service
*Consider how to add xserver to basic node image, to login from portal using vnc protocol
*Continue separate portal service, which is really a gateway service, within a user workspace service
*Consider how to add interface to archive service, e.g., iRODs

*Spiral 4 supported aggregates: protoGENI/Emulab servers/VMs and InstaGENI (HP) racks
*Spiral 5 supported aggregates: ORCA servers/VMs and ExoGENI (RENCI) racks

*Continue INSTOOLS portal service, in user workspace, to find GUIs
*Continue LAMP local and global UNIS service, to register available measurement data, and network topology

Members of the GEMINI project will review: (Martin Swany)

*Current architecture and design

*Current status

slides

*Preview of tonight's GEC-13 demo

poster

*Plan for GEC-14 tutorial

5) Reports on I&M design topics

8:40m

At GEC12, nine I&M priority design topics were identified, and nine teams were assigned to understand and resolve these topics by GEC13.

See the list of topics, including a brief summary of each topic.

T4) GENI I&M Tool Testing Environment

8:40am

Team

LEAD Harry Mussman (GPO)

Tasks

Provide a concise definition of the environment where the GENI I&M tools will be tested, including:
User tools for experiment and measurement setup and orchestration
Interfaces/protocols between user tools and GENI aggregates
Supported GENI aggregtaes

Work in Progress (Harry Mussman)

T8) GENI User Tools and Services

8:45am

Team

LEAD Jeanne Ohren (GPO)

Goals

Work towards Max Ott's vision for experiment support
Provide a way for a GENI user (e.g., experimenter or operator) to access a wide variety of "GENI User Services", where each user service provides an interface (e.g., API or GUI) to the user. Those user services with a GUI (web) interface are often called "portal services".
Together, the "GENI User Services" should provide all of the functions the user needs to setup and run their experiment, then gather, analyze and present the measurement data.
These services should work together via APIs, etc., to streamline the experiment process.

Tasks

Based upon the configuration defined below, the implementation is split into:
1) A GENI User Workspace, which is a persistent Linux OS environment dedicated to the user, that serves as a container for multiple user tools
2) Multiple GENI User Tools, where each provides a service with an interface or a "portal" to the user.

Define, prototype, deploy and operate a GENI User Workspace. It can be hosted on a server dedicated to the user, or on a server hosting multiple user workspaces for multiple users.
Gather the various "user tools" that have been implemented to date, and fit into GENI User Workspace Service so that GENI I&M users can begin to conveniently conduct experiments or instrument infrastructure.
Optimize "user tools" and their interfaces to better meet the needs of GENI users (e.g., experimenters and operators).

Work in Progress

Summary of current status: (Jeanne Ohren)

slides

T5) Descriptors, Objects and Registries

8:50am

Team

LEAD Harry Mussman (GPO)

Tasks

Need to finalize MDOD schema, for XML file. References
Want to extend MDOD to cover all types of objects, i.e., software images. (NetKarma)
Want to use MDOD schema to define Event Record schema. (NetKarma)
Do we need MDOD registry? Use UNIS lookup service? Use DOR registry? Include in iRODS? Consider IF-MAP server?
Need MDOD creation and editing service. (CNRI)
Need Measurement Data Object identifiers (names); sometimes need a persistent, public reference; consider DataCite approach, which uses handle

Work in Progress

Summary of current status (Giridhar Manepalli):

slides

Conclusions:

Good things:

Excellent start
Collaborative Specification
Great Coverage
Nicely broken down into elements
Mandatory vs. optional elements identified
Genuine Use Cases: Gathering, transferring, and sharing

Jensen's proposal (NetKarma):

Current: Identifiers, Descriptors, Holders
Proposed: Identification, Lineage/Provenance, Constraints/Security, MDO Description

Zurawski's comments:

Too many secondary identifiers
Descriptors should be contextualized
Variations based on the type of object
GENI specific descriptions should be clearly marked and separated
Slight changes to names & enclosing elements recommended

Comments/suggestions based on metadata practices:

Too many optional elements

Too many choices given to users
Users bound to take the path of least resistance
Keep the scope restricted to only mandatory elements – at least in the beginning
Try those out. Implement them.

One size fits all ---- No!

Capturing descriptions, formats, policies, transactions, etc. in a monolithic fashion
Register individual components separately
E.g., Capture legal formats & interpretations in their own records, and reference them here
E.g., Same with accepted policies

Identifiers cannot be semantic

Domain, sub-domain, and object-type are part of an ID
World view changes frequently
Non-semantic Ids are worth every penny
Search engines & registries mask the opaqueness
After all, IDs are just entities behind the scenes

Object Type controlled vocabulary enumerates apples and oranges

Collection, flow, directory, file, database, gui are not mutually exclusive
Doesn’t help the recipient make any decision looking at the descriptor
Bundle type & format into format interpretation method
Covers too many corner cases, e.g., flow-rate
Expects too many details, e.g., locator (type, access method, etc.)

6) Reports on operational I&M services

9:00am

Goal: Use by both GIMI and GEMINI

S1) XML Messaging Service

9:00am

Team

LEAD Anirban Mandal (RENCI)

Tasks

Define, prototype, demonstrate and operate a GENI Messaging service, starting at GEC13.
The XML Mesaaging service will be built by the IMF project. For a detailed task list, see the Spiral 4 SOW of the IMF project.
Support use in GENI by many tools, including GEMINI and GIMI I&M tools
Define operations plan for XML Messaging service.

Work in Progress

Summary of current status: (Rudra Dutta)

Slides

Discussion:

How do we set "topics" in XMPP server?
What is the efficiency of XML processing in XMPP?
How does the authorization work?

S2) iRODS Service

9:10am

Team

LEAD Shu Huang (RENCI)

Tasks

Start with CNRI prototype digital object archive service
Plan transition to use of iRODS service
How to move data to service: move file; move SQL DB; or?
How is persistent object identifier (i.e., handle) generated? also registered? and later resolved? Include necessary Object Identifier (OI) service.
How are partitions for different users managed?
How is data in MDOD used to set archive sharing policy? what is required in MDOD?
Operations plans for Digital Object Archive (DOA) and Object Identifier (OI) services.

Work in Progress

Summary of current status

Review of prototype Measurement Data Archive (MDA) service: (Giridhar Manepalli, CNRI)

slides

Introduction to iRODS service: (Anirban for Shu Huang, RENCI)

slides

S3) IREEL Portal Service

9:20am

Team

LEAD Christoph Dwertmann (NICTA)

Tasks:

What persistent operational services must be supported:
a) Descriptor registry required? use UNIS? DOR? iRODS? IF-MAP?
b) Object registry required? use DOR?
c) XML Messaging service
d) Measurement Information (MI)service
e) GENI User services
f) Digital Object Archive (DOA) service
Can all services can be shared by GIMI and GEMINI tools?
How do these relate to other GENI services/functions, including clearinghouse?

S4 INSTOOLS Portal Service

9:30am

Team

LEAD Jim Griffioen (U Kentucky)

Tasks:

What persistent operational services must be supported:
a) Descriptor registry required? use UNIS? DOR? iRODS? IF-MAP?
b) Object registry required? use DOR?
c) XML Messaging service
d) Measurement Information (MI)service
e) GENI User services
f) Digital Object Archive (DOA) service
Can all services can be shared by GIMI and GEMINI tools?
How do these relate to other GENI services/functions, including clearinghouse?

S5) UNIS Service

9:40am

Team

LEAD Ahmed El-Hassany (IU)
Martin Swany (IU)

Tasks

The MI service will be built by the GEMINI project using UNIS technology; see the Spiral 4 SOW of the GEMINI project for the detailed steps, which include:
1) Refactor UNIS (combined Lookup and Topology Services) to support hierarchical operation with local and global instances:
2) Update topology model to Rspec version 3 and AM API (February 2012)
3) Modifications to allow local UNIS to register with global UNIS (March 2012)
4) Initial deployment of persistent GENI Global I&M Registry (GGR) service, based on UNIS. (May 2012)
5) Complete noSQL (MongoDB) backend (September 2012)
6) Unify Perl and Python implementations (September 2012)
Define, prototype, demonstrate and operate a MI service, starting in May, 2012.
Support use in GENI by many tools, including GEMINI and GIMI I&M tools
Define operations plan for MI service.

Work in Progress

Summary of current status: (Ahmed El-Hassany)

slides

7) Goals for the next 4 months

9:50am

Team

LEAD Harry Mussman (GPO)

8) Adjourn

10:00am

Attachments (13)