2nd GENI Instrumentation and Measurement Workshop

Tuesday, June 8, 1pm - Wednesday, June 9, 2pm
Chicago O'Hare Hilton
NOTE: By invitation only

Announcement with goals, topics and reference material

The following announcement (with figures and references) was sent to the attendees to prepare for the workshop:

​Announcement, v1.1 (060410)
​Figures, v1.1 (060410)

Attendees at workshop

(Attended workshop: yes or no)

Paul Barford - University of Wisconsin – Madison (no)
Bruce Maggs – Duke University and Akamai (yes)
Harry Mussman – BBN/GPO (yes)
Vic Thomas - BBN/GPO (yes)
Evan Zhang – BBN/GPO (yes)

OML (ORBIT Measurement Library) OMF (ORBIT Management Framework)

Max Ott – NICTA (yes, by phone)
Ivan Seskar – Rutgers WINLAB (yes)

Instrumentation Tools

Jim Griffioen - Univ Kentucky (yes)

perfSONAR

Matt Zekauskas - Internet2 (no)
Jason Zurawski – Internet2 (yes)
Martin Swany - Univ Delaware (yes)
Guilherme Fernandes – Univ Delaware (yes)
Ezra Kissel – Univ Delaware (yes)

Scalable Sensing Service (S3)

Sonia Fahmy – Purdue (yes)
Puneet Sharma - HP Labs (yes)

OnTimeMeasure for network measurements

Prasad Calyam - Ohio Supercomputing Ctr (yes)

GENI Meta-Operations Center and NetKarma

Jon-Paul Herron - Indiana Univ
Camilo Viecco - Indiana Univ (yes)
Chris Small - Indiana Univ (yes)
Beth Plale - Indiana Univ (no)

Virtual Machine Introspection (VMI)

Brian Hay – Univ Alaska (yes)

Data-Intensive Cloud Control for GENI

Michael Zink - UMass Amherst (yes)

Experiment Management Service – Digital Object Registry

Jim French - CNRI (yes)
Giridhar Manepalli - CNRI (yes)
Larry Lannom – CNRI (no)

Announcement with notes

The following announcement now includes all notes from the workshop:

​Announcement, v1.2 (062510) ​word version
​Figures, v1.2 (062510)

Priority topics

The following priority topics were identified at the workshop, and teams of attendees (and some non-attendees0 were identified for each topic to discuss it, and write a text summary for review by the WG at GEC8.

Topic 1 GENI I&M Use Cases

Team members:

Paul Barford - University of Wisconsin – Madison (no)
Jim Griffioen - Univ Kentucky (yes)
Prasad Calyam* - Ohio Supercomputing Ctr (yes)
Camilo Viecco - Indiana Univ (yes)
Brian Hay – Univ Alaska (yes)

*agreed to organize first writing and discussion

Identify all user groups, and provide basic use cases:

1. Experiment researchers
2. Experiment (opt-in) users (see ​http://groups.geni.net/geni/attachment/wiki/041409NYCOptInWGAgenda/071509%20%20GENI-SE-OI-Overview-01.4.pdf for listing of opt-in issues, such as privacy)
3. Central (i.e., GMOC) operators
4. Cluster and aggregate providers and operators
5. Archive service providers and operators
6. Researchers that use archived measurement data, archived by other researchers (DatCat model)

Topic 2 GENI I&M Services

Team members:

Harry Mussman* – BBN/GPO (yes)
Evan Zhang – BBN/GPO
Giridhar Manepalli - CNRI (yes)
Chris Small - Indiana Univ (yes)
Beth Plale - Indiana Univ (yes)

*agreed to organize first writing and discussion

Summarize current view of:

Measurement Orchestration (MO) Service
Measurement Point (MP) Service
Measurement Information (MI) Service
Measurement Collection (MC) Service
Measurement Analysis and Presentation (MAP) Service
Measurement Data Archive (MDA) Service

Need: Basic definition of a Measurement Data Archive (MDA) Service

Identify different types of services:

Type 1: Dedicated service platform, with dedicated sliver, for customized information.
(Completely dedicated to an experiment)
Type 2: Common service platform, with dedicated slivers, for customized information.
(Common portion, plus parts associated with different experiments)
Type 3: Common service, for common or customized information.
(Common service, with data provided to multiple experiments)

Topic 3 GENI I&M Resources

Team members:

Vic Thomas - BBN/GPO (yes)
Jim Griffioen* - Univ Kentucky (yes)
Martin Swany - Univ Delaware (yes)
Camilo Viecco - Indiana Univ (yes)
Brian Hay – Univ Alaska (yes)
Giridhar Manepalli - CNRI (yes)

*agreed to organize first writing and discussion

Significant question uncovered at workshop:
Jim on 6/25 via email: We should involve Rob Ricci in the discussion.

Consider these resources for I&M capabilities:

1. Hosts, VMs, etc.
2. Network connectivity
3. Software, e.g., I&M software that can be included in an experiment
4. I&M services
5. I&M data flows and file transfers
6. I&M data files stored in archives

For each item, consider how to:

Create
Name
Register and discover
Authorize and assign

Does each item have:

Unique and persistent name?
Unique and persistent identifier?
Need to carefully consider this for all of GENI

For each item, consider:

Ownership
What sort of policies the owner may want to apply

How are each of these discovered, specified, authorized and assigned:

Always by mechanisms provided by the CF? With CF plus additional mechanisms?

Consider example of LS in perfSONAR
Consider example of data file stored in archive, owned by an experimenter Need to define and then compare these options
Need to understand interop with CF for each option
Does CF setup secondary authorization mechanisms in some cases? If so, how?

Topic 4 GENI I&M Measurement Plane and Interfaces

Team members:

Harry Mussman* – BBN/GPO (yes)
Ezra Kissel – Univ Delaware (yes)
Chris Small - Indiana Univ (yes)

*agreed to organize first writing and discussion

Consider:

IP network
Layer 2 (VLAN) connections

Discuss:

Which protocols are active
Access to resources in aggregates, even when resources are in private address space, via GWs or proxies
How to provide authentication and authorization
How to provide QoS to protect measurement traffic
How to provide QoS to protect other traffic when measurement traffic is large.
Reserve bandwidth?

Martin on 6/28: Consider XSP (extensible session protocol) to provide transport layer GW functions.

Topic 5 GENI I&M Interfaces and Protocols (APIs): Manage Services

Team members:

Vic Thomas - BBN/GPO (yes)
Ivan Seskar – Rutgers WINLAB (yes)
Max Ott – NICTA (yes, by phone)
Sonia Fahmy* – Purdue (yes)
Giridhar Manepalli - CNRI (yes)

*agreed to organize first discussion and writing

Define an approach based on OMF/OML and S3:

HTTP(S)
REST vs SOAP
Authorization by credentials or ? If credentials, how to revoke?
Pass XML fragments
Define basic API

Topic 6 GENI I&M Interfaces and Protocols (APIs): Data Flows and Data File Transfers

Team members:

Harry Mussman* – BBN/GPO (yes)
Ivan Seskar – Rutgers WINLAB (yes)
Max Ott – NICTA (yes, by phone)
Ezra Kissel – Univ Delaware (yes)
Prasad Calyam - Ohio Supercomputing Ctr (yes)
Michael Zink - UMass Amherst (yes)

*agreed to organize first writing and discussion

Consider data flows and data file transfers between all services

Define range of options:

What:

Data flows
Data files transfers

Type:

Pull
Push
Pub/Sub

Protocol:

SNMP
SCP
FTP and gridFTP
HTTP
XMPP
TCP
SCTP

Consider:

Naming
Discovery
Connectivity
Authentication and authorization mechanisms

Map to current projects, giving examples:
Consider: Minimum set required for GENI

Topic 7 GENI I&M Interfaces and Protocols (APIs): Registration and Discovery of Services with Available Measurement Data

Team members:

Jason Zurawski* – Internet2 (yes)
Prasad Calyam - Ohio Supercomputing Ctr (yes)
*agreed to organize first writing and discussion

Consider approach used in perfSONAR

Summarize for:

Services with data flows
Also sources of file transfers?
Also GUIs?

Topic 8 GENI I&M Interfaces and Protocols (APIs): GUIs

Team members:

Jeremy Reed - Univ Kentucky (yes)
Guilherme Fernandes* – Univ Delaware (yes)
Puneet Sharma - HP Labs (yes)
*Agreed to organize team

Define overall goals for GENI GUIs

Types of GUIs:

Control experiments
Display I&M results
Report status
View archive service

Consider portal, for access to multiple GUIs

Consider need for authentication and authroization

Topic 9 GENI Measurement Data Schema

Team members:

Bruce Maggs – Duke University and Akamai (yes)
Max Ott – NICTA (yes, by phone)
Ivan Seskar – Rutgers WINLAB (yes)
Martin Swany* - Univ Delaware (yes)
Camilo Viecco - Indiana Univ (yes)
Michael Zink - UMass Amherst (yes)
Jim French - CNRI (yes)

*agreed to organize first discussion and writing

Consider:

Measurement data schema
Metadata schema
Metadata contents

Consider measurement data schema and/or metadata schema from:

perfSONAR
GMOC-provided
Current OML
Proposed using IPFIX
NetCDF (as used by DI Cloud)

Consider: Minimum set required for GENI

Provide overall template for GENI metadata, considering above.
Which items in GENI metadata template are:

Required?
Invariant?

Text

All text, word and PDF

Figures

​060310 v1.2 All I&M figures
All figures, visio and PDF

Individual figures, visio and PDF

Project Summaries

Instrumentation Tools

OMF/OML

perfSONAR

Scalable Sensing Service

On Time Measure

Data Intensive Cloud

Digital Object Registry

References

All references

Individual references

Ref GIMS_Design_UseCases: “Use-cases for GENI Instrumentation and Measurement Architecture Design”

Ref MeasPlane-1: “RESTful Web Services vs. “Big” Web Services: Making the Right Architectural Decision”

Ref OMF_OML-1: “XDR: External Data Representation Standard”

Ref OMF_OML-2: “ORBIT Measurements Framework and Library (OML): Motivations, Design, Implementation, and Features”

Ref OMF-OML-3: “OML Overview” slides

Ref OMF-OML-4: “Measurement Architectures for Network Experiments with Disconnected Mobile Nodes”

Ref InsTools-1: “Architectural Design and Specification of the INSTOOLS Measurement System”

Ref perfSONAR-1: “Scalable Framework for Representation and Exchange of Network Measurements”

Ref perfSONAR-2: “An Extensible Schema for Network Measurement and Performance Data”

Ref perfSONAR-3: “NM-WG/perfSONAR Topology Schema”

Ref GMOC-1: “GMOC Topology-Entity Data Exchange Format Specification”

Ref GMOC-2: “Proposal: Use of URN's as GENI Identifiers”