2nd GENI Instrumentation and Measurement Workshop

Tuesday, June 8, 1pm - Wednesday, June 9, 2pm
Chicago O'Hare Hilton
NOTE: By invitation only

Announcement with goals, topics and reference material

The following announcement (with figures and references) was sent to the attendees to prepare for the workshop:

​Announcement, v1.1 (060410)
[ Figures, v1.1 (060410)]

Attendees at workshop

(attended workshop: yes or no)

Paul Barford - University of Wisconsin – Madison (no) Bruce Maggs – Duke University and Akamai (yes) Harry Mussman – BBN/GPO (yes) Vic Thomas - BBN/GPO (yes) Evan Zhang – BBN/GPO (yes)

OML (ORBIT Measurement Library) OMF (ORBIT Management Framework)

Max Ott – NICTA (yes, by phone) Ivan Seskar – Rutgers WINLAB (yes)

Instrumentation Tools

Jim Griffioen - Univ Kentucky (yes)

perfSONAR

Matt Zekauskas - Internet2 (no) Jason Zurawski – Internet2 (yes) Martin Swany - Univ Delaware (yes) Guilherme Fernandes – Univ Delaware (yes) Ezra Kissel – Univ Delaware (yes)

Scalable Sensing Service (S3)

Sonia Fahmy – Purdue (yes) Puneet Sharma - HP Labs (yes)

OnTimeMeasure for network measurements

Prasad Calyam - Ohio Supercomputing Ctr (yes)

GENI Meta-Operations Center and NetKArma

Jon-Paul Herron - Indiana Univ Camilo Viecco - Indiana Univ (yes) Chris Small - Indiana Univ (yes) Beth Plale - Indiana Univ (yes)

Virtual Machine Introspection (VMI)

Brian Hay – Univ Alaska (yes)

Data-Intensive Cloud Control for GENI

Michael Zink - UMass Amherst (yes)

Experiment Management Service – Digital Object Registry

Jim French - CNRI (yes) Giridhar Manepalli - CNRI (yes) Larry Lannom – CNRI (no)

Announcement with notes

The following announcement now includes all notes from the workshop:

[ Announcement, v1.2 (062510)]
[ Figures, v1.2 (062510)]

Priority topics

The following priority topics were identified at the workshop, and teams of attendees (and some non-attendees0 were identified for each topic to discuss it, and write a text summary for review by the WG at GEC8.

6.1 GENI I&M use cases

Team members:

Paul Barford - University of Wisconsin – Madison (no) Jim Griffioen - Univ Kentucky (yes)

• Prasad Calyam - Ohio Supercomputing Ctr (yes) Camilo Viecco - Indiana Univ (yes) Brian Hay – Univ Alaska (yes)
• agreed to organize first discussion and writing

Identify all user groups, and provide basic use cases:

1) GMOC operations group (NOC) 2) Cluster and aggregate operations groups (NOCs) 3) Archive service providers 4) Experiment researchers 5) Experiment (opt-in) users (see ​http://groups.geni.net/geni/attachment/wiki/041409NYCOptInWGAgenda/071509%20%20GENI-SE-OI-Overview-01.4.pdf for listing of opt-in issues, such as privacy) 6) Researchers that use measurement data archived by other researchers (DatCat model)

6.2 GENI I&M services

Team members:

• Harry Mussman – BBN/GPO (yes) Evan Zhang – BBN/GPO Giridhar Manepalli - CNRI (yes) Chris Small - Indiana Univ (yes) Beth Plale - Indiana Univ (yes)
• agreed to organize first discussion and writing

Summarize current view

Identify different types of services:

Completely dedicated to an experiment Common portion, plus parts associated with different experiments Common service, with data provided to multiple experiments

Need: Basic definition of an archive service

6.3 GENI I&M resources

Team members:

Vic Thomas - BBN/GPO (yes)

• Jim Griffioen - Univ Kentucky (yes) Martin Swany - Univ Delaware (yes) Camilo Viecco - Indiana Univ (yes) Brian Hay – Univ Alaska (yes) Giridhar Manepalli - CNRI (yes)
• agreed to organize first discussion and writing

Significant question uncovered at workshop! Jim on 6/25 via email: We should involve Rob Ricci in the discussion.

What are resources:

1) Hosts, VMs, etc. 2) Network connectivity 3) Software, e.g., I&M software that can be included in an experiment 3) I&M services 4) I&M data flows and file transfers 5) I&M data files stored in archives

How are each of these discovered, specified, authorized and assigned:

a) Always by mechanisms provided by the CF? b) With CF plus additional mechanisms? Consider example of LS in perfSONAR Consider example of data file stored in archive, owned by an experimenter

Goals:

Need to define and then compare these options Need to understand interop with CF for each option Does CF setup secondary authorization mechanisms in some cases? If so, how?

Does each item have:

Unique and persistent name? Unique and persistent identifier? Need to carefully consider this for all of GENI

For each item, consider how to:

Create Name Register and discover Authorize and assign

For each item, consider:

Ownership What sort of policies the owner may want to apply

6.4 GENI I&M measurement plane and interfaces

Team members:

• Harry Mussman – BBN/GPO (yes) Ezra Kissel – Univ Delaware (yes) Chris Small - Indiana Univ (yes)
• agreed to organize first discussion and writing

Consider:

IP network Layer 2 (VLAN) connections

Discuss

Which protocols are active Access to resources in aggregates, even when resources are in private address space, via GWs or proxies How to provide authentication and authorization How to provide QoS to protect measurement traffic How to provide QoS to protect other traffic when measurement traffic is large. Reserve bandwidth?

Martin on 6/28: Consider XSP (extensible session protocol) to provide transport layer GW functions. 6.5 GENI I&M interfaces and protocols (APIs): manage services

Vic Thomas - BBN/GPO (yes) Ivan Seskar – Rutgers WINLAB (yes) Max Ott – NICTA (yes, by phone)

• Sonia Fahmy – Purdue (yes) Giridhar Manepalli - CNRI (yes)
• agreed to organize first discussion and writing

Define an approach based on OMF/OML and S3:

HTTP(S) REST vs SOAP Authorization by credentials or ? If credentials, how to revoke? Pass XML fragments Define basic API

6.6 GENI I&M interfaces and protocols (APIs): data flows and data file transfers

• Harry Mussman – BBN/GPO (yes) Ivan Seskar – Rutgers WINLAB (yes) Max Ott – NICTA (yes, by phone) Ezra Kissel – Univ Delaware (yes) Prasad Calyam - Ohio Supercomputing Ctr (yes) Michael Zink - UMass Amherst (yes)
• agreed to organize first discussion and writing

Consider data flows and data file transfers between all services

Define range of options:

What:

Data flows Data files transfers

Type:

Pull Push Pub/Sub

Protocol:

SNMP SCP FTP and gridFTP HTTP XMPP TCP SCTP

Consider:

Naming Discovery Connectivity Authentication and authorization mechanisms

Map to current projects, giving examples: Consider: Minimum set required for GENI

6.7 GENI I&M interfaces and protocols (APIs): service registration and discovery

Team members:

• Jason Zurawski – Internet2 (yes) Prasad Calyam - Ohio Supercomputing Ctr (yes)
• agreed to organize first discussion and writing

Consider approach used in perfSONAR

Summarize for:

Services with data flows Also sources of file transfers? Also GUIs?

6.8 GENI I&M interfaces and protocols (APIs): GUIs

Team members:

Jeremy Reed - Univ Kentucky (yes)

• Guilherme Fernandes – Univ Delaware (yes) Puneet Sharma - HP Labs (yes)
• Agreed to organize team

Define overall goals for GENI GUIs

Types of GUIs:

Control experiments Display I&M results Report status View archive service

Consider portal, for access to multiple GUIs

Consider need for authentication and authroization

6.9 GENI measurement data schema

Team members:

Bruce Maggs – Duke University and Akamai (yes) Max Ott – NICTA (yes, by phone) Ivan Seskar – Rutgers WINLAB (yes)

• Martin Swany - Univ Delaware (yes) Camilo Viecco - Indiana Univ (yes) Michael Zink - UMass Amherst (yes) Jim French - CNRI (yes)
• agreed to organize first discussion and writing

Consider:

Measurement data schema Metadata schema Metadata contents

Consider measurement data schema and/or metadata schema from:

perfSONAR GMOC-provided Current OML Proposed using IPFIX NetCDF (as used by DI Cloud)

Consider: Minimum set required for GENI

Provide overall template for GENI metadata, considering above. Which items in GENI metadata template are:

Required? Invariant?

Text

All text, word and PDF

Figures

All figures, visio and PDF

Individual figures, visio and PDF

References

All references

Individual references

Ref GIMS_Design_UseCases: “Use-cases for GENI Instrumentation and Measurement Architecture Design”

Ref MeasPlane-1: “RESTful Web Services vs. “Big” Web Services: Making the Right Architectural Decision”

Ref OMF_OML-1: “XDR: External Data Representation Standard”

Ref OMF_OML-2: “ORBIT Measurements Framework and Library (OML): Motivations, Design, Implementation, and Features”

Ref OMF-OML-3: “OML Overview” slides

Ref OMF-OML-4: “Measurement Architectures for Network Experiments with Disconnected Mobile Nodes”

Ref InsTools-1: “Architectural Design and Specification of the INSTOOLS Measurement System”

Ref perfSONAR-1: “Scalable Framework for Representation and Exchange of Network Measurements”

Ref perfSONAR-2: “An Extensible Schema for Network Measurement and Performance Data”

Ref perfSONAR-3: “NM-WG/perfSONAR Topology Schema”

Ref GMOC-1: “GMOC Topology-Entity Data Exchange Format Specification”

Ref GMOC-2: “Proposal: Use of URN's as GENI Identifiers”