[[PageOutline]] = Instrumentation and Measurement (I&M) Working Session = Wednesday, November 2, 2011, 8:3oam - 12noon [[BR]] Room: [[BR]] Session Leader: Harry Mussman, (GPO, Raytheon BBN Technologies) [[BR]] == Description == This session will review the GENI I&M design and prototyping projects planned for Spiral 4 (this year), including the two large Solicitation 3 I&M D&P projects and those continuing projects from Solicitation 2. Then, it will define and discuss the priority topics that must be resolved between now and GEC13. [[BR]] == Summary == == Agenda == This is a tentative agenda, which may change. [[BR]] == 1) Introductions == 8:30am [[BR]] == 2) GENI I&M D&P Projects for Spiral 4 == 8:35am [[BR]] Harry Mussman [[BR]] === 2.1) New Sol 3 Projects === GIMI [[BR]] PI: Mike Zink (UMass Amherst) [[BR]] Co-PI/Key: Max Ott (NICTA); Ilia Baldine (RENCI) [[BR]] Goals: [[BR]] Starts with OML, and provides easy-to-use tools in GENI environment [[BR]] First focus on I&M use cases for experiments, with default OML Client in nodes that gathers passive measurements like INSTOOLS [[BR]] First supported aggregates: ORCA servers/VMs and RENCI racks [[BR]] First utilized experiment control tools: GUSH, with additions [[BR]] First utilized interfaces: GENI AM API for resources; ns3 for topology; ssh; add OMF for orchestration [[BR]] Requires XML messaging service, with pub/sub, in public IP space [[BR]] Introduces portal service, with: OML Server, for collection; Results Service, for presentation; measurement orchestration service using OMF; documentation service to create and edit MDOD. Can this be shared with GEMINI project? [[BR]] Introduces iRODS service for user workspace and archive. Start with CNRI prototype? Can GEMINI use iRODS? Can iRODS serve as registry for MDOD? [[BR]] Begin: support for WiMAX sites [[BR]] Consider: adding OML server with perfSONAR interface, for sharing data [[BR]] Later: add sensor measurement data, carried in data network [[BR]] GEMINI [[BR]] PI: Martin Swany (Indiana U) [[BR]] Co-PI/Key: Chris Small (Indiana U); Eric Boyd (Internet2); Jim Griffioen (U Kentucky); Zongming Fei (U Kentucky) [[BR]] Goals: [[BR]] Starts with INSTOOLS, and continues to provide easy-to-use tools in GENI environment [[BR]] Consider: how to gather customized data from application [[BR]] First focus on I&M use cases for experiments [[BR]] First supported aggregates: protoGENI servers/VMs; can these be used for HP racks? [[BR]] First utilized experiment control tools: FLACK, with additions [[BR]] First utilized interfaces: protoGENI API for resources; ns3 for topology; ssh; [[BR]] Continues portal to presentation service; need to add documentation service to create and edit MDOD. Can portal be shared with GIMI project? [[BR]] Continues use of Kentucky and CNRI services for user workspace and archive. Consider move to iRODS? [[BR]] Starts with LAMP/perfSONAR, and continues to provide for infrastructure measurement, and for sharing of data [[BR]] Continue to setup within resources assigned by slice mechanism [[BR]] Consider: monitoring of clusters/racks [[BR]] Consider: measurements of Layer 2 and OpenFlow paths [[BR]] First focus on I&M use cases for infrastructure measurement [[BR]] First supported aggregates: protoGENI servers/VMs; can these be used for HP racks? [[BR]] First utilized experiment control tools: FLACK, with additions [[BR]] First utilized interfaces: protoGENI API for resources; ns3 for topology; ssh; [[BR]] Continues home and global Lookup Services [[BR]] Create and map MDOD into metadata that is registered [[BR]] Continue to use perfSONAR clients to present data; can these be integrated into portal? [[BR]] Consider how INSTOOLS and LAMP/perfSONAR tools might be shared or merged [[BR]] === 2.2) Continuing Sol 2 Projects === LAMP/perfSONAR [[BR]] PI: Martin Swany (Indiana U) [[BR]] Co-PI/Key: Eric Boyd (Internet2); Jason Zurawski (Internet 2); Guilherme Fernandes (U Delaware); Ezra Kissel (U Delaware) [[BR]] Goals: [[BR]] Provides operational home and global Lookup Services [[BR]] Creates and maps MDOD into metadata that is registered in Lookup Service [[BR]] Expand authorization at interface where data is shared; introduce use of GENI credentials [[BR]] Continue to participate in the GENI I&M effort, particularly in defining the schema for the “measurement data object descriptor”, and mapping it to the perfSONAR metadata that is registered with the Lookup Service. [[BR]] Provide LAMP images for creating “infrastructure measurement slices” in both protoGENI and ORCA environments. [[BR]] Install and operate a perfSONAR Lookup Service, for use with “infrastructure measurement slices”. [[BR]] Demonstrate and/or help others to demonstrate “infrastructure measurement slices” in both protoGENI and ORCA environments, with the data available at the perfSONAR MP is registered with the operational Lookup Service. [[BR]] Demonstrate using GENI credentials to authorize access to the data available at the perfSONAR MP. [[BR]] Define a plan for using ABAC to replace simple credentials to authorize the access to the data available at a perfSONAR MP. [[BR]] OnTimeMeasure [[BR]] PI: Prasad Calyam (Ohio Super Center) [[BR]] Co-PI: Paul Schopis (OARnet) [[BR]] Goals: Continue to participate in the GENI I&M effort, particularly in defining the schema for the “measurement data object descriptor”. [[BR]] Work with the LAMP project to understand the LAMP/perfSONAR tools, and demonstrate using them to implement “infrastructure measurement slices” in today’s GENI backbone and access networks. [[BR]] Assuming LAMP/perfSONAR tools, develop a plan to evaluate the performance of today’s GENI IP backbone and access networks, using persistent or temporary “infrastructure measurement slices”, and utilizing existing perSONAR nodes in I2 and/or other networks. [[BR]] Define a procedure for evaluating the performance of GENI’s backbone and access networks when carrying L2 VLANs; start with the existing IP measurement tools including in the LAMP/perfSONAR tools, but consider tools that are specialized for evaluating the performance of L2 networks. [[BR]] Define a procedure for evaluating the performance of GENI’s backbone and access networks when using OpenFlow (e.g., “Tango GENI”); consider how to monitor the performance of OF networks, and introducing new specialized tools. [[BR]] NetKarma [[BR]] PI: Beth Plale (Indiana U) [[BR]] Co-PI/Key: Chris Small (Indiana U); Scott Jensen (Indiana U) [[BR]] Goals: Participate in the GENI I&M effort, particularly in defining the schema for the “measurement data object descriptor”, and uses of “descriptors” in tracking provenance. [[BR]] Work with the GENI I&M community to define extensions to the “descriptor” that would locate and describe objects beyond “measurement data objects” associated with an experiment, such as scripts, images, etc. [[BR]] Modify your tools to create, gather and forward “descriptors”, that could locate and describe all objects associated with an experiment; demonstrate how “descriptors” can be automatically created. [[BR]] Work with the GENI I&M community to define a modified “descriptor” that would provide a standardized GENI “resource event record”; these records could describe events such as assignments, faults or errors. [[BR]] Add modules to your tools to create, gather and forward “resource event records”, that when logged could fully describe the sequence of an experiment. [[BR]] Demonstrate forwarding your “records” using pub/sub servers and to repositories provided by the IMF project. [[BR]] Demonstrate your tools, and support experimenters in using them. [[BR]] Participate in the GENI I&M effort, in fitting your provenance tools into the “experimenter’s portal”. [[BR]] IMF [[BR]] PI: Rudra Dutta (NCSU) [[BR]] Co-PI/Key: George Rouskas (NCSU); Ilia Baldine (RENCI); Keren Bergman (Columbia U) [[BR]] Goals: [[BR]] Participate in the GENI I&M effort, focusing on providing basic capabilities, and on extending the architecture to include a pub/sub mechanism. [[BR]] Prototype and demonstrate a GENI I&M capability to manage I&M services/slivers within a slice using a “measurement orchestration service”; use ORBIT Management Framework (OMF) software modules provided by NICTA, including the Experiment Controller (EC) and the Resource Controller (RC), that communicate using a pub/sub mechanism utilizing an XMPP server. [[BR]] Define and prototype an authorization mechanism for your sliver management service; consider message signing being proposed by NICTA. [[BR]] Demonstrate your capability to manage a sliver. [[BR]] Prototype and demonstrate a GENI I&M capability to transport event (or measurement) records; include software modules that collect the records, communicate using a pub/sub mechanism utilizing an XMPP server, archive the records in a repository, search the records, and display the records. [[BR]] Define and prototype an authorization mechanism for your event record transport service. [[BR]] Demonstrate your capability by transporting and distributing standardized GENI “resource event records”, as defined by the NetKarma project; these records could describe events such as assignments, faults or errors. [[BR]] Prototype and operate a GENI I&M service to provide pub/sub services utilizing an XMPP server, for use by your sliver management and event record transport services. [[BR]] Define and prototype a resource assignment mechanism (using the GENI AM API, if possible) for users (experimenters and operators) to define nodes within your GENI pub/sub service, as they establish modules that publish or subscribe. [[BR]] Prototype and operate a data repository associated with your pub/sub service, that could subscribe to a node and archive all event messages. [[BR]] DOR [[BR]] PI: Larry Lannom (CNRI) [[BR]] Co-PI/Key: Giridhar Manepalli (CNRI); Jim French (CNRI); Christophe Blanchi (CNRI) [[BR]] Goals: [[BR]] Continue to deploy a prototype Measurement Data Archive (MDA) service, and support integration with GENI I&M tools. [[BR]] Participate in the GENI I&M effort, particularly in the MDA specification effort that includes defining metadata model for archived entities, defining access-control layer, and integrating control framework authorization scheme with the MDA. [[BR]] Enforce metadata schema on archived objects, and enable access control rights as defined by the MDA specification. [[BR]] Integrate GENI identity management and authorization scheme if feasible and applicable, resources permitting. [[BR]] Evangelize and demonstrate how the MDA/DOR service can be used to archive published data. [[BR]] === 2.3 GIMI Project Overview === 8:45am [[BR]] PI: Mike Zink (UMass Amherst) [[BR]] co-PI/Key: Max Ott (NICTA); Ilia Baldine (RENCI) [[BR]] Goals [[BR]] Configuration [[BR]] Issues to be resolved [[BR]] [ slides] [[BR]] === 2.4 GEMINI Project Overviews === 9:05am [[BR]] PI: Martin Swany (Indiana U) [[BR]] Co-PI/Key: Chris Small (Indiana U); Eric Boyd (Internet2); Jim Griffioen (U Kentucky); Zongming Fei (U Kentucky) [[BR]] Goals [[BR]] Configuration [[BR]] Issues to be resolved [[BR]] [ slides] [[BR]] === 2.4 LAMP/perfSONAR Project Overviews === 9:25am [[BR]] PI: Martin Swany (Indiana U) [[BR]] Co-PI/Key: Eric Boyd (Internet2); Jason Zurawski (Internet 2); Guilherme Fernandes (U Delaware); Ezra Kissel (U Delaware) [[BR]] Goals [[BR]] Configuration [[BR]] Issues to be resolved [[BR]] [ slides] [[BR]] === 2.6 OnTimeMeasure Project Overview === 9:30am [[BR]] PI: Prasad Calyam (Ohio Super Center) [[BR]] Co-PI: Paul Schopis (OARnet) [[BR]] Goals [[BR]] Configuration [[BR]] Issues to be resolved [[BR]] [ slides] [[BR]] === 2.7 NetKarma Project Overview === 9:35am [[BR]] PI: Beth Plale (Indiana U) [[BR]] Co-PI/Key: Chris Small (Indiana U); Scott Jensen (Indiana U) [[BR]] Goals [[BR]] Configuration [[BR]] Issues to be resolved [[BR]] [ slides] [[BR]] === 2.8 IMF Project Overview === 9:40am [[BR]] PI: Rudra Dutta (NCSU) [[BR]] Co-PI/Key: George Rouskas (NCSU); Ilia Baldine (RENCI); Keren Bergman (Columbia U) [[BR]] Goals [[BR]] Configuration [[BR]] Issues to be resolved [[BR]] [ slides] [[BR]] === 2.9 DOR Project Overview === 9:45am [[BR]] PI: Larry Lannom (CNRI) [[BR]] Co-PI/Key: Giridhar Manepalli (CNRI); Jim French (CNRI); Christophe Blanchi (CNRI) [[BR]] Goals [[BR]] Configuration [[BR]] Issues to be resolved [[BR]] [ slides] [[BR]] === 2.10 Discussion === 9:50am == 3) Break == 10:00am [[BR]] == 4) GENI I&M Operational Environment at End of Spiral 4 == 10:30am [[BR]] == 5) Priority Topics and Next Steps == 10:35am [[BR]] === 5.1) Overview === 10:35am [[BR]] I&M Use Cases for Experiments [[BR]] I&M Use Cases for Infrastructure Measurement [[BR]] Platform Targets [[BR]] Experiment and Measurement Orchestration Tools [[BR]] Interfaces/Protocols Between Tools and Services [[BR]] Descriptor Schema and Registry Service [[BR]] Object Names and Registry Service [[BR]] XML Messaging Service [[BR]] Lookup Service [[BR]] Portal Service [[BR]] User Workspace and Archive Services [[BR]] Operational Services [[BR]] Services Shared by GIMI and GEMINI [[BR]] Support for Experimenters [[BR]] Support for Operators [[BR]] === 5.2) I&M for Experiments === 10:40am [[BR]] I&M Use Cases for Experiments [[BR]] Support for Experimenters [[BR]] a) How do we work toward's Max Ott's vision for experiment support? b) Experiment process: 1) Setup and run experiment using Experiment Control Tools, Experiment Orchestration Service and Lab Portal. 2) Setup and run measurements using services within slice, and persistent services. Option: get measurement data from persistent MP. 3) Gather MD, and observe during experiment; formulate MDOD. 4) Archive MD with MDOD. 5) Share archived MD with others, per policy included within MDOD. 6) Pull MD out of archive, to analyze and/or visualize. c) What support must be provided for Experimenters? how? [[BR]] d) Who: === 5.3) I&M Use Cases for Infrastructure Measurement === 10:50am [[BR]] I&M Use Cases for Infrastructure Measurement [[BR]] Support for Operators [[BR]] a) Infrastructure measurement process: 1) Setup persistent or on demand infrastructure measurement slices. 2) Make passive measurements or make active measurements, and gather results. 3) Gather MD, and observe as it is gathered; formulate MDOD. 4) Store MD in collector, describe with MDOD, and register MDOD so that MD can be shared. 5) Typically share MD with Aggregate Operator, GMOC and/or Experimenters, per policy written into MDOD. 6) Pull MD out of collector, analyze and visualize. 7) Archive MD with MDOD. 8) Share archived MD with others, per policy included within MDOD. 9) Pull MD out of archive, to analyze and/or visualize. b) What support must be provided for Operator? how? [[BR]] c) Who: === 5.4) GENI Environment === 11:00am [[BR]] Platform Targets [[BR]] Experiment and Measurement Orchestration Tools [[BR]] Interfaces/Protocols Between Tools and Services [[BR]] === 5.5) Descriptors, Objects and Registries === 11:10am [[BR]] Descriptor Schema and Registry Service [[BR]] Object Names and Registry Service [[BR]] === 5.6) Messaging Service === 11:20am [[BR]] XML Messaging Service [[BR]] === 5.7) Lookup Service === 11:25am [[BR]] Lookup Service [[BR]] === 5.8) Portal Service === 11:30am [[BR]] Portal Service [[BR]] === 5.9) User Workspace and Archive Services === 11:40am [[BR]] User Workspace and Archive Services [[BR]] === 5.10 Operational Services === 11:50am [[BR]] Operational Services [[BR]] Services Shared by GIMI and GEMINI [[BR]] == 6) Adjourn == 12noon [[BR]] == Background Reading == [http://groups.geni.net/geni/wiki/GeniInstMeas I&M integration wiki][[BR]] [http://groups.geni.net/geni/wiki/GENIIandMCAPCAT GENI I&M Capabilities Catalog][[BR]] [http://groups.geni.net/geni/wiki/GeniInstrumentationandMeasurementsArchitecture GENI I&M Architecture document][[BR]]