[[PageOutline]] = Instrumentation and Measurement (I&M) Working Session = Thursday, July 28, 2011, 1:00pm - 3:00pm [[BR]] Room: [[BR]] Session Leader: Harry Mussman, (GPO, Raytheon BBN Technologies) [[BR]] == Description == This session will review the latest DRAFT of the measurement_data_object_descriptor, commonly known as "metadata"; review the status of current I&M design and prototyping efforts; and review new topics and next steps. [[BR]] == Summary == == Agenda == This is a tentative agenda, which may change. [[BR]] === 1) Introductions === 1:00pm [[BR]] === 2) Measurement Data Object Descriptor (MDOD) === 1:05pm [[BR]] Harry Mussman[[BR]] ==== 2.1) Overview ==== Format (syntax) [[BR]] Goal remains: one data model for GENI [[BR]] Assume: transfer format is XML record (file) [[BR]] Have example in XML-like format, and schema/types in RelaxNG [[BR]] Need: first software to generate (and consume) MDODs [[BR]] Contents (semantics) [[BR]] Few required elements, many optional elements [[BR]] To meet use cases in arch document, sec 4.5 [[BR]] Want one-for-one mapping to key elements in current tools (started: perfSONAR; GENI rspec) [[BR]] Point-of-view [[BR]] Often global; therefore includes locators for data objects; must be updated as objects are copied, moved [[BR]] Sometimes associated; need to understand [[BR]] Should we consider: storing all MDODs in global, MI srvc? this becomes a MD object registry [[BR]] Structure [[BR]] identifier(s) [[BR]] descriptor(s) [[BR]] holder(s) [[BR]] ==== 2.2) Use Cases ==== Read the following sections of the [http://groups.geni.net/geni/attachment/wiki/GeniInstrumentationandMeasurementsArchitecture/122010i%20GENI-SE-IM-ARCH-1.0.doc Architecture document:] [[BR]] 4.5 GATHERING, TRANSFERRING AND SHARING MD [[BR]] 4.5.1 Schema and Provenance of MD [[BR]] schema for MD object contained in MDOD [[BR]] provenance of MD object contained in MDOD [[BR]] 4.5.2 Gathering MD into a Slice [[BR]] by first holder, the "owner" [[BR]] starts MDOD [[BR]] as owner, sets sharing policy, disposal policy [[BR]] as gatherer, deals with privacy [[BR]] 4.5.3 Transferring MD between I&M Services in the Same Slice [[BR]] holder updates MDOD as transfers MD object between locations [[BR]] option: registers MDOD with MI service [[BR]] 4.5.4 Transferring MD between I&M Services in Different Slices [[BR]] e.g., holder 1 to holder 2 [[BR]] holder 1 authorizes transfer, sends object and copy of MDOD [[BR]] holder 1 records the transfer in their MDOD [[BR]] holder 2 accepts MD object and MDOD, and adds their local location to their MDOD [[BR]] holder 2 tracks further transfers in their MDOD [[BR]] 4.5.5 Sharing MD with Others [[BR]] e.g., holder 1 offers MD object at a portal, and registers this with MI srvc [[BR]] user 7 requests MD object [[BR]] holder 1 checks their sharing policy, authorizes transfer, and records the transfer in their MDOD [[BR]] e.g., holder 1 transfers MD object to DOA srvc [[BR]] user 9 requests MD object from DOA srvc [[BR]] DOA srvc checks sharing policy in associated MDOD, authorizes transfer, and records the transfer in their MDOD [[BR]] Use these [http://groups.geni.net/geni/attachment/wiki/GeniInstrumentationandMeasurementsArchitecture/Visio-4.x%20%20030911%20_IM-ARCH-Figures.pdf figures][[BR]] A summary of the topic is presented in these [http://groups.geni.net/geni/attachment/wiki/GEC10InstMeasWorkingSession/031711_GatheringTransferringMD_I%26MArch_Overview.pptx slides][[BR]] See: I&M in Experimenter's Slice [[BR]] [[Image(Visio-070111_UseCases_Services_MDOD_Page_01.jpg, 90%)]] [[BR]] See: I&M in Operator's Infrastructure Measurement Slice [[BR]] For example: a GENI operator could get resources, setup a long-running measurement slice, and make data available to other operators or even experimenters. [[BR]] [[Image(Visio-070111_UseCases_Services_MDOD_Page_02.jpg, 90%)]] [[BR]] ==== 2.3) Content (Semantics) ==== Read the following sections of the [http://groups.geni.net/geni/attachment/wiki/GeniInstrumentationandMeasurementsArchitecture/122010i%20GENI-SE-IM-ARCH-1.0.doc Architecture document:] [[BR]] 8 SCHEMA AND ELEMENTS FOR MEASUREMENT DATA OBJECT DESCRIPTORS (MDODS) [[BR]] 8.1 MEASUREMENT DATA OBJECTS (MDOS) [[BR]] 8.2 MEASUREMENT DATA (MD) OBJECT DESCRIPTOR [[BR]] 8.3 MD OBJECT DESCRIPTOR DATA MODEL [[BR]] 8.4 MD OBJECT DESCRIPTOR SCHEMAS [[BR]] A summary of the topic is presented in these [http://groups.geni.net/geni/attachment/wiki/GEC10InstMeasWorkingSession/031711_MDObjectsDescriptors%20_I%26MArch_Overview.pptx slides][[BR]] DRAFT MDOD Data Models, Elements and Values [[BR]] v0.2.x XML-like example (HEM) [[BR]] [http://groups.geni.net/geni/attachment/wiki/GEC11InstMeasWorkingSession/072011_ver0.2_MDOD_DataModel.txt v0.2 text][[BR]] [http://groups.geni.net/geni/attachment/wiki/GEC11InstMeasWorkingSession/072511_ver0.2.1_MDOD_DataModel.txt v0.2.1 text][[BR]] [http://groups.geni.net/geni/attachment/wiki/GEC11InstMeasWorkingSession/072511_ver0.2.1_mandatoryonly_MDOD_DataModel.txt v0.2.1 text, only mandatory elements][[BR]] v0.3.x RelaxNG-Compact version (EK); includes RNC and converted XSD files, both schema and types [[BR]] [http://groups.geni.net/geni/attachment/wiki/GEC11InstMeasWorkingSession/072111_v0.3_MDOD.rnc v0.3 schema RNC] [[BR]] [http://groups.geni.net/geni/attachment/wiki/GEC11InstMeasWorkingSession/072111_v0.3_MDOD.xsd v0.3 schema converted XSD] [[BR]] [http://groups.geni.net/geni/attachment/wiki/GEC11InstMeasWorkingSession/072111_v0.3_MDOD_types.rnc v0.3 types RNC] [[BR]] [http://groups.geni.net/geni/attachment/wiki/GEC11InstMeasWorkingSession/072111_v0.3_MDOD_types.xsd v0.3 types converted XSD] [[BR]] XML examples provided by Jason Zurawski: [[BR]] [http://groups.geni.net/geni/attachment/wiki/GEC11InstMeasWorkingSession/072811Zurawski_instance1.xml example of proposed schema, with questions] [[BR]] [http://groups.geni.net/geni/attachment/wiki/GEC11InstMeasWorkingSession/072811Zurawski_instance_proposed.xml modified schema, for further discussion] [[BR]] MDOD Discussion Points [[BR]] For meeting on 072611 [http://groups.geni.net/geni/attachment/wiki/GEC11InstMeasWorkingSession/072511_MDOD_Discussion.txt text] [[BR]] ==== 2.4) Supported Object Types and Formats ==== Instrumentation Tools (Kentucky) [[BR]] OMF/OML (NICTA) [[BR]] perSONAR/LAMP (Delaware, I2) [[BR]] OnTimeMeasure (Ohio State) [[BR]] Scalable Sensing Service? (Purdue, HP Labs) [[BR]] GMOC? (Indiana)[[BR]] NetCDF?[[BR]] others? [[BR]] ==== 2.5 Next Steps ==== First review of data model and schema (completed) [[BR]] For each tool: map key elements into current data model; review expected use cases who? when? [[BR]] Review suggested approach to recording provenance who? when? [[BR]] Review choice for primary and secondary identifiers who? when? [[BR]] Review formats for slice_id, etc. who? when? [[BR]] Understand relationship to rspec? who? when? [[BR]] Complete third pass at data model and schema who? when?[[BR]] Complete first prototype software to create MDOD who? when?[[BR]] === 3) Current I&M Design and Prototyping Efforts === 1:35pm [[BR]] ==== 3.1) Instrumentation Tools (Kentucky) ==== 1:35pm [[BR]] Jim Griffioen, Hussam Nassir[[BR]] [http://groups.geni.net/geni/attachment/wiki/GEC11InstMeasWorkingSession/Instrumentation%20Tools.pdf slides] [[BR]] Summary and status [[BR]] Intended use(s): experimenter's slice operator's infrastructure measurement slice?[[BR]] Creates or includes info for MDOD? in druple DB? mapping to MDOD elements? object_type? object_format? interpretation_method?[[BR]] Includes persistent operational services? portal local archive [[BR]] Integrates with MDA service? how? stores tarball status? issues?[[BR]] Authorization methods supported? issues?[[BR]] Note from Jim on 7/6: As for other topics you may want to discuss, one of the things that continues to be a big headache for us is limiting access to measurement data. Currently we have to deal with several different users accounts, authentication schemes, object granularity, etc. BTW, we have integrated INSTOOLS with the CNRI workspace model, but again, they have yet another set of user accounts and passwords to deal with. If we could start moving toward some standards in this area, that would be very helpful. [[BR]] ==== 3.2) OMF/OML (NICTA) ==== 1:45pm [[BR]] Max Ott[[BR]] [http://groups.geni.net/geni/attachment/wiki/GEC11InstMeasWorkingSession/GEC11%20-%20OML%20update.pptx slides] [[BR]] Summary and status [[BR]] Intended use(s): experimenter's slice operator's infrastructure measurement slice?[[BR]] Creates or includes info for MDOD? mapping to MDOD elements? object_type? object_format? interpretation_method?[[BR]] Includes persistent operational services? centralized OML data service (will be needed for WiMAX base stations)[[BR]] Integrates with MDA service? how? status? issues?[[BR]] Authorization methods supported? issues?[[BR]] ==== 3.3) perSONAR/LAMP (Delaware, I2) ==== 1:55pm [[BR]] Jason Zurawski, Ezra Kissel, Martin Swany [[BR]] [http://groups.geni.net/geni/attachment/wiki/GEC11InstMeasWorkingSession/20110728-IM-LAMP.ppt slides] [[BR]] Summary and status [[BR]] Intended use(s): experimenter's slice? operator's infrastructure measurement slice?[[BR]] Creates or includes info for MDOD? mapping to MDOD elements? object_type? object_format? interpretation_method?[[BR]] Includes persistent operational services? lookup (meas info) services[[BR]] Integrates with MDA service? how? status? issues?[[BR]] Authorization methods supported? issues?[[BR]] ==== 3.4) OnTimeMeasure (Ohio State) ==== 2:05pm [[BR]] Prasad Calyam[[BR]] [http://groups.geni.net/geni/attachment/wiki/GEC11InstMeasWorkingSession/OnTimeMeasure_GEC11-I%26M-Talk.pdf slides] [[BR]] Summary and status [[BR]] Intended use(s): experimenter's slice? operator's infrastructure measurement slice?[[BR]] Creates or includes info for MDOD? mapping to MDOD elements? object_format? interpretation_method?[[BR]] Includes persistent operational services?[[BR]] Integrates with MDA service? how? status? issues?[[BR]] Authorization methods supported? issues?[[BR]] ==== 3.5) Meas Data Archive (MDA) Prototype Service ==== 2:15pm [[BR]] Giridhar Manepalli, Larry Lannom [[BR]] [http://groups.geni.net/geni/attachment/wiki/GEC11InstMeasWorkingSession/MDA_IntegrationEffort.pptx slides] [[BR]] Summary and status [[BR]] Intended use(s): experimenter's slice operator's infrastructure measurement slice [[BR]] Receives MDOD? mapping to MDOD elements? object_format? uses sharing_policy? uses disposal policy?[[BR]] Provides persistent operational services? user workspace service object archive service[[BR]] Integrates with other tools and services? how? status? issues?[[BR]] Authorization methods supported? issues?[[BR]] ==== 3.6) Others? ==== Scalable Sensing Service? (Purdue, HP Labs) [[BR]] GMOC? (Indiana)[[BR]] NetKarma: Experiment Provenance [[BR]] Mehmet Aktas, Beth Plale, Scott Jensen [[BR]] [http://groups.geni.net/geni/attachment/wiki/GEC11InstMeasWorkingSession/I%26M%20Meeting_discussion_ma.ppt slides] [[BR]] === 4) New Topics and Next Steps === 2:30pm [[BR]] Open discussion, led by Harry Mussman[[BR]] ==== 4.1) Measurement Data Object Descriptors (MDODs) (“metadata”) ==== First review of data model and schema (completed) [[BR]] For each tool: map key elements into current data model; review expected use cases who? when? [[BR]] Review suggested approach to recording provenance who? when? [[BR]] Review choice for primary and secondary identifiers who? when? [[BR]] Review formats for slice_id, etc. who? when? [[BR]] Understand relationship to rspec? who? when? [[BR]] Complete third pass at data model and schema who? when?[[BR]] Complete first prototype software to create MDOD who? when?[[BR]] ==== 4.3) Deploy Prototype Operational Services ==== Measurement Data Archive (MDA) service, including User Workspace (UW) service and Digital Object Archive (DOA) service (ongoing) [[BR]] INSTOOLS portal? who? when? [[BR]] INSTOOLS archive service? who? when? [[BR]] perfSONAR/LAMP lookup (Measurement Information) service? who? when?[[BR]] other MI service? who? when? [[BR]] Centralized OML server? (useful for monitoring WiMAX sites) who? when? [[BR]] others? [[BR]] ==== 4.4) Integrate I&M services for Experimenters ==== INSTOOLS tools with MDA service (ongoing) [[BR]] OnTimeMeas tools, with MDA service (ongoing) [[BR]] ==== 4.5) Integrate I&M services for Operators monitoring infrastructure ==== LAMP for OpenFlow slice monitoring (Ali Sydney) [[BR]] ==== 4.6) Use DOA for Sharing Data with Others ==== Who? when? [[BR]] ==== 4.7) Authorization Mechanisms ==== Define situation who? when? [[BR]] Note from Jim on 7/6: As for other topics you may want to discuss, one of the things that continues to be a big headache for us is limiting access to measurement data. Currently we have to deal with several different users accounts, authentication schemes, object granularity, etc. BTW, we have integrated INSTOOLS with the CNRI workspace model, but again, they have yet another set of user accounts and passwords to deal with. If we could start moving toward some standards in this area, that would be very helpful. [[BR]] Suggest approaches who? when? [[BR]] ==== 4.8) Extensions ==== Use MDA as a full Experiment Repository to archive all experiment objects, including: meas data; software; topology/rspec? [[BR]] Use various versions of MDOD to negotiate transfer of data object, e.g., like GENI resource assignment; use versions of MDOD to like rspec advertisement, request, amnifest. [[BR]] Use MDOD with descriptor and holder/transaction records for "object assignment record"; use similar structure in GENI resource assignment to give "resource usage record" [[BR]] === Adjourn === 3:00pm [[BR]] == Background Reading == [http://groups.geni.net/geni/wiki/GeniInstMeas I&M integration wiki][[BR]] [http://groups.geni.net/geni/wiki/GENIIandMCAPCAT GENI I&M Capabilities Catalog][[BR]] [http://groups.geni.net/geni/wiki/GeniInstrumentationandMeasurementsArchitecture GENI I&M Architecture document][[BR]]