= NetKarma = '''Report for the period ending GEC13''' Beth Plale, School of Informatics and Computing, Indiana University[[br]] Chris Small, InCNTRE, Indiana University == Summary == == Milestones Delivered == __S4.c Provenance System Demo; I&M Contributions__ '''NetKarma Tutorial and Demo''' At GEC14 we presented a tutorial of the NetKarma Portal which is layered on top of the NetKarma server as well as NetKarma plug-ins for visualizing large provenance graphs. After first providing participants with an overview of what provenance is and how it can be beneficial to GENI experimenters, the tutorial continued with three hands-on components that allowed GENI experimenters to capture provenance, ingest and visualize provenance in the NetKarma Portal, and download provenance graphs from the portal for visualization in Cytoscape using the GENI NetKarma plug-ins. The tutorial first walked the participants through an experiment using Emulab ProtoGENI resources to run an experiment doing a GridFTP transfer based on the eXtensible Session Protocol (XSP). Since the focus was on provenance generation, the resources had already been requested and configured, and the participants ran scripts loaded in the tutorial VM on VirtualBox to run the experiment – generating a log from the XSP experiment, some basic bandwidth measurements using iperf, and downloading the manifest for the resources they are using. The NetKarma adaptor which is incorporated into the NetKarma portal used these two log files and the manifest in the second hands-on portion of the tutorial to extract provenance and load it into NetKarma through the Portal. The third hands-on portion of the tutorial showed participants how they can easily download a provenance graph generated by NetKarma from the Portal. The downloaded graph is an XML file based on the Open Provenance Model (OPM) standard for provenance. For very large provenance graphs, the NetKarma plug-in can be used to visualize and explore large provenance graphs using the desktop version of Cytoscape (the web version of cytoscape is used in the Portal). In this section participants downloaded provenance graphs for experiments we had run on both NS2 and ORBIT and loaded into the portal using the corresponding NetKarma adaptors. The provenance graphs for these experiments differ from the XSP experiment in that whereas the adaptor for the XSP experiment uses the manifest to incorporate the topology of the experiment into the provenance graph, the focus of the NS2 and ORBIT adaptors is on capturing the success of each packet transferred. This focus on the packet level results in large provenance graphs, but as participants were able to experience by loading the graphs for both experiments into the copy of Cytoscape included in the tutorial VM, they could quickly see the performance of NS2 and ORBIT experiments as to packets successfully transferred or dropped by using visualization capabilities included in the NetKarma plug-in. The full instructions, including step-by-step graphics and illustrations of the provenance generation, Portal, and visualization process are included in the tutorial instructions available at: http://d2i.indiana.edu/wiki/NetKarma_Tutorial During the Demo session at GEC14 we also presented a live demo that followed up on our tutorial to show the latest version of the NetKarma portal, the visualization of an ORBIT experiment using the latest version of the NetKarma visualization plug-in for Cytoscape, and the propsed MDOD schema with NetKarma provenance. '''NetKarma and the Measurement Data Object Descriptor (MDOD)''' Following up on our proposed changes to the MDOD at GEC13 and subsequent discussions, we presented an XML schema of our proposed MDOD that was shared with the Measurement & Instrumentation group to stimulate discussion and we presented a poster at the GEC14 Demo session along with the schema and an example based on the XSP experiment used in the tutorial. The schema is discussed further in the sectin below on work performed this quarter. At GEC14, we also participated in the follow-on discussion Wednesday afternoon regarding next steps for the MDOD which was hosted by Harry Mussman and Jeanne Ohren of BBN. The proposed MDOD schema, supporting schemas, and the example based on the XSP experiment are included below in the documents section. == Work Performed this Quarter == '''NetKarma Portal''' '''NetKarma Visualization and Adaptor Enhancements''' '''Capturing the Provenance of an XSP Experiment''' '''Proposed MDOD Schema Integrating Provenance''' == Project Participants == During this period, active participants in the NetKarma project included: Beth Plale and Chris Small, as well as Scott Jensen, Postdoctoral Fellow in the Data To Insight Center, and students Peng Chen, Devarshi Ghoshal, and Yuan Luo. Robert Ping provided project management for the project. == Collaborations == * Katherine Cameron (Clemson University) – ORBIT experiments on DDoS attacks and ORBIT configuration * Nilanjan Paul (Rutgers) – ORBIT experiment configuration * Fraida Fund (NYU-Poly WITest Lab) ORBIT reservation system * Martin Swany (Indiana University) and Ezra Kissel (University of Delaware) – Discussion of measurement metadata capture in the MDOD. XSP-based experiment used in NetKarma tutorial * Harry Mussman (BBN) – Metadata capture for measurement data in the MDOD * Jeanne Ohren (BBN) – tutorial coordination and MDOD * Ahmet Babaoglu (North Carolina State University) capturing NetKarma notifications as events in IMF Messaging == Publications & Documents == Posters at the GEC14 Poster and Demo Session: [[BR]] * NetKarma Reveals the Provenance of WiMAX Experiments on ORBIT [http://groups.geni.net/geni/attachment/wiki/netKarma/GEC14_report/Poster-NetKarma_Reveals_the_Provenance_of_WiMAX_Experiments_on_ORBIT.pdf PDF] * GENI Provenance, Instrumentation and Measurement: Integrating Provenance into the GENI Measurement Data Object Descriptor [http://groups.geni.net/geni/attachment/wiki/netKarma/GEC14_report/Poster-GENI%20Provenance_Instrumentation_and_Measurement.pdf PDF] Step-by-Step Instructions for the NetKarma Tutorial at GEC14: http://d2i.indiana.edu/wiki/NetKarma_Tutorial Slides from Presentation on NetKarma at the GEC14 Experimenter's Roundtable: [http://groups.geni.net/geni/attachment/wiki/netKarma/GEC14_report/Slides-NetKarma-GEC14_Experimenters_RoundTable.pdf slides] Proposed MDOD Schema for capturing the Metadata and Provenance of GENI Experiments: [http://groups.geni.net/geni/attachment/wiki/netKarma/GEC14_report/MDOD_Provenance_0.2.xsd MDOD_Provenance_0.2.xsd] Supporting schemas imported into the proposed MDOD schema: * Third-party DOI schema imported into the MDOD schema for the identifier: [http://groups.geni.net/geni/attachment/wiki/netKarma/GEC14_report/DOIMetadataKernel_v2.0_120308.xsd DOIMetadataKernel_v2.0_120308.xsd] * Third-party ISO schema required by the DOI schema: [http://groups.geni.net/geni/attachment/wiki/netKarma/GEC14_report/iso3166a2.xsd iso3166a2.xsd] * OPM schema imported into the MDOD for provenance: [http://groups.geni.net/geni/attachment/wiki/netKarma/GEC14_report/opm.1_1.xsd opm.1_1.xsd] * Karma extensions to the OPM schema: [http://groups.geni.net/geni/attachment/wiki/netKarma/GEC14_report/opm.1_1.ext.xsd opm.1_1.ext.xsd] Sample MDOD example based on capturing the metadata and provenance for an XSP experiment as used in the NetKarma tutorial: [http://groups.geni.net/geni/attachment/wiki/netKarma/GEC14_report/SampleMDOD-XSP.xml SampleMDOD-XSP.xml] == GENI Documents == none