Version 11 (modified by 12 years ago) (diff) | ,
---|
NetKarma
Report for the period ending GEC13
Beth Plale, School of Informatics and Computing, Indiana University
Chris Small, InCNTRE, Indiana University
Summary
Milestones Delivered
S4.c Provenance System Demo; I&M Contributions
NetKarma Tutorial and Demo
At GEC14 we presented a tutorial of the NetKarma Portal which is layered on top of the NetKarma server as well as NetKarma plug-ins for visualizing large provenance graphs. After first providing participants with an overview of what provenance is and how it can be beneficial to GENI experimenters, the tutorial continued with three hands-on components that allowed GENI experimenters to capture provenance, ingest and visualize provenance in the NetKarma Portal, and download provenance graphs from the portal for visualization in Cytoscape using the GENI NetKarma plug-ins.
The tutorial first walked the participants through an experiment using Emulab ProtoGENI resources to run an experiment doing a GridFTP transfer based on the eXtensible Session Protocol (XSP). Since the focus was on provenance generation, the resources had already been requested and configured, and the participants ran scripts loaded in the tutorial VM on VirtualBox to run the experiment – generating a log from the XSP experiment, some basic bandwidth measurements using iperf, and downloading the manifest for the resources they are using. The NetKarma adaptor which is incorporated into the NetKarma portal used these two log files and the manifest in the second hands-on portion of the tutorial to extract provenance and load it into NetKarma through the Portal.
<Portal section>
The third hands-on portion of the tutorial showed participants how they can easily download a provenance graph generated by NetKarma from the Portal. The downloaded graph is an XML file based on the Open Provenance Model (OPM) standard for provenance. For very large provenance graphs, the NetKarma plug-in can be used to visualize and explore large provenance graphs using the desktop version of Cytoscape (the web version of cytoscape is used in the Portal). In this section participants downloaded provenance graphs for experiments we had run on both NS2 and ORBIT and loaded into the portal using the corresponding NetKarma adaptors. The provenance graphs for these experiments differ from the XSP experiment in that whereas the adaptor for the XSP experiment uses the manifest to incorporate the topology of the experiment into the provenance graph, the focus of the NS2 and ORBIT adaptors is on capturing the success of each packet transferred. This focus on the packet level results in large provenance graphs, but as participants were able to experience by loading the graphs for both experiments into the copy of Cytoscape included in the tutorial VM, they could quickly see the performance of NS2 and ORBIT experiments as to packets successfully transferred or dropped by using visualization capabilities included in the NetKarma plug-in.
The full instructions, including step-by-step graphics and illustrations of the provenance generation, Portal, and visualization process are included in the tutorial instructions available at: http://d2i.indiana.edu/wiki/NetKarma_Tutorial
During the Demo session at GEC14 we also presented a live demo that followed up on our tutorial to show the latest version of the NetKarma portal, the visualization of an ORBIT experiment using the latest version of the NetKarma visualization plug-in for Cytoscape, and the propsed MDOD schema with NetKarma provenance.
NetKarma and the Measurement Data Object Descriptor (MDOD)
Following up on our proposed changes to the MDOD at GEC13 and subsequent discussions, we presented an XML schema of our proposed MDOD that was shared with the Measurement & Instrumentation group to stimulate discussion and we presented a poster at the GEC14 Demo session along with the schema and an example based on the XSP experiment used in the tutorial. The schema is discussed further in the sectin below on work performed this quarter. At GEC14, we also participated in the follow-on discussion Wednesday afternoon regarding next steps for the MDOD which was hosted by Harry Mussman and Jeanne Ohren of BBN. The proposed MDOD schema, supporting schemas, and the example based on the XSP experiment are included below in the documents section.
Work Performed this Quarter
NetKarma Portal
<add Portal Details>
NetKarma Visualization and Adaptor Enhancements
Capturing the Provenance of an XSP Experiment
Proposed MDOD Schema Integrating Provenance
Project Participants
During this period, active participants in the NetKarma project included: Beth Plale and Chris Small, as well as Scott Jensen, Postdoctoral Fellow in the Data To Insight Center, and students Peng Chen, Devarshi Ghoshal, and Yuan Luo. Robert Ping provided project management for the project.
Collaborations
- Katherine Cameron (Clemson University) – ORBIT experiments on DDoS attacks and ORBIT configuration
- Nilanjan Paul (Rutgers) – ORBIT experiment configuration
- Fraida Fund (NYU-Poly WITest Lab) ORBIT reservation system
- Martin Swany (Indiana University) and Ezra Kissel (University of Delaware) – Discussion of measurement metadata capture in the MDOD. XSP-based experiment used in NetKarma tutorial
- Harry Mussman (BBN) – Metadata capture for measurement data in the MDOD
- Jeanne Ohren (BBN) – tutorial coordination and MDOD
- Ahmet Babaoglu (North Carolina State University) capturing NetKarma notifications as events in IMF Messaging
Publications & Documents
Posters at the GEC14 Poster and Demo Session:
- GENI Provenance, Instrumentation and Measurement: Integrating Provenance into the GENI Measurement Data Object Descriptor PDF
Step-by-Step Instructions for the NetKarma Tutorial at GEC14: http://d2i.indiana.edu/wiki/NetKarma_Tutorial
Slides from Presentation on NetKarma at the GEC14 Experimenter's Roundtable: slides
Proposed MDOD Schema for capturing the Metadata and Provenance of GENI Experiments: MDOD_Provenance_0.2.xsd
Supporting schemas imported into the proposed MDOD schema:
- Third-party DOI schema imported into the MDOD schema for the identifier: DOIMetadataKernel_v2.0_120308.xsd
- Third-party ISO schema required by the DOI schema: iso3166a2.xsd
- OPM schema imported into the MDOD for provenance: opm.1_1.xsd
- Karma extensions to the OPM schema: opm.1_1.ext.xsd
Sample MDOD example based on capturing the metadata and provenance for an XSP experiment as used in the NetKarma tutorial: SampleMDOD-XSP.xml
GENI Documents
none
Attachments (22)
-
Poster-GENI Provenance_Instrumentation_and_Measurement.pdf (147.9 KB) - added by 12 years ago.
Poster at GEC14
-
Poster-NetKarma_Reveals_the_Provenance_of_WiMAX_Experiments_on_ORBIT.pdf (275.6 KB) - added by 12 years ago.
Poster at GEC14
-
MDOD_Provenance_0.2.xsd (18.8 KB) - added by 12 years ago.
Proposed MDOD Schema with Provenance
-
DOIMetadataKernel_v2.0_120308.xsd (50.1 KB) - added by 12 years ago.
DOI Schema required by the MDOD schema
-
iso3166a2.xsd (36.7 KB) - added by 12 years ago.
schema required by the DOI schema that's imported into the MDOD schema
-
opm.1_1.xsd (14.0 KB) - added by 12 years ago.
extended OPM schema required by the MDOD schema
-
opm.1_1.ext.xsd (2.3 KB) - added by 12 years ago.
NetKarma extensions to the OPM schema required for the MDOD schema
-
SampleMDOD-XSP.xml (7.4 KB) - added by 12 years ago.
Sample MDOD describing an XSP experiment
- ORBIT-Provenance-Generation.png (227.8 KB) - added by 12 years ago.
- Orbit-provenanance-zoomed-in.png (462.1 KB) - added by 12 years ago.
-
ORBIT_WiMAX.xml (3.3 MB) - added by 12 years ago.
ORBIT Experiment OPM Provenance Graph Generated using NetKarma
- XSPDiagram.png (39.6 KB) - added by 12 years ago.
- MDOD-LifeCycle.png (213.3 KB) - added by 12 years ago.
- MDOD-Example-Structure.png (166.1 KB) - added by 12 years ago.
-
NetKarma_Portal_diagram.png (22.4 KB) - added by 12 years ago.
Diagram
-
NetKarma_Portal_home.png (120.3 KB) - added by 12 years ago.
Portal home page
-
NetKarma_Portal_exp.png (118.5 KB) - added by 12 years ago.
Portal Experiment Page
-
NetKarma_Portal_workflow.png (93.4 KB) - added by 12 years ago.
Portal WorkFlow
-
NetKarma_Portal_workflow.2.png (93.4 KB) - added by 12 years ago.
Portal Workflow
-
NetKarma_Portal_archive.png (57.2 KB) - added by 12 years ago.
Portal Archive
-
NetKarma_Portal_topo.png (190.8 KB) - added by 12 years ago.
Portal Topology Page
- Slides-NetKarma-GEC14_Experimenters_RoundTable.pdf (479.8 KB) - added by 12 years ago.