[[PageOutline]] == Project Number == 1706 == Project Title == netKarma: GENI Provenance Registry[[BR]] a.k.a. NETKARMA === Technical Contacts === '''PI:''' Beth Plale [mailto:plale@cs.indiana.edu][[BR]] Chris Small [mailto:chsmall@cs.indiana.edu][[BR]] === Participating Organizations === [http://www.informatics.indiana.edu/ School of Informatics and Computing][[BR]] Indiana University Bloomington[[BR]] === GPO Liaison System Engineer === [mailto:vthomas@geni.net Vic Thomas] == Scope == The project collect and represents provenance for experiments conducted on the GENI platform. The provenance of an experiment is relevant information collected at experiment plane, control plane, and measurement plane. The provenance results will be available in a GENI Provenance Registry (netKarma) but can also be used to augment other collection mechanisms, for instance at the instrument level. [http://pti.iu.edu/d2i/provenance_netKarma NetKarma] is based on [http://pti.iu.edu/d2i/provenance Karma], a provenance collection and representation service that has been used to collect provenance in diverse applications including satellite imagery pipeline (NASA funded), Linked Environments for Atmospheric Discovery (LEAD, NSF funded), and the Life Science Grid (Lilly Corp. funded). === Current Capabilities === The NetKarma Provenance Repository is an Axis2 Web service that resides on a server in the GENI Meta-Operations Center (GMOC), located at Indiana University. It has a WSDL access API so provenance can be retrieved programmatically. The persistent Axis2 web service is available at http://netkarma.testlab.grnoc.iu.edu:8080/axis2/services/KarmaService. The persistent Web Service provides both publish and query API to interact with the provenance repository to capture and browse provenance of GENI experiments. The current capabilities include the ingestion of provenance information from the Gush experiment control tool. The NetKarma project produced the GENI Adaptor software, which parses Gush log files to obtain provenance information. Further information about the GENI Adaptor software is available at http://pti.iu.edu/d2i/provenance_netkarma. With such capabilities, the persistent Web Service will enable GENI experimenters to capture data about experiments including: time ordering and relationships within the experiment, changes made between runs, and relationships between the experiment and control framework. The current capabilities also include the ability to query the persistent Karma Service for provenance of GENI experiment executions. In turn, this will enable GENI experimenters show such provenance information in visualization such as Google Earth View. We Need Your Help! :: Give us your GUSH logs. We'll mine them for provenance data and drop the provenance into the netKarma provenance repository. You'll get a message back with information that you can use to query for the graph of your GUSH run. Submit through (http://pti.iu.edu/d2i/provenance/submit-gush-log). We're working on customizations to the cytoScape tool that you can download to visualize your provenance. Planned capacities: The project team is currently working to process logs from the Raven provisioning service to obtain provenance information. This new capability will enable GENI experimenters capture provenance information such as experiment node locations, time of deployment of software packages and versions of software deployed. The planned capabilities also include actively gathering information from the GMOC database. This will help the GENI experimenters relate the provenance on-the-fly to network measurements. All this information will be accessible through the persistent axis2 Web service layer to the community. Because provenance collection captures details about the experiment, it raises issues of privacy that we address by giving the experimenter control of when collection occurs. Another planned capability is the visualization of the provenance graph generated from Gush experiment log files using the CytoScape visualization tool (http://www.cytoscape.org). === Milestones === * [[MilestoneDate(NETKARMA: S4.a Provence system evaluation; I&M contributions)]] * [[MilestoneDate(NETKARMA: S4.b Provenance system demo; I&M contributions)]] * [[MilestoneDate(NETKARMA: S4.c Provenance system demo; I&M contributions)]] * [[MilestoneDate(NETKARMA: S4.d Software and documentation delivery; I&M contributions)]] == Project Technical Documents == [http://www.pti.iu.edu/d2i/provenance_netkarma NetKarma project page] maintained by the [http://www.pti.iu.edu/d2i Data to Insight Center] [attachment:"MDOD Provenance and Extensibility.pdf" MDOD Provenance and Extensibility] [[BR]] [attachment:"GENI - Provenance System Evaluation.pdf" Plan for Evaluating netKarma Provenance System] [[BR]] [attachment:"NetKarma_GEC7 deliverable Plale-Small.pdf" netKarma: a tool for obtaining a provenance-based record of experimentation] (overview document)[[BR]] [attachment:"NETKARMA-S2.f-experience-report.pdf" Experience Report][[BR]] [attachment:"NETKARMA-S2.g-UserDoc.pdf" User Guide][[BR]] [attachment:"NETKARMA-S2.h-Prototype.pdf" Description of the netKarma Prototype][[BR]] === Software === '''NetKarma GUSH Adaptor Version 2.5.1 (released April 5, 2012)''' Version 2.5.1 of the NetKarma GUSH Adaptor can be used to process GUSH logs from GENI experiments by extracting and uploading provenance to a Karma server installed at Indiana University. The resulting provenance can be visualized using version 1.2.1 of the NetKarma Cytoscape visualization plug-in. * [http://d2i.indiana.edu/sites/default/files/karma-adaptor-2.5.1.tar.gz NetKarma GUSH Adaptor version 2.5.1] * [http://d2i.indiana.edu/sites/default/files/netkarma_version2.5.1_quick_start_guide.pdf Quick Start Guide for version 2.5.1] '''NetKarma NS2 Adaptor Version 1.0.0 (released March 21, 2012)''' The Cytyoscape visualization plug-in version 1.2.1 includes new visualzation algorithms for the capture of provenance from experiments run on the NS2 network simulator. The NetKarma NS2 Adaptor provides the ability to capture provenance from NS2 trace files and has a Tcl extension to capture the network topology of your NS2 experiment. The adaptor allows you to ingest provenance for NS2 experiments from these files and then visualize it from NetKarma using version 1.2.1 or later of the NetKarma visualization plug-ins. Following is the download for the adaptor, the manual for the adaptor, and instructions on configuring your NS2 experiment to capture and process provenance. * [http://d2i.indiana.edu/sites/default/files/ns2adaptor.tar NetKarma NS2 Adaptor Version 1.0.0] * [http://d2i.indiana.edu/sites/default/files/netkarma_ns2_adaptor_user_manual_v1.0.0.pdf NetKarma NS2 Adaptor User Manual] * [http://d2i.indiana.edu/sites/default/files/instructions_for_the_netkarma_provenance_toolkit_for_ns2_experiments.pdf Instructions for the NetKarma Provenance Toolkit for NS2 Experiments] '''NetKarma Cytoscape Visualization Plug-In Version 1.2.1 (released March 6, 2012)''' Version 2.5 of NetKarma included version 1.1.0 of the NetKarma visualization plug-in for Cytoscape. This version of the plug-in includes upgrades to the visualization layout algorithms and new visualization styles. These were added as part of our efforts to capture provenance for additional GENI experiments using the NS2 simulator. The visualization plug-in manual includes further details, but the features added in this release of the plug-in include: * New layout algorithms that handle provenance graphs generated based on Network Simulator (NS2) experiments. * New visualization style for NS2 simulations. Downloads for NetKarma Visualization Plug-in v1.2.1 * [http://d2i.indiana.edu/sites/default/files/netkarma_visualization_plugin-1.2.1.tar.gz NetKarma Cytoscape Visualization Plug-In Version 1.2.1] * [http://d2i.indiana.edu/sites/default/files/karmavisualizationusermanual-1.2.0.pdf NetKarma Cytoscape Visualization Plug-In Manual for Version 1.2.0] '''Version 2.5 (released August 26, 2011)''' NetKarma version 2.5 builds on the NetKarma Adaptor and NetKarma visualization plug-ins for Cytoscape to provide more detailed provenance data from the GUSH logs of experiments run on GENI slices. In addition to extracting more detailed provenance from GUSH logs, version 2.5 of the NetKarma plug-in includes richer annotations of the provenance visualized and is more configurable by the user - allowing greater flexibility in visualizing and comprehending the provenance of an experiment. Version 2.5 also includes the NetKarma GMOC Adaptor which automates the annotation of NetKarma provenance graphs with network data from GMOC. Features Added in Version 2.5: * Enhancements to the NetKarma Adaptor used to process GUSH logs and ingest the provenance of GENI experiments run on GUSH into the NetKarma Provenance System. * Enhancements to the NetKarma Cytoscape plug-in for NetKarma that includes more detailed provenance and the addition of annotations to processes, artifacts, and the edges that connect the processes and artifacts in the OPM graphs generated using NetKarma. This version of the visualization plug-in also has increased configurability by the user as to the details displayed when visualizing provenance graphs. * The new NetKarma GMOC Adaptor is being released which extracts additional metadata from the GMOC database and annotates the GUSH provenance graphs with data from the GMOC database. This Adaptor is configured to run as a background process and automatically annotates the provenance ingested into NetKarma without any additional steps required by GENI users. The NetKarma Provenance System and the NetKarma GMOC Adaptor are running as a persistent service on servers at Indiana University. This allows GENI users to process a GUSH log through the NetKarma Adaptor and visualize the resulting provenance graph, including the automated GMOC annotations, without having to install the NetKarma server. To try out the NetKarma provenance visualization, GENI users can process their own GUSH logs or the sample log file included with the NetKarma adaptor. Downloads for v2.5 * [http://pti.iu.edu/sites/default/files/netkarma_version2.5_quick_start_guide.pdf NetKarma Version 2.5 Quick Start Guide] * [http://pti.iu.edu/sites/default/files/karma-adaptor-2.5.tar.gz NetKarma GUSH Adaptor Version 2.5] * [http://pti.iu.edu/sites/default/files/netkarma-gush-adaptor_v2.5_user_manual.pdf NetKarma GUSH Adaptor Manual] * [http://pti.iu.edu/sites/default/files/netkarmavisplugin-1.1.0.tar.gz NetKarma Cytoscape Visualization Plug-In Version 1.1.0] * [http://pti.iu.edu/sites/default/files/netkarma_provenance_retrieval_and_visualization_plug-in_for_cytoscape_user_manual_v1.1.0.pdf NetKarma Cytoscape Visualization Plug-In Manual] * [http://pti.iu.edu/sites/default/files/netkarmagmocadaptor-1.0.0.tar.gz NetKarma GMOC Adaptor Version 1.0.0] * [http://pti.iu.edu/sites/default/files/netkarma_gmoc_adaptor_user_manual_v1.0.0.pdf NetKarma GMOC Adaptor Manual] Version 2.0 Karma Adaptor is one of the collection tools that make up the Karma provenance collection toolkit to harvest provenance from log files. It uses a rule file specific to an application to map raw data into Karma specific provenance events. The provenance of the data is stored into a relational database which can be visualized through various plugins. The provenance data can be used by the researcher to analyze their data, allow for the suspension and resumption of an experiment and provide references to find the details and data collected in an experiment. Downloads for v2.0 * [https://pti.iu.edu/sites/default/files/karma-adaptor-2.0.tar.gz Karma Adaptor] * [https://pti.iu.edu/sites/default/files/karma-adaptor_v2.0_user_manual.pdf Karma Adaptor User Manual] Code to allow capture of provenance from GUSH [http://pti.iu.edu/d2i/provenance_netkarma/gush2netKarma-1.0.tar.gz gush2netkarma-1.0.tar.zip]. This code does not store events to the Karma system; it simply writes the provenance to a file that is then used to create a visual graph. === Presentations and posters === [http://groups.geni.net/geni/attachment/wiki/netKarma/GEC14_report/Poster-GENI%20Provenance_Instrumentation_and_Measurement.pdf GEC14 Poster - Integrating Provenance into the GENI Measurement Data Object Descriptor][[br]] [http://groups.geni.net/geni/attachment/wiki/netKarma/GEC14_report/Poster-NetKarma_Reveals_the_Provenance_of_WiMAX_Experiments_on_ORBIT.pdf GEC14 Poster - NetKarma Reveals the Provenance of WiMAX Experiments on ORBIT][[br]] [attachment:"GEC 13-Role of Provenance in Visualizing Packet Throughput and Packet Loss.pdf" GEC 13 Poster - Role of Provenance in Visualizing Packet Throughput and Packet Loss][[br]] [attachment:"GEC13-NetKarma_Portal.pdf" GEC 13 Poster - NetKarma Portal] [[br]] [attachment:"GEC13-Provenance_Handout.pdf" Handout at GEC 13 on Provenance for Experimenters] [[br]] [attachment:"karma_poster_gec-11-lettersize.pdf" Experiment Provenance: Towards Links to Network Measurement Data Poster (GEC11)] [[br]] [attachment:"GEC-10 Demo Handout.pdf" GEC10 Demonstration Slides][[br]] [attachment:"GEC9_Poster_final.pdf" GEC9 poster][[br]] [attachment:"GEC8_IU_NetKarma Poster8x11.pdf" GEC8 poster][[br]] [attachment:netKarma-update-20100720.pdf" GEC8 update][[br]] [attachment:"NetKarma_PL_cluster_GEC7_slides.pdf" netkarma Presentation given at GEC7 Planetlab cluster meeting] [[BR]] [attachment:"Netkarma_poster_gec7.pdf" GEC7 poster][[BR]] [attachment:"NetKarma_Poster.pdf" GEC6 poster][[BR]] === Quarterly Status Reports === [wiki:netKarma-4Q09-status 4Q09 Status Report][[BR]] [wiki:netKarma-1Q10-status 1Q10 Status Report][[br]] [wiki:netKarma-2Q10-status 2Q10 Status Report][[br]] [wiki:netKarma-GEC9 Post GEC9 Status Report][[BR]] [attachment:NetKarmaReportGEC9-GEC10-bp.2.pdf Post GEC10 Status Report][[BR]] [attachment:NetKarmaReportGEC10-GEC11.pdf Post GEC 11 Status Report][[BR]] [attachment:NetKarmaReportGEC11-GEC12.pdf Post GEC12 Status Report][[BR]] [wiki:netKarma/GEC13_report Post GEC13 Status Report][[BR]] [wiki:netKarma/GEC14_report Post GEC14 Status Report][[BR]] === Spiral 2 Connectivity === This project will use existing Indiana University address and connections. No addition connectivity is needed as initial data sources are globably available on the Internet or over R&E networks. === Related Projects === Initial integration GushProto [[BR]] ProvisioningService (Raven) [[BR]] Possible future integration [wiki:PGTools PGTools] [[BR]] DigitalObjectRegistry [[BR]] [wiki:GENIMetaOps GMOC] [[BR]] OnTimeMeasure [[BR]]