Changes between Initial Version and Version 1 of netKarma-GEC9


Ignore:
Timestamp:
01/27/11 12:17:17 (9 years ago)
Author:
Beth Plale
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • netKarma-GEC9

    v1 v1  
     1NetKarma Report 07/01/2010-11/05/2010 (GEC9)
     2
     3PI:  Beth Plale, School of Informatics and Computing, Indiana University Bloomington
     4Co-PI:  Chris Small, Global NOC, Indiana University
     5
     6Summary 
     7
     8•       Documents:  Experience Report, User Doc, Prototype
     9•       Karma 3.1.1 released
     10
     11Major Accomplishments
     12
     13Now hosting persistent netKarma service. Accepting GUSH log files from community.  Please see http://pti.iu.edu/d2i/provenance_netkarma
     14
     15Milestones Achieved
     16
     17S2.f Experience Report for NetKarma Project: This report summarizes our experiences in provenance capture in GENI.   It is available from the netKarma GENI Wiki page as NETKARMA-S2.f-experience-report.pdf
     18
     19S2.g User Doc: The user guide describes the NetKarma Provenance Collection tool and give assistance to developers using the persistent Axis2 Web Service on how one uses the XML API to interact with the NetKarma repository. The user guide is available here: NETKARMA-S2.g-UserDoc.pdf
     20
     21S2.h Prototype: During GEC9, the netKarma team demonstrated live the execution of a MapReduce graph coloring algorithm running on in a PlanetLab slice utilizing a set of nodes.  Provenance from GUSH is captured, stored to the netKarma repository, and retrieved through the XML query API.  A description can be found at: NETKARMA-S2.h-Prototype.pdf
     22
     23Description of Work Performed Since Last Report
     24
     251.      Karma 3.1.1 Released: The new release of the core Karma system utilizes RabbitMQ enterprise messaging system for events ingest. The provenance data is efficiently stored in a relational database, and supports Open Provenance Model (OPM) v1.1 standard for interfacing with the tool.  Karma is available at http://pti.iu.edu/d2i/provenance_karma
     26
     272.      Engaged with Instrumentation and Measurement Working Group.  Talked at length with Larry Lannom about provenance, its representation, and how it can fit into the repository scheme has and his team envision. 
     28
     293.      Worked with Luisa Nevers of BBN to run netKarma on a GUSH script she generated.  After much back and forth, it was determined that her mode of running GUSH (through the command line versus using the XML file) generated a different log file and this was causing our netKarma Adaptor which parsed the log file to have problems.  Resolved over Sep timeframe.
     30
     314.      Demo for GEC9:. Successful live demonstration of a MapReduce graph coloring algorithm running on in a PlanetLab slice utilizing a set of nodes.  Provenance from GUSH is captured, stored to the netKarma repository, and retrieved through the XML query API.  Examined other application but discarded.  Thought to use GUSH to install Codeen and Coblitz, as these are good parallel applications that run on PlanetLab, but determined that these come preinstalled on PlanetLab so do not exercise GUSH as we need.
     32
     335.      Rewrite code for netKarma Adaptor so it no longer creates code on the fly.  This makes the code easier to maintain and build.
     34
     356.      Installed version of Karma on the GMOC machine.  This is the persistent netKarma service. 
     36
     377.      Successful build of GUSH from the source code. 
     38
     39
     40Project Participants
     41During this time, key participants in the NetKarma project included:
     42 
     43Beth Plale, PI
     44Chris Small, Co-PI
     45Mehmet Aktas, Postdoctoral Fellow
     46Devarshi Goshal, PhD student
     47Peng Chen, PhD student
     48You-Wei Cheah, PhD student
     49David Ripley, Technical Staff
     50Robert Ping, Project and Information Management
     51
     52Publications & Documents
     53GEC 9 poster GEC9_Poster_final.pdf
     54
     55NetKarma Status Update during GEC8: http://groups.geni.net/geni/attachment/wiki/netKarma/netKarma-update-20100720.pdf
     56
     57NetKarma Poster used at the GEC8 demo session:
     58http://groups.geni.net/geni/attachment/wiki/netKarma/GEC8_IU_NetKarma%20Poster8x11.pdf
     59
     60Spiral 2 Annual Review Slides:
     61http://groups.geni.net/geni/attachment/wiki/netKarma/Spiral2ProjectReview_Netkarma-27Aug2010-2.pptx
     62
     63NetKarma Provenance Repository Research Poster presented at GEC9:
     64http://groups.geni.net/geni/attachment/wiki/netKarma/GEC9_Poster_final.pdf
     65
     66Collaborations
     67
     68Engaged with Instrumentation and Measurement Working Group.  Talked at length with Larry Lannom about provenance, its representation, and how it can fit into the repository scheme has and his team envision. 
     69From the GRNOC the main information on provenance we will get is normalized data from each of the control clusters. The GMOC has already done much of the work of rectifying divergent data sets and placing it the GMOC database. The GMOC is also collecting status and topologies of the substrate and increasingly more views into the slice level of individual experiments.
     70Planned Activities
     71
     72During the Annual review and subsequent communication with GENI program manager Vic Thomas, new milestones were created for the coming year to include:
     73
     74Milestone a. GEC9 demonstration and outreach. Due 11/5/2010
     75•       NetKarma demonstration
     76o       provenance collected from runs of GUSH and retrieved from a persistent netKarma server deployed at the GNOC at Indiana University
     77o       Obtain feedback from experimenters on the kinds of provenance information that will be useful
     78o       Identify a new source of provenance information : Global Research Network Operations Center
     79
     80Milestone b. Plan for making provenance information available to experimenters. Due 1/7/2011
     81•       Document or wiki page with plan for how provenance information will be provided to experimenters.
     82•       Plan for how provenance source from Milestone a will be used
     83
     84Milestone c. GEC10 demonstration and outreach. Due 3/5/2011
     85•       Demonstration of a GENI experiment and display of provenance information for data collected by experiment. Demonstration should include at least one new source of provenance information : GNOC
     86•       Get feedback from experimenters on the kinds of provenance information that will be useful
     87•       Identify at least one other source of provenance information
     88
     89Milestone d. Updated plan for making provenance information available to experimenters.  Due 4/15/2011
     90•       Document or wiki page updates on how provenance information is provided to experimenters.
     91•       Description of how additional source of provenance information identified in Milestone c will be used
     92
     93Milestone e. GEC11 demonstration and outreach. Due July 2011
     94•       Demonstration of an GENI experiment and display of provenance information for data collected by experiment.  Demonstration should include at least one new source of provenance information
     95•       Get feedback from experimenters on the kinds of provenance information that will be useful
     96•       Identify at least one other source of provenance information
     97
     98Milestone f. Deliver software and documentation. Due 8/26/2011
     99•       Documentation for experimenters on how to collect and use provenance information
     100•       NetKarma software and documentation
     101•       Description of how additional source of provenance information identified in Milestone e will be used.