Changes between Version 1 and Version 2 of netKarma/GEC13_report


Ignore:
Timestamp:
03/21/12 16:38:11 (12 years ago)
Author:
chsmall@grnoc.iu.edu
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • netKarma/GEC13_report

    v1 v2  
    1 NetKarma
    2 Report for the period ending GEC12
    3 =======================
     1==NetKarma
     2
     3=Report for the period ending GEC12
    44
    55Beth Plale, School of Informatics and Computing, Indiana University
    … …  
    77
    88
    9 Summary
    10 ======
     9= Summary
     10
     11For the quarter ending with GEC13, we released the NetKarma portal which allows experimentors to easily add and retrieve experimental data from the NetKarma provenance archive. The NetKarma Portal is integrated with the NetKarma Provenance System and provides a persistent service on servers at Indiana University to enable GENI experimentors to help understand the conditions of their experiment. The NetKarma links the experiment workflow data obtained from tools such as GUSH to representations of the experiment's GENI topology, the GENI Instramentation and Measurement systems, and log term data archives. NetKarma links all of these systems together to simplify the complexity of running an experiment on the GENI framework.
     12[[BR]]
     13WiMax summary
     14
     15= Milestones Delivered
     16
     17'''S4b.1 Demonstration of the provenance system with WiMax'''.
     18
     19WiMax description
     20[[BR]]
     21We have utalised the work on the NetKarma Portal to greatly improve the usability and utility of the experimental data. From the portal the ingestion process becomes
     22just requires the experimentor drag the files related to there experiment and have the experimental data ingested into NetKarma in one easy step. In addition to the portal, the AXIS2 and RabbitMQ mechanisms for adapters are supported allowing for experimentors to automated the ingestion process if desired.
    1123
    1224
    13 For the quarter ending with GEC13, we released the NetKarma portal which allows experimentors to easily add and retrieve experimental data from the NetKarma provenance archive. The NetKarma Portal is integrated with the NetKarma Provenance System and provides a persistent service on servers at Indiana University to enable GENI experimentors to help understand the conditions of their experiment.
    14 
    15 NetKarma
     25'''S4b.2 Linking workflow provenance information to metadata'''
     26The NetKarma portal is designed to link metadata such as the metadata object descriptor to the workflow information and provide the experimentor a way to navigate between not only the metadata and provenance information but also provides a way to visualise the measurement data that any metadata refers to and provide links to additional sources of data such as GEMINI, GIMS or the GMOC. The NetKarma portal supports ingestion of the metadata object directly or reference. For example a reference to a resiurce in a UNIS or NMWG file could be linked directly to the measurement data collected for that resource on the GENI topology. NetKarma provides an experimenter focused view to the data collected for the entire experiment.
    1627
    1728
    18 Deliverables
    19 ========
    20  
    21 '''Demonstration of the provenance system with WiMax'''. Written report (document or web page) on possible improvements to NetKarma to improve its utility and usability. We may have a person from the GPO repeat this experiment and evaluate NetKarma in the context of this experiment; provide reasonable support to this person.
    22 For the above experiment, show how the measurement data object descriptor may be populated automatically to point to provenance information such as software scripts/programs used by the experiment and possibly information about the execution environment that you collect from the GMOC.
    23 Written plan to work with a second experiment/experimenter for evaluation of the provenance system.
    24 Work with the GENI I&M community to define a standardized GENI event records. These records describe events of interest to the experiment such as failures, assignment/release of resources, etc. Present an initial schema design at GEC13.
     29'''S4b.3 Written plan to work with a second experiment/experimenter for evaluation of the provenance system'''
     30
     31'''Work with the GENI I&M community to define a standardized GENI event records.'''
    2532
    2633
     34= Work Preformed this Quarter
    2735
     36'''Work with the GENI I&M community to define a standardized GENI event records.'''  We have begone to examine some of the issues involved with creation of a standardised GENI event record. While NetKarma captures provenance information including "events" of the infrastructure used for a GENI experiment, NetKarma doesn't produce events but may capture events related to an experiment as part of the proveance capture. We have worked with ExoGeni, GMOC, I&M projects and the Mesoscale monitoring efforts as possible sources of event information about the creation, resource allocation and possible outages on the experimental infrastucture.
     37
     38
     39'''NetKarma Portal'''
    2840
    2941The NetKarma Portal provides a easy way to:
    … …  
    3143 * Create a new experiment,pointed to by a universial handle (DOI or ARK)
    3244 * Capture experiment workflow information from log files
    33  * Integrate GENI topology information
     45 * Integrate GENI topology information to an experiment
     46 * Intigrate measurements from GENI I&M implementations to an experiment
     47 * Visualise workflow, topology, measurement, status
     48 * Produce unified experment data representations such as a KML "movie" 
     49 * Interface to archival services (local and iRODS)
     50 
     51
     52= Experimental WorkFlow using the NetKarma Portal
     53
     54An experimentor creates an experimental handle to reference from the front page of the NetKarma portal. Any data products from the experiment can be uploaded to NetKarma. All are stored as archives but some supported file types are parsed and additional information gathered.
     55
     56Supported file types include:
     57
     58* Gush log files
     59* UNIS/NMWG topology representation (used in PerfSonar and GEMINI)
     60* GENI v3 RSpec files
     61
     62Once the files are uploaded to NetKarma Provenance store through the portal the workflow can be visualized both on the portal itself and through the NetKarma plugin for the Cytoscape desktop application. 
    3463
    3564
    36 We have continued to engage with the Measurement and Instrumentation team led by Harry Mussman and presented a lightening talk on provenance and initial thoughts on the MDOD at the I&M session at GEC12.
     65More information on the NetKarma Portal