wiki:netKarma/GEC13_report

Version 9 (modified by chsmall@grnoc.iu.edu, 8 years ago) (diff)

--

NetKarma

Report for the period ending GEC12

Beth Plale, School of Informatics and Computing, Indiana University
Chris Small, InCNTRE, Indiana University

Summary

For the quarter ending with GEC13, we released the NetKarma portal which allows experimentors to easily add and retrieve experimental data from the NetKarma provenance archive. The NetKarma Portal is integrated with the NetKarma Provenance System and provides a persistent service on servers at Indiana University to enable GENI experimentors to help understand the conditions of their experiment. The NetKarma links the experiment workflow data obtained from tools such as GUSH to representations of the experiment's GENI topology, the GENI Instramentation and Measurement systems, and log term data archives. NetKarma links all of these systems together to simplify the complexity of running an experiment on the GENI framework.

WiMax summary


Milestones Delivered

S4b.1 Demonstration of the provenance system with WiMax
WiMax description
We have utalised the work on the NetKarma Portal to greatly improve the usability and utility of the experimental data. From the portal the ingestion process becomes just requires the experimentor drag the files related to there experiment and have the experimental data ingested into NetKarma in one easy step. In addition to the portal, the AXIS2 and RabbitMQ mechanisms for adapters are supported allowing for experimentors to automated the ingestion process if desired.

S4b.2 Linking workflow provenance information to metadata The NetKarma portal is designed to link metadata such as the metadata object descriptor to the workflow information and provide the experimentor a way to navigate between not only the metadata and provenance information but also provides a way to visualise the measurement data that any metadata refers to and provide links to additional sources of data such as GEMINI, GIMS or the GMOC. The NetKarma portal supports ingestion of the metadata object directly or reference. For example a reference to a resiurce in a UNIS or NMWG file could be linked directly to the measurement data collected for that resource on the GENI topology. NetKarma provides an experimenter focused view to the data collected for the entire experiment.

S4b.3 Written plan to work with a second experiment/experimenter for evaluation of the provenance system_

Work Preformed this Quarter

Sb.4 Work with the GENI I&M community to define a standardized GENI event records We began to examine some of the issues involved with creation of a standardised GENI event record. While NetKarma captures provenance information including "events" of the infrastructure used for a GENI experiment, NetKarma doesn't produce events but may capture events related to an experiment as part of the proveance capture. We have worked with ExoGeni, GMOC, I&M projects and the Mesoscale monitoring efforts as possible sources of event information about the creation, resource allocation and possible outages on the experimental infrastucture.

NetKarma Portal

The NetKarma Portal provides a easy way for experimenters to:

  • Create a new experiment,pointed to by a universial handle (DOI or ARK)
  • Capture experiment workflow information from log files
  • Integrate GENI topology information to an experiment
  • Intigrate measurements from GENI I&M implementations to an experiment
  • Visualise workflow, topology, measurement, status
  • Produce unified experment data representations such as a KML "movie"
  • Interface to archival services (local and iRODS)
  • Browse the work of other experimenters (given proper permissions)

Experimental WorkFlow using the NetKarma Portal

An experimentor creates an experimental handle to reference from the front page of the NetKarma portal. Any data products from the experiment can be uploaded to NetKarma. All are stored as archives but some supported file types are parsed and additional information gathered.

Supported file types include:

  • Gush log files
  • UNIS/NMWG topology representation (used in PerfSonar and GEMINI)
  • GENI v3 RSpec files

Once the files are uploaded to NetKarma Provenance store through the portal the workflow can be visualized both on the portal itself and through the NetKarma plugin for the Cytoscape desktop application.

Project Participants

Collaborations

  • GMOC (status)
  • GEMINI/LAMP (monitoring)
  • iRODS
  • ExoGENI (status)
  • GPO Monitoring team for GENI MesoScale
  • Univ of Alaska (metadata and monitoring retrieval from VM instances
  • Mike Zink Univ of Massachusetts Amherst on GIMI data collection and archival mechanisms

Publications & Documents

Small, C, NetKarma Portal Poster GENI Engineering Conference 13, Los Angeles, CA http://groups.geni.net/geni/attachment/wiki/netKarma/Netkarma_GEC13.pdf

Small, C, NetKarma Portal Slides http://groups.geni.net/geni/attachment/wiki/netKarma/Netkarma_portal.pdf

GENI Documents

None

Attachments (12)