Changes between Version 30 and Version 31 of netKarma/GEC14_report


Ignore:
Timestamp:
07/23/12 02:23:06 (12 years ago)
Author:
chsmall@grnoc.iu.edu
Comment:

NetKarma Portal description

Legend:

Unmodified
Added
Removed
Modified

  • netKarma/GEC14_report

    v30 v31  
    11= NetKarma =
    22
    3 '''Report for the period ending GEC13'''
     3'''Report for the period ending GEC14'''
    44
    55Beth Plale, School of Informatics and Computing, Indiana University[[br]]
    … …  
    2727The tutorial first walked the participants through an experiment using Emulab ProtoGENI resources to run an experiment doing a GridFTP transfer based on the eXtensible Session Protocol (XSP).  Since the focus was on provenance generation, the resources had already been requested and configured, and the participants ran scripts loaded in the tutorial VM on VirtualBox to run the experiment – generating a log from the XSP experiment, some basic bandwidth measurements using iperf, and downloading the manifest for the resources they are using.  The NetKarma adaptor which is incorporated into the NetKarma portal used these two log files and the manifest in the second hands-on portion of the tutorial to extract provenance and load it into NetKarma through the Portal.
    2828
    29 <Portal section>
     29The second part of the tutorial shows how you can take and data related to an experiment and create a simple way to store, refer to and analyse the results and other information about the experiment. Participants of the tutorial created an experiment on a demo version of the [http://netkarma-demo.incntre.iu.edu NetKarma Portal]. Creating an experiment created a [http://www.doi.org/ DOI handle] to create a universal way to refer to all the data artifacts and information involved in creating the experiment. The tutorial participants then took the sample results and the manifest file from the XSP experiment and uploaded it to the Portal. The Portal analysed the files uploaded and ingested them into the Karma provenance system. Information on what network resources and the workflow of the experiment is gleaned from the uploaded data in order to construct a model of what happened in the experiment. Users could use the portal to visualise the new data products produced and download both a OpenProvence Model graph.
    3030
    3131The third hands-on portion of the tutorial showed participants how they can easily download a provenance graph generated by NetKarma from the Portal.  The downloaded graph is an XML file based on the Open Provenance Model (OPM) standard for provenance.  For very large provenance graphs, the NetKarma plug-in can be used to visualize and explore large provenance graphs using the desktop version of Cytoscape (the web version of cytoscape is used in the Portal).  In this section participants downloaded provenance graphs for experiments we had run on both NS2 and ORBIT and loaded into the portal using the corresponding NetKarma adaptors.   The provenance graphs for these experiments differ from the XSP experiment in that whereas the adaptor for the XSP experiment uses the manifest to incorporate the topology of the experiment into the provenance graph, the focus of the NS2 and ORBIT adaptors is on capturing the success of each packet transferred.  This focus on the packet level results in large provenance graphs, but as participants were able to experience by loading the graphs for both experiments into the copy of Cytoscape included in the tutorial VM, they could quickly see the performance of NS2 and ORBIT experiments as to packets successfully transferred or dropped by using visualization capabilities included in the NetKarma plug-in.
    … …  
    4545'''NetKarma Portal'''
    4646
    47 <add Portal Details>
     47The NetKarma Portal simplifies the analysis of experiments and provides a way to a visualise, save and reference GENI Experiments. Using the NetKarma portal experimenters can easily recreate existing experiments using the information used in creating the experiment. It also visualizes the workflow and topology of experiments to aid in the understanding of the exact workflow and circumstances of an experiment.
    4848
     49A key part of the scientific method is that experiments must be reproducible. In order to reproduce experiments you will need to have knowledge of exactly the circumstances of an experiment. This includes not only the results but the specification of the experiment. This is especially an issue in large scale network experiments that GENI makes possible. The initial conditions may be complex and include hundreds of resources. The NetKarma strives to make the reproducibility of experiments simple by providing a place to store all the files used to create an experiment, logs from the process, such as GUSH logs and the results themselves. The NetKarma Portal also provides value for the initial experimenter by producing data products from the raw files uploaded. Another possible use of the NetKarma Portal is an archival mechanism as part of the NSF Data Management requirements for all NSF funded research.
     50
     51[[Image(NetKarma_Portal_Diagram.png,65%)]]
     52
     53
     54The first action when using the NetKarma Portal is to create a new experiment. This allows all data uploaded to be grouped together in that experiment. Creating the experiment also creates a DOI handle that can be used to reference the data. The DOI handle can be used for example in the references of any papers written about the work. The DOI data reference will then allow a reader of the paper to retrive all relevant data about the experiment that created the work. It also allows the linking of published work to the GENI framework that help produce the work.  The DOI handle can also be used with any DOI resolver such as [http://www.crossref.org/ CrossRef] or the one at [http://dx.doi.org/ http://dx.doi.org/] to search for the work. The NetKarma Portal utilizes the [http://n2t.net/ezid/ EZID web service] and [http://www.datacite.org/ DataSite] organization to provide unique handles. The home page also provides a listing of all experiments that have been previously created in the Portal. These experiments can be searched and browsed by other researchers.
     55
     56
     57[[Image(NetKarma_Portal_home.png,65%)]]
     58
     59Once an experiment is created the experimenter can easily load files by just dragging and dropping them on the experiment page. When a file is uploaded it is analysed for useful information and if it contains information in a format that the NetKarma Portal can understand and parse is ingested into the Karma DB service. All files are archived as well as analysed. The Karma DB contains a model of the workflowof the experiment. This can include what processes were run in which order and the resources the ran on. The portal makes information stored in the Karma DB available in two ways. 1) It creates a visual graph using the [http://cytoscapeweb.cytoscape.org/ Cytoscape Web] tool directly in the portal 2) It makes the OPM graph XML representation available for download. Using the OPM file directly allows you to use the [http://www.cytoscape.org/ CytoScape] desktop tool which can produce more elaborate and customizable visualization than available directly in the portal.
     60
     61[[Image(NetKarma_Portal_exp.png,65%)]]
     62 
     63In addition to workflow visualization the NetKarma Portal also provides a topological view if files contain topology representations are uploaded in to the portal. The Portal understands GENI API  UNIS and NMWG formats used in GEMINI and PerfSonar. Uploading these files produce a CytoScape Web representation similar to the
    4964
    5065
    … …  
    137152== GENI Documents ==
    138153none
    139