wiki:GIR2.1_NetKarma

Version 2 (modified by lnevers@bbn.com, 13 years ago) (diff)

--

NetKarma Evaluation

The gush2netkarma package deliverable is a script that uses a set of rules to read a log file and generate appropriate provenance information. The initial version was deemed too old (GEC7 version), a new version geni_adaptor.tar.gz (GEC8 version) was made available on Aug 3rd and requires a download password. Time frame for GEC8 version: This evaluation took place August 25-28, 2010.

A second version was made available on September 13, which delivered a new ruleset to handle log issues discovered in the previous evaluation.

NetKarma Findings

Requested account to get access to code that allows experiment provenance capture using GUSH. The gush2netkarma-1.0.tar.zip file is available Global Research NOC.

  • Pre-requisite of openjdk-6-jre and sun-java6-jdk not documented.
  • Installation instructions need clarification about the expected location of the various files.
  • Initial make fails with what seems like a missing file error.
  • env.sh sets CLASSPATH for java-1.6.0-open-jdk but does not set PATH for /usr/lib/jvm/java-1.6.0-openjdk/bin

Found from PI that version available is the deliverable of the first generation adaptor and that the version posted is suited to the purposes of the GEC7 demo only. Developer is working on cleaning that up and posting a version that has a clear separation of the front end adaptor, and demo-only back end graph generation.

The GEC8 version was downloaded on August 3. Upon use, it was found that some prerequisites where not captured:

  • graphviz: rich set of graph drawing tools
  • libgtkglext1: OpenGL Extension to GTK+ (shared libraries)
  • libgts-dev: GNU Triangulated Surface Library

Only the graphviz tool was added to the list of pre-requisistes.

Once all pre-requisites were installed, using logs from gush testing revealed an issue with the ruleset.xml delivered. Tried to verify that new ruleset addresses issues in September 3rd version.

NetKarma How-to

NetKarma GEC7 Version How-to

The GEC7 version of the netkarma tgz files deliver a Readme.txt which describes installation and run time:

Installation
-------------
The package has 4 parts:
i) src: contains the java files for parsing the logfiles.
ii) lib: contains jar files required for successful execution of the parser.
iii) toolbox-1.1.1-release.zip: the visualization tool for OPM graphs.
iv) config-files and shell-scripts: required for executing theend-to-end process for generating OPM graphs.

Configuration-info:
i) Copy the src/*.java files in a directory where the config-files & shell-scripts have been kept.
ii) Copy the lib directory.
iii) Unzip the OPM-toolbox
iv) Modify the config-parameters ($CLASSPATH & $JAVA_HOME) in env.sh and provenance_collector.sh, if required.

Compilation:
Compile the source-code using the following commands:-
i) source env.sh
ii) make

Additional-Info:
The package also contains the following files which are required for the programs to execute:-
i) ruleset.xml: the rule file to parse the geni logfiles.
ii) ruleset.properties: some decode-info for the ruleset.

Execution
----------
To generate the OPM graph, run the shell-script as:
./provenance_collector.sh <rule.xml file> <logfile> <out_opm_file> <.dot file> <.pdf file> <opmtoolbox-path>
  1. Unpacked the gush2netkarma-1.0.tar.zip which created directory named NetKarmaFiles.
  1. Executed the "Configuration-info" steps:
  • Not sure what is the target of step i. Some configuration files are delivered at the top level of the NetKarmaFiles

directory, so I created links in the NetKarmaFiles to the "src*.java" files.

  • The file "env.sh" and "provenance_collector.sh" files set CLASSPATH to a directory named ~/geni_parser, based on this, the step ii copies the lib to ~/geni_parser directory.
  • Unzip a files named "toolbox-1.1.1-release.zip" which created a directory named "OPM-toolbox".
  • Did not modify configuration parameters as files were placed in expected locations.
  1. Executed "Compilation" steps, ran into undocumented requirements:
    • Two packages had to be installed: openjdk-6-jre and sun-java6-jdk.
    • The env.sh sets the CLASSPATH, but also PATH needs to be set to point to /usr/lib/jvm/java-1.6.0-openjdk/bin

Once the above were completed, was able to execute the compilation steps:

 $ source env.sh
 $ make
 javac -g PropertyReader.java
 javac -g Decoder.java
 javac -g ProvenanceSummary.java
 javac -g ProvenanceInformation.java
 javac -g Comparators.java
 javac -g Constants.java
 javac -g SelectionQueries.java
 javac -g GeniLogParser.java
 javac -g OPMMapper.java
 javac -g OPMGraphGenerator.java

Tried to use Provenance collector script, but not enough information is available, could not get details about parameters and their meaning:

 $ ./provenance_collector.sh 
Usage: ./provenance_collector.sh <rule.xml file> <logfile> <out_opm_file> <.dot file> <.pdf file> <opmtoolbox-path>

NetKarma GEC8 Version How-to

  1. Downloaded geni_adaptor.tar.gz from https://www.slashtmp.iu.edu/public/download.php?FILE=dghoshal/44577wbJ1BL
  1. Unpacked the tgz bundle which delivers a directory named geni_parser containing a Readme.txt:
#
# Copyright 2010 The Trustees of Indiana University
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# -----------------------------------------------------------------
#
# Project:  NetKarma Provenance Service Adaptor
# File:  Readme.txt
# Description:  
#    This prototype provenance collection adaptor mines a log file against a 
#    set of rules and generates provenance events.  The provenance events would
#    normally be sent to the Karma service, but in this GEC7 demo (March 2010), the provenance
#    events are written to a file.  The file is then fed as input to a visualization
#    client that reads the file and generates a visual representation of the provenance 
#    graph.
# -----------------------------------------------------------------
# 

This module can be built from the included source.


Building from Source:
---------------------
Prerequisites:
i)  Apache-Ant  (the directory where this is installed is referred to as ANT_HOME)
ii) JDK 1.6     (the directory where this is installed is referred to as JAVA_HOME)

Edit the Configuration File:
In the env.sh file, set the paths for the GENI_HOME, JAVA_HOME, and ANT_HOME 
environment variables to match the directory where you unzipped the tarball (GENI_HOME)
and where you installed java and ant respectively.

Aside from the above prerequisites, all of the necessary source, configuration files, 
and jars are included in the tarball named geni_adaptor.tar.gz.  Unzip this tarball and it will 
create a directory named geni_parser (containing this readme file) that will be 
your build and installation directory.
This installation directory is referred to as <GENI_HOME>.

Also, unzip the toolbox-1.1.1-release.zip.

This module is built and run using a set of scripts.  The scripts in the <GENI_HOME>
directory are configured for the bash shell.  If you are using the C shell, please 
use the scripts in the <GENI_HOME>/tcsh_scripts instead.

To build the parser, run the following script in the <GENI_HOME> directory.
./compile.sh

If you encounter an error, please be sure that both ANT and JDK 1.6 are on 
your path. 

Running the Geni Parser:
------------------------
Edit the Configuration File:
Similar to compiling, please edit the settings for GENI_HOME and JAVA_HOME in the
geni_stdenvs.cfg file used to set the environment for running the parser.

Specifying the Rules Files:
When the parser is run it expects the following two files which are used to specify 
the rules used to map log entries into notifications. Defaults are included for these
in the <GENI_HOME> directory:
 i) ruleset.xml: the rule file to parse the geni logfiles.
 ii) ruleset.properties: some decode-info for the ruleset.

Execution:
-----------
To generate the OPM graph, run the shell-script as:
provenance_collector.sh <logfile>
where <logfile> is the log-file which is to be parsed

There is a sample log file named "logfile-ubuntu-15555-1267499048.txt" in the tarball.

Troubleshooting:
----------------
In case, you get any java-runtime linkage error for jaxb-api, put the
lib/jaxb-api-2.1.jar in <JAVA_HOME>/lib/endorsed directory.
  1. Install Apache Ant and JDK 1.6:
      $ sudo apt-get install ant
      $ sudo apt-get install sun-java6-jdk 
    
  2. Edit the configuration file env.sh to set GENI_HOME, JAVA_HOME, and ANT_HOME:
  export GENI_HOME=/home/lnevers/NetKarma/geni_parser
  export JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.20
  export ANT_HOME=/usr/share/ant
  export PATH=$ANT_HOME/bin:$JAVA_HOME/bin:$PATH

  export CLASSPATH=$GENI_HOME/lib/provInfo.jar:$GENI_HOME/lib/commons-lang-2.3.jar:$GENI_HOME/lib/jaxb-impl.jar:$GENI_HOME/lib/jsr173_1.0_api.jar:$GENI_HOME/lib/opm-core-1.1.1.jar:$GENI_HOME/lib/opm-elmo-1.1.1.jar:$GENI_HOME/lib/Ruleset.jar:$GENI_HOME/lib/runtime-0.4.1.5.jar:$GENI_HOME/lib/xbean.jar:$GENI_HOME/lib/xml-apis-1.0.b2.jar:.

  1. Unzipped toolbox-1.1.1-release.zip, which created a directory named opmToolbox:
      $ unzip toolbox-1.1.1-release.zip 
    
  1. Compiled the parser tool:
     lnevers@sendaria:~/NetKarma/geni_parser$ ./compile.sh 
     Buildfile: build.xml
    
     build-subprojects:
    
     init:
    
     build-project:
         [echo] Geni-Update-1.0: /home/lnevers/NetKarma/geni_parser/build.xml
        [javac] Compiling 11 source files to /home/lnevers/NetKarma/geni_parser/bin
    
     build:
    
     BUILD SUCCESSFUL
     Total time: 1 second
    
  1. Running the parser tool requires the following:
  • Edit the geni_stdenvs.cfg gile to set GENI_HOME and JAVA_HOME
  • Check rule files ruleset.xml and ruleset.properties.

Note: Initial attempt to run the parser showed a failure which resulted in the following packages being identified as required:

 $ sudo apt-get install graphviz
 $ sudo apt-get install libgtkglext1
 $ sudo apt-get install libgts-dev

Once the prerequisites are install, the parser can be run:

 $ ./provenance_collector.sh logfile-ubuntu-15555-1267499048.txt 

The provenance_collector.sh script generates 4 output files:

  • opm_out.dot
  • opm_out.pdf
  • opm_out.rdf
  • opm_out.xml

The opm_out.pdf for logfile-ubuntu-15555-1267499048.txt shows the following:

The opm_out.rdf for logfile-ubuntu-15555-1267499048.txt when viewed with suggested RDF tools Welkin:

A "Play" button on the drawing tab for welkin tools moved the nodes in the upper-right-hand quadrant:

The opm_out.dot for logfile-ubuntu-15555-1267499048.txt shows the following:

Attachments (4)

Download all attachments as: .zip