Changes between Initial Version and Version 1 of GIR2.1_NetKarma


Ignore:
Timestamp:
03/14/11 22:00:54 (9 years ago)
Author:
lnevers@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • GIR2.1_NetKarma

    v1 v1  
     1== !NetKarma Evaluation ==
     2
     3The gush2netkarma package deliverable is a script that uses a set of rules to read a log file and generate appropriate provenance information.   The initial version was deemed too old (GEC7 version), a new version [https://www.slashtmp.iu.edu/public/download.php?FILE=dghoshal/44577wbJ1BL geni_adaptor.tar.gz ](GEC8 version) was made available on Aug 3rd and requires a download password. Time frame for GEC8 version: This evaluation took place August 25-28, 2010.
     4
     5A second version was made available on September 13, which delivered a new ruleset to handle log issues discovered in the previous evaluation.
     6
     7== !NetKarma Findings ==
     8
     9Requested account to get access to code that allows experiment provenance capture using GUSH. The [http://globalnoc.iu.edu/grnoc-internal/file-bin/system-engineering/software-projects/netkarma/gush2netkarma-1.0.tar.zip gush2netkarma-1.0.tar.zip] file is available Global Research NOC.
     10
     11 * Pre-requisite of openjdk-6-jre and sun-java6-jdk not documented.
     12 * Installation instructions need clarification about the expected location of the various files. 
     13 * Initial make fails with what seems like a missing file error.
     14 * env.sh sets CLASSPATH for java-1.6.0-open-jdk but does not set PATH for /usr/lib/jvm/java-1.6.0-openjdk/bin
     15
     16Found from PI that version available is the deliverable of the first generation adaptor and that the version
     17posted is suited to the purposes of the GEC7 demo only.  Developer is working on cleaning that up and posting
     18a version that has a clear separation of the front end adaptor, and demo-only back end graph generation. 
     19
     20The GEC8 version was downloaded on August 3. Upon use, it was found that some prerequisites where not captured:
     21    * graphviz: rich set of graph drawing tools
     22    * libgtkglext1: OpenGL Extension to GTK+ (shared libraries)
     23    * libgts-dev: GNU Triangulated Surface Library
     24Only the graphviz tool was added to the list of pre-requisistes.
     25
     26Once all pre-requisites were installed, using logs from gush testing revealed an issue with the ruleset.xml delivered.
     27Tried to verify that new ruleset addresses issues in September 3rd version.
     28
     29== !NetKarma How-to ==
     30
     31'''!NetKarma GEC7 Version How-to'''
     32
     33The GEC7 version of the netkarma tgz files deliver a Readme.txt which describes installation and run time:
     34{{{
     35Installation
     36-------------
     37The package has 4 parts:
     38i) src: contains the java files for parsing the logfiles.
     39ii) lib: contains jar files required for successful execution of the parser.
     40iii) toolbox-1.1.1-release.zip: the visualization tool for OPM graphs.
     41iv) config-files and shell-scripts: required for executing theend-to-end process for generating OPM graphs.
     42
     43Configuration-info:
     44i) Copy the src/*.java files in a directory where the config-files & shell-scripts have been kept.
     45ii) Copy the lib directory.
     46iii) Unzip the OPM-toolbox
     47iv) Modify the config-parameters ($CLASSPATH & $JAVA_HOME) in env.sh and provenance_collector.sh, if required.
     48
     49Compilation:
     50Compile the source-code using the following commands:-
     51i) source env.sh
     52ii) make
     53
     54Additional-Info:
     55The package also contains the following files which are required for the programs to execute:-
     56i) ruleset.xml: the rule file to parse the geni logfiles.
     57ii) ruleset.properties: some decode-info for the ruleset.
     58
     59Execution
     60----------
     61To generate the OPM graph, run the shell-script as:
     62./provenance_collector.sh <rule.xml file> <logfile> <out_opm_file> <.dot file> <.pdf file> <opmtoolbox-path>
     63}}}
     64
     651. Unpacked the gush2netkarma-1.0.tar.zip which created directory named !NetKarmaFiles.
     66
     672. Executed the "Configuration-info" steps:
     68
     69   * Not sure what is the target of step i. Some configuration files are delivered at the top level of the !NetKarmaFiles
     70directory, so I created links in the !NetKarmaFiles to the "src*.java" files.
     71
     72   * The file "env.sh" and "provenance_collector.sh" files set CLASSPATH to a directory named ~/geni_parser, based on this, the step ii copies the '''lib''' to '''~/geni_parser''' directory.
     73
     74   * Unzip a files named "toolbox-1.1.1-release.zip" which created a directory named "OPM-toolbox".
     75
     76   * Did not modify configuration parameters as files were placed in expected locations.
     77
     783. Executed "Compilation" steps, ran into undocumented requirements:
     79  * Two packages had to be installed: openjdk-6-jre and sun-java6-jdk.
     80  * The env.sh sets the CLASSPATH, but also PATH needs to be set to point to /usr/lib/jvm/java-1.6.0-openjdk/bin
     81
     82Once the above were completed, was able to execute the compilation steps:
     83{{{
     84 $ source env.sh
     85 $ make
     86 javac -g PropertyReader.java
     87 javac -g Decoder.java
     88 javac -g ProvenanceSummary.java
     89 javac -g ProvenanceInformation.java
     90 javac -g Comparators.java
     91 javac -g Constants.java
     92 javac -g SelectionQueries.java
     93 javac -g GeniLogParser.java
     94 javac -g OPMMapper.java
     95 javac -g OPMGraphGenerator.java
     96}}}
     97
     98Tried to use Provenance collector script, but not enough information is available, could not get details
     99about parameters and their meaning:
     100{{{
     101 $ ./provenance_collector.sh
     102Usage: ./provenance_collector.sh <rule.xml file> <logfile> <out_opm_file> <.dot file> <.pdf file> <opmtoolbox-path>
     103}}}
     104
     105'''!NetKarma GEC8 Version How-to'''
     106
     1071. Downloaded geni_adaptor.tar.gz from https://www.slashtmp.iu.edu/public/download.php?FILE=dghoshal/44577wbJ1BL
     108
     1092. Unpacked the tgz bundle which delivers a directory named '''geni_parser''' containing a Readme.txt:
     110
     111{{{
     112#
     113# Copyright 2010 The Trustees of Indiana University
     114#
     115# Licensed under the Apache License, Version 2.0 (the "License");
     116# you may not use this file except in compliance with the License.
     117# You may obtain a copy of the License at
     118#
     119# http://www.apache.org/licenses/LICENSE-2.0
     120#
     121# Unless required by applicable law or agreed to in writing, software
     122# distributed under the License is distributed on an "AS IS" BASIS,
     123# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
     124# See the License for the specific language governing permissions and
     125# limitations under the License.
     126#
     127# -----------------------------------------------------------------
     128#
     129# Project:  NetKarma Provenance Service Adaptor
     130# File:  Readme.txt
     131# Description: 
     132#    This prototype provenance collection adaptor mines a log file against a
     133#    set of rules and generates provenance events.  The provenance events would
     134#    normally be sent to the Karma service, but in this GEC7 demo (March 2010), the provenance
     135#    events are written to a file.  The file is then fed as input to a visualization
     136#    client that reads the file and generates a visual representation of the provenance
     137#    graph.
     138# -----------------------------------------------------------------
     139#
     140
     141This module can be built from the included source.
     142
     143
     144Building from Source:
     145---------------------
     146Prerequisites:
     147i)  Apache-Ant  (the directory where this is installed is referred to as ANT_HOME)
     148ii) JDK 1.6     (the directory where this is installed is referred to as JAVA_HOME)
     149
     150Edit the Configuration File:
     151In the env.sh file, set the paths for the GENI_HOME, JAVA_HOME, and ANT_HOME
     152environment variables to match the directory where you unzipped the tarball (GENI_HOME)
     153and where you installed java and ant respectively.
     154
     155Aside from the above prerequisites, all of the necessary source, configuration files,
     156and jars are included in the tarball named geni_adaptor.tar.gz.  Unzip this tarball and it will
     157create a directory named geni_parser (containing this readme file) that will be
     158your build and installation directory.
     159This installation directory is referred to as <GENI_HOME>.
     160
     161Also, unzip the toolbox-1.1.1-release.zip.
     162
     163This module is built and run using a set of scripts.  The scripts in the <GENI_HOME>
     164directory are configured for the bash shell.  If you are using the C shell, please
     165use the scripts in the <GENI_HOME>/tcsh_scripts instead.
     166
     167To build the parser, run the following script in the <GENI_HOME> directory.
     168./compile.sh
     169
     170If you encounter an error, please be sure that both ANT and JDK 1.6 are on
     171your path.
     172
     173Running the Geni Parser:
     174------------------------
     175Edit the Configuration File:
     176Similar to compiling, please edit the settings for GENI_HOME and JAVA_HOME in the
     177geni_stdenvs.cfg file used to set the environment for running the parser.
     178
     179Specifying the Rules Files:
     180When the parser is run it expects the following two files which are used to specify
     181the rules used to map log entries into notifications. Defaults are included for these
     182in the <GENI_HOME> directory:
     183 i) ruleset.xml: the rule file to parse the geni logfiles.
     184 ii) ruleset.properties: some decode-info for the ruleset.
     185
     186Execution:
     187-----------
     188To generate the OPM graph, run the shell-script as:
     189provenance_collector.sh <logfile>
     190where <logfile> is the log-file which is to be parsed
     191
     192There is a sample log file named "logfile-ubuntu-15555-1267499048.txt" in the tarball.
     193
     194Troubleshooting:
     195----------------
     196In case, you get any java-runtime linkage error for jaxb-api, put the
     197lib/jaxb-api-2.1.jar in <JAVA_HOME>/lib/endorsed directory.
     198}}}
     199
     2003. Install Apache Ant and JDK 1.6:
     201{{{
     202  $ sudo apt-get install ant
     203  $ sudo apt-get install sun-java6-jdk
     204}}}
     2054. Edit the configuration file env.sh to set GENI_HOME, JAVA_HOME, and ANT_HOME:
     206
     207{{{
     208  export GENI_HOME=/home/lnevers/NetKarma/geni_parser
     209  export JAVA_HOME=/usr/lib/jvm/java-6-sun-1.6.0.20
     210  export ANT_HOME=/usr/share/ant
     211  export PATH=$ANT_HOME/bin:$JAVA_HOME/bin:$PATH
     212
     213  export CLASSPATH=$GENI_HOME/lib/provInfo.jar:$GENI_HOME/lib/commons-lang-2.3.jar:$GENI_HOME/lib/jaxb-impl.jar:$GENI_HOME/lib/jsr173_1.0_api.jar:$GENI_HOME/lib/opm-core-1.1.1.jar:$GENI_HOME/lib/opm-elmo-1.1.1.jar:$GENI_HOME/lib/Ruleset.jar:$GENI_HOME/lib/runtime-0.4.1.5.jar:$GENI_HOME/lib/xbean.jar:$GENI_HOME/lib/xml-apis-1.0.b2.jar:.
     214
     215}}}
     216
     2175. Unzipped toolbox-1.1.1-release.zip, which created a directory named opmToolbox:
     218{{{
     219  $ unzip toolbox-1.1.1-release.zip
     220}}}
     221
     2226. Compiled the parser tool:
     223{{{
     224 lnevers@sendaria:~/NetKarma/geni_parser$ ./compile.sh
     225 Buildfile: build.xml
     226
     227 build-subprojects:
     228
     229 init:
     230
     231 build-project:
     232     [echo] Geni-Update-1.0: /home/lnevers/NetKarma/geni_parser/build.xml
     233    [javac] Compiling 11 source files to /home/lnevers/NetKarma/geni_parser/bin
     234
     235 build:
     236
     237 BUILD SUCCESSFUL
     238 Total time: 1 second
     239}}}
     240
     2417. Running the parser tool requires the following:
     242
     243 * Edit the geni_stdenvs.cfg gile to set GENI_HOME and JAVA_HOME
     244 * Check rule files ruleset.xml and ruleset.properties.
     245
     246Note: Initial attempt to run the parser showed a failure which resulted in the following packages being identified as required:
     247{{{
     248 $ sudo apt-get install graphviz
     249 $ sudo apt-get install libgtkglext1
     250 $ sudo apt-get install libgts-dev
     251}}}
     252
     253Once the prerequisites are install, the parser can be run:
     254{{{
     255 $ ./provenance_collector.sh logfile-ubuntu-15555-1267499048.txt
     256}}}
     257
     258The provenance_collector.sh script generates 4 output files:
     259
     260 * opm_out.dot
     261 * opm_out.pdf
     262 * opm_out.rdf
     263 * opm_out.xml
     264
     265The opm_out.pdf for logfile-ubuntu-15555-1267499048.txt shows the following:
     266[[Image(2010-09-15_netkarma-pdf.jpg)]]
     267
     268The opm_out.rdf for logfile-ubuntu-15555-1267499048.txt when viewed with suggested RDF tools Welkin:
     269[[Image(2010-09-15_netkarma-rdf.jpg)]]
     270
     271A "Play" button on the drawing tab for welkin tools moved the nodes in the upper-right-hand quadrant:
     272[[Image(2010-09-15_netkarma-rdf2.jpg)]]
     273
     274The opm_out.dot for logfile-ubuntu-15555-1267499048.txt shows the following:
     275[[Image(2010-09-15_netkarma-dot.jpg)]]