Subject: Re: NetKarma Feedback From: Scott Jensen Date: 10/28/2011 1:50 PM To: Luisa Nevers CC: "Ping, Robert James" , "Jensen, Scott" , Beth Plale , Vicraj Thomas , Heidi Picher Dempsey Hi Luisa, Thanks for the continued feedback on NetKarma - it makes it a stronger provenance tool. Based on your feedback, we have uploaded new versions of the Quick Start Guide, the NetKarma Gush Adapter Manual, and the tarball for the NetKarma Adaptor (for changes in the README file). Please see embedded responses below. On Fri, Oct 21, 2011 at 3:16 PM, Luisa Nevers wrote: While waiting on the answers to my previous email. I tried to use the provenance script and have some additional feedback. 9. The karma-adaptor-2.5.tar.gz file when unpacked creates a directory with no versioning. Configuration files examples show versions as part of the name although not 2.5: "export ADAPTOR_HOME=/home/peng/workspace/Karma-Adaptor-2.0" As you noted, the default directory name when the tarball is unpacked is Karma-Adaptor, so we updated the example in the configuration file to not include a version in the path name. Although ADAPTOR_HOME needs to be set to each user's path (based on where they unpacked the tarball), it could be confusing to have the version in the example path so we removed it. 10. Using the instructions provided in the NetKarma Version 2.5 Quick Start Guide , configured and compiled the karma-adaptor and then proceeded to run the provenance_collector.sh script, which fails as follows: lnevers@sendaria:~/netkarma/Karma-Adaptor$ ./provenance_collector.sh logfile-sendaria-15555-1319031471.txt Connecting to Server... Creating Notifications... java.io.IOException: Possibly caused by authentication failure at com.rabbitmq.client.impl.AMQChannel.wrap(AMQChannel.java:127) at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:298) at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:382) at com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:427) at edu.indiana.dsi.karma.messaging.Sender.(Sender.java:49) at edu.indiana.d2i.adaptor.LogParser.(Unknown Source) at edu.indiana.d2i.adaptor.client.KarmaAdaptor.main(Unknown Source) Caused by: com.rabbitmq.client.ShutdownSignalException: connection error; reason: java.io.EOFException at com.rabbitmq.utility.ValueOrException.getValue(ValueOrException.java:81) at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:47) at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:342) at com.rabbitmq.client.impl.AMQChannel.rpc(AMQChannel.java:215) at com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:296) ... 5 more Caused by: java.io.EOFException at java.io.DataInputStream.readUnsignedByte(DataInputStream.java:290) at com.rabbitmq.client.impl.Frame.readFrom(Frame.java:118) at com.rabbitmq.client.impl.SocketFrameHandler.readFrame(SocketFrameHandler.java:133) at com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.java:409) at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:437) doMapping................. doMapping................. Exception in thread "main" java.lang.NullPointerException at edu.indiana.d2i.adaptor.LogParser.closeConnection(Unknown Source) at edu.indiana.d2i.adaptor.LogParser.parseRuleFile(Unknown Source) at edu.indiana.d2i.adaptor.client.KarmaAdaptor.main(Unknown Source) We had a miscommunication on our end and the password for uploading logs was set to a different value on the server here at IU than was specified in the Quick Start Guide. The password has now been reset to the value specified in the Quick Start Guide and we had a couple new users walk through the process based on the Quick Start Guide and NetKarma GUSH Adaptor Manual. 11. The "NetKarma GUSH Adaptor Tool User Manual V2.5" (August 26, 2011) instructs the user to unpack a file named "karma-adaptor-tool-v2.5.tar.gz" the name of the tgz that is available is "karma-adaptor-2.5.tar.gz". The manual has been corrected to the correct name of the tarball. 12. There are inconsistencies in the instructions between the README and the User guide: - README instructs to define ANT_HOME and the User Guide does not. - README instucts run "ant karma-adaptor", User Guide instructs run "ant adaptor". The manual was updated to specify that ANT_HOME needs to be set and reflect that the correct ant task is karma-adaptor. 13. The netkarma_gmoc_adaptor_user_manual_v1.0.0.pdf expects a tar file named NetKarmaGMOC_Adaptor.tar the tgz file available is named netkarmagmocadaptor-1.0.0.tar.gz We will update the user manual. For this release, the GMOC adaptor is being made available through the server located at Indiana University as software as a resource. Users would not need to download and install the GMOC adapter themselves, but we made the source code and build instructions available. 14. Using instructions from GMOC Adaptor instructions untarred and ran ant, which failed: lnevers@sendaria:~/netkarma/GMOCAdaptor$ ant create.jar Buildfile: build.xml init: [mkdir] Created dir: /home/lnevers/netkarma/GMOCAdaptor/client_gen [mkdir] Created dir: /home/lnevers/netkarma/GMOCAdaptor/client_gen/classes [mkdir] Created dir: /home/lnevers/netkarma/GMOCAdaptor/dist/lib [mkdir] Created dir: /home/lnevers/netkarma/GMOCAdaptor/dist/test/classes wsdl2java: [java] Retrieving document at '/home/lnevers/netkarma/GMOCAdaptor/config/KarmaService.wsdl'. [java] Retrieving schema at 'karma_query_schema.xsd', relative to 'file:/home/lnevers/netkarma/GMOCAdaptor/config/KarmaService.wsdl'. [java] Retrieving schema at 'rdf.xsd', relative to 'file:/home/lnevers/netkarma/GMOCAdaptor/config/karma_query_schema.xsd'. [java] Retrieving schema at 'karma_ingest_type_schema.xsd', relative to 'file:/home/lnevers/netkarma/GMOCAdaptor/config/karma_query_schema.xsd'. [java] Retrieving schema at 'opm.1_1.xsd', relative to 'file:/home/lnevers/netkarma/GMOCAdaptor/config/karma_query_schema.xsd'. [java] Retrieving schema at 'karma_ingest_type_schema.xsd', relative to 'file:/home/lnevers/netkarma/GMOCAdaptor/config/KarmaService.wsdl'. [java] log4j:WARN No appenders could be found for logger (org.apache.axis2.description.WSDL11ToAllAxisServicesBuilder). [java] log4j:WARN Please initialize the log4j system properly. [java] Exception in thread "main" org.apache.axis2.wsdl.codegen.CodeGenerationException: org.apache.axis2.wsdl.codegen.CodeGenerationException: org.apache.axis2.wsdl.databinding.UnmatchedTypeException: No type was mapped to the name serviceInitialized with namespace http://www.dataandsearch.org/karma/2010/08/ [java] at org.apache.axis2.wsdl.codegen.CodeGenerationEngine.generate(CodeGenerationEngine.java:271) [java] at org.apache.axis2.wsdl.WSDL2Code.main(WSDL2Code.java:35) [java] at org.apache.axis2.wsdl.WSDL2Java.main(WSDL2Java.java:24) [java] Caused by: org.apache.axis2.wsdl.codegen.CodeGenerationException: org.apache.axis2.wsdl.databinding.UnmatchedTypeException: No type was mapped to the name serviceInitialized with namespace http://www.dataandsearch.org/karma/2010/08/ [java] at org.apache.axis2.wsdl.codegen.emitter.AxisServiceBasedMultiLanguageEmitter.emitStub(AxisServiceBasedMultiLanguageEmitter.java:534) [java] at org.apache.axis2.wsdl.codegen.CodeGenerationEngine.generate(CodeGenerationEngine.java:260) [java] ... 2 more [java] Caused by: org.apache.axis2.wsdl.databinding.UnmatchedTypeException: No type was mapped to the name serviceInitialized with namespace http://www.dataandsearch.org/karma/2010/08/ [java] at org.apache.axis2.wsdl.databinding.TypeMappingAdapter.getTypeMappingName(TypeMappingAdapter.java:82) [java] at org.apache.axis2.wsdl.codegen.emitter.AxisServiceBasedMultiLanguageEmitter.getInputParamElement(AxisServiceBasedMultiLanguageEmitter.java:2913) [java] at org.apache.axis2.wsdl.codegen.emitter.AxisServiceBasedMultiLanguageEmitter.getInputElement(AxisServiceBasedMultiLanguageEmitter.java:2697) [java] at org.apache.axis2.wsdl.codegen.emitter.AxisServiceBasedMultiLanguageEmitter.generateMethodElement(AxisServiceBasedMultiLanguageEmitter.java:2261) [java] at org.apache.axis2.wsdl.codegen.emitter.AxisServiceBasedMultiLanguageEmitter.loadOperations(AxisServiceBasedMultiLanguageEmitter.java:2151) [java] at org.apache.axis2.wsdl.codegen.emitter.AxisServiceBasedMultiLanguageEmitter.createDOMDocumentForInterfaceImplementation(AxisServiceBasedMultiLanguageEmitter.java:826) [java] at org.apache.axis2.wsdl.codegen.emitter.AxisServiceBasedMultiLanguageEmitter.writeInterfaceImplementation(AxisServiceBasedMultiLanguageEmitter.java:718) [java] at org.apache.axis2.wsdl.codegen.emitter.AxisServiceBasedMultiLanguageEmitter.emitStub(AxisServiceBasedMultiLanguageEmitter.java:504) [java] ... 3 more [java] Java Result: 1 jar: [javac] Compiling 204 source files to /home/lnevers/netkarma/GMOCAdaptor/client_gen/classes [jar] Building jar: /home/lnevers/netkarma/GMOCAdaptor/dist/lib/client-stub-types.jar create.compile: [javac] Compiling 4 source files to /home/lnevers/netkarma/GMOCAdaptor/dist/test/classes [javac] /home/lnevers/netkarma/GMOCAdaptor/test/edu/indiana/dsi/karma/test/GMOCDCollector.java:75: cannot find symbol [javac] symbol : class KarmaServiceStub [javac] location: package org.dataandsearch.www.karma._2010._08 [javac] import org.dataandsearch.www.karma._2010._08.KarmaServiceStub; [javac] ^ [javac] /home/lnevers/netkarma/GMOCAdaptor/test/edu/indiana/dsi/karma/test/GMOCDCollector.java:148: cannot find symbol [javac] symbol : class KarmaServiceStub [javac] location: class edu.indiana.dsi.karma.test.GMOCDCollector [javac] KarmaServiceStub stub = null; [javac] ^ [javac] /home/lnevers/netkarma/GMOCAdaptor/test/edu/indiana/dsi/karma/test/GMOCDCollector.java:155: cannot find symbol [javac] symbol : class KarmaServiceStub [javac] location: class edu.indiana.dsi.karma.test.GMOCDCollector [javac] stub = new KarmaServiceStub(serviceURL); [javac] ^ [javac] /home/lnevers/netkarma/GMOCAdaptor/test/edu/indiana/dsi/karma/test/GMOCDCollector.java:224: cannot find symbol [javac] symbol : method setWorkflowID(java.lang.String) [javac] location: interface org.dataandsearch.www.karma.query._2010._10.FindServiceRequestType [javac] findServiceRequest.setWorkflowID(":"); [javac] ^ [javac] /home/lnevers/netkarma/GMOCAdaptor/test/edu/indiana/dsi/karma/test/GMOCDCollector.java:271: cannot find symbol [javac] symbol : method getServiceIDList() [javac] location: interface org.dataandsearch.www.karma.query._2010._10.FindServiceResponseType [javac] ServiceIDListType serviceids = servicetype.getServiceIDList(); [javac] ^ [javac] 5 errors BUILD FAILED /home/lnevers/netkarma/GMOCAdaptor/build.xml:76: Compile failed; see the compiler error output for details. Total time: 7 seconds Please see response to item #13. Unpacking the tarball, the build process ran cleanly when I retried it. was JAVA_HOME set and on your path? There may be further details we need to include in the manual. Thanks again for your feedback. Scott Thanks, Luisa On 10/12/2011 12:28 PM, Luisa Nevers wrote: > Hello Robert, > > I have been taking a look a the NetKarma Version 2.5 and have some feedback. > > 1. The NetKarma Version 2.5 Quick Start Guide instruct the user to download > version 2.0 files rather than version 2.5 : > "https://pti.iu.edu/sites/default/files/karma-adaptor-2.0.tar.gz" and > " https://pti.iu.edu/sites/default/files/karma-adaptor_v2.0_user_manual.pdf" > > 2. The NetKarma Version 2.5 Adaptor requires ANT and JDK to be installed, which > are not listed as prerequisites in the Quick Start Guide. Also RabbitMQ shoudl > be listed as a prerequisite. > > 3. Using the README for the NetKarma Adaptor. In the section named > "Edit the Environment Configuration File" the user is instructed to set > "ADAPTOR_HOME, JAVA_HOME, and ANT_HOME", the first two exist in > the configuration file, but ANT_HOME does not exist. > > 4. The Quick Start Guide Steps 4 and 5 instruct to modify karma-adaptor.properties > and KarmaQueryConfig.txt, but does not provide a path to these files. > > 5. The "NetKarma GUSH Adaptor Tool User Manual" section 2.2 "Service Dependencies" > instructs me to download RabbitMQ, but does not specify if I need the server, the client, > both, or some other component. > > 6. The "NetKarma GUSH Adaptor Tool User Manual" section 3 lists "karma-adaptor.homedir" > as a karma-adaptor.properties setting, but is not found in the file. Is this an optional setting? > > 7. The "NetKarma GUSH Adaptor Tool User Manual" section 3 describes the setting > "messaging.hostname" as "Hostname that hosts RabbitMQ Server" but the configuration > file karma-adaptor.properties describes the same setting as the > "". After trying both seems the the URL description > is incorrect, properties file needs update. > > 8. Question, should I be able to process a gush logfile using the ruleset.xml delivered as part of the > adaptor package? I tried parsing a very simple gush log (start and stop gush) with the > provenance_collection.sh script and it fails with "Invalid rulefile, use the correct version of the rulefile!" > > Thanks, > > Luisa > > > > > > -- Scott Jensen, PhD Senior Researcher Data to Insight Center Pervasive Technology Institute Indiana University