= Provisioning Service Evaluation = The [http://raven.cs.arizona.edu/projects/project Raven Provisioning Service] project provides a suite of tools: * Raven - Meta-tool that provides combined interface to Stork, Tempest, and Owl tools. * Stork - Secure package management * Tempest - Collective package management * Owl - Slice monitoring * IFTD - Efficient file transfer * Service Monitor - Monitor services and restart as necessary The Raven version 2.2.14-2652 and the Owl v1.0 packages were evaluated for this effort. Time Frame: This evaluation took place Jyly 20-28, 2010. = Provisioning Service Findings = As Owl was reviewed, information was sufficient to get to the resource and use them. Found it difficult to determine the content of my SFA GID credentials, used the [http://groups.geni.net/geni/attachment/wiki/SliceFedArch/SFA2.0.pdf Slice-Based Federation Architecture Specification] document to determine what this credential is about. Initially Stork Version 2.0b was evaluated, which was deemed incorrect. After discussion with Scott Baker found that the repository web interface is deprecated and that the command line raven tool should be used. Also while installing SFA on an ubuntu system found that I should not use the trunk version, but that the tagged version should be used due to the fact that the SFA trunk uses a different credential and certificate format than the PlanetLab's public SFA server. The Tag version sfa-0.9-14 was used. = Provisioning Service How-to = The GENI Integration section at the [http://raven.cs.arizona.edu/projects/project Raven] page states that there are 2 GENI Integration tasks done: * Authenticating using SFA credentials - The Raven repository has been updated to allow logging in via SFA (Cluster B) GIDs and/or Credentials instead of using a !PlanetLab account. The supported format is the GENI Cluster B GID or Credential file. This support has been developed in conjunction with Princeton's Geniwrapper. * Owl Support on SeattleGENI - The Owl client has been ported to the Seattle Million Node Geni project. == Raven tool Authentication using SFA Credentials How-to == The Raven tool is listed as tested on Fedora 8 and requiring: Python 2.5 or newer, with Python libraries: pyOpenSSL, PyXML, and M2Crypto, also required is rpm-build package. Instruction from the [http://raven.cs.arizona.edu/projects/project/wiki/RavenPage Raven Enduser Tool] page were used to executed the steps captured below. 1. The SFA should be installed which can be achieved in two ways. 1a. '''SFA install from SVN check out''' {{{ $ svn co http://svn.planet-lab.org/svn/sfa/tags/sfa-0.9-14 sfa-0.9-14 $ cd sfa-0.9-14/ $ sudo make instal }}} SFA required two additionall packages to be installed: * libxml2: {{{ $ git clone git://git.gnome.org/libxml2 $ cd libxml2 $ ./autogen $ make $ sudo make install }}} * libxslt-1.1.22 downloaded from [from http://www.at.linuxfromscratch.org/blfs/view/6.3/general/libxslt.html libxslt] site. {{{ $ tar xvzf libxslt-1.1.22.tar.gz $ cd libxslt-1.1.22 $ ./configure --prefix=/usr $ make && sudo make install }}} 1b. '''SFA install with yum''' Edit the vi /etc/yum.repos.d/myplc.repo, which should point to the latest RPMS: {{{ [myplc] name=MyPLC baseurl=http://build.planet-lab.org/planetlab/f8/planetlab-f8-i386-5.0-rc14/RPMS/ enabled=1 gpgcheck=0 }}} Install the sfa and sfa-client packages: {{{ $ sudo yum install sfa sfa-client }}} 2. Created an ~/.sfi/sfi_config containing: {{{ SFI_AUTH='plc.bbn' SFI_USER='plc.bbn.lnevers' SFI_REGISTRY='http://www.planet-lab.org:12345/' SFI_SM='http://www.planet-lab.org:12347/' SFI_GENI_AM='http://www.planet-lab.org:12348' }}} 3. Copy RSA private key to ~/.sfi directory: {{{ $ cp ~/.ssh/id_rsa ~/.sfi/lnevers.pkey }}} 4. Get Planet Lab Central listing to verify that settings are correct: {{{ $ sfi.py list plc.bbn $ sfi.py show plc.bbn }}} The above command will generate 2 additional files in the ~/.sfi directory: ''.cert'' and ''.cred'' 5. Using the files ''raven'' and ''arizona-lib'' from the [http://raven.cs.arizona.edu/projects/project/wiki/RavenDownload Raven Software Download] area and installed packages: {{{ $ sudo rpm --install arizona-lib-2.2.14-2652.noarch.rpm raven-2.2.14-2652.noarch.rpm }}} The above creates a ''/usr/local/raven'' and places a binary in ''/usr/bin/''. 6. Created a Raven Experiment: {{{ $ mkdir ~/Provisioning/raven_experiment $ cd ~/Provisioning/raven_experiment/ $ raven create Experiment name: [None] ? ln_raven_experiment Location of your private key: [None] ? ~/.sfi/lnevers.pkey A GENI Credential file may be used to automatically upload files to the Raven repository. This file is optional, but without it you will be responsible for manually uploading the files. Location of GENI cred file: [None] ? ~/.sfi/lnevers.cred Raven may be configured to manage the config files on your slices for you. You may enter multiple slice names separated by commas. Enterning no slice names will cause packages and tpfiles to be uploaded, but not slice configuration files. Slices that should be managed by this experiment: [] ? bbn_gusheval The packages.pacman file controls which packages will be installed on your nodes. This tool can be configured to automatically manage this file, by installing all of your packages on all of your nodes. Automatically manage your packages.pacman (y/n): [y] ? $ }}} The above populates a raven.conf file: {{{ [packagerules] noinstall = [container] version = 1 [experiment] name = ln_raven_experiment slices = bbn_gusheval [manage_packages] upgrade_owld = True manage_packages = True upgrade_stork = True [identity] credname = /home/lnevers/.sfi/lnevers.cred privatekeyname = /home/lnevers/.sfi/lnevers.pkey [dir] uploaddir = upload userdir = users packagedir = packages builderdir = build tempestdir = tempest configdir = config }}} 7. Created a tar package in the ''raven_experiment'' directory, using instructions from [http://raven.cs.arizona.edu/projects/project/wiki/RavenTarPackage Raven Package Instructions] page: {{{ $ mkdir helloworld $ echo "date >> /tmp/helloworld.txt" > helloworld/autorun.sh $ tar -czf helloworld-1.0.tar.gz helloworld $ cp helloworld-1.0.tar.gz packages/. }}} The package must be published, from the experiment directory: {{{ $ cd raven_experiment $ raven publish building: helloworld2 version incremented to 0.1 built: ./packages/helloworld2-0.1-0.noarch.rpm trusting: helloworld-1.0.tar.gz trusting: helloworld2-0.1-0.noarch.rpm trusting user: stork trusting user: fedora8 adding to packages.pacman: helloworld adding to packages.pacman: helloworld2 building: ln_raven_experiment_ravenconfig RPM version incremented to 0.1 re-building tpfiles due to change in tempest rpm trusting: helloworld-1.0.tar.gz trusting: helloworld2-0.1-0.noarch.rpm trusting: ln_raven_experiment_ravenconfig-0.1-0.noarch.rpm trusting user: stork trusting user: fedora8 adding to packages.pacman: helloworld adding to packages.pacman: helloworld2 building: ln_raven_experiment_ravenconfig RPM already up-to-date linking: helloworld-1.0.tar.gz linking: helloworld2-0.1-0.noarch.rpm linking: ln_raven_experiment_ravenconfig-0.1-0.noarch.rpm signing: ln_raven_experiment.tpfile signing: bbn_gusheval.stork.conf repository: https://stork-repository.cs.arizona.edu/REPOAPI/ uploading: bbn_gusheval.27fcf8b05f7cbbedcd5ca6bd2ba63a683d779d5d.stork.conf True uploading: ln_raven_experiment_ravenconfig-0.1-0.noarch.rpm True uploading: helloworld2-0.1-0.noarch.rpm True uploading: helloworld-1.0.tar.gz True uploading: ln_raven_experiment.27fcf8b05f7cbbedcd5ca6bd2ba63a683d779d5d.tpfile True $ }}} 8. Accessed the node via gush to verify that package raven_experiment is being run: {{{ $ ./gush -P 15555 here gush> Gush has learned about the slice bbn_gusheval. gush> connect node2.lbnl.nodes.planet-lab.org Connecting to host bbn_gusheval@node2.lbnl.nodes.planet-lab.org:61414. bbn_gusheval@node2.lbnl.nodes.planet-lab.org:61414 has joined the mesh. gush> shell "ps -eaf|grep raven|egrep -v grep" bbn_gusheval@nis-planet2.doshisha.ac.jp:61414,7493: root 7410 7371 28 14:48 ? 00:00:00 python /usr/bin/stork --upgrade ln_raven_experiment_ravenconfig }}} 9. It is also possible to install external packages by referencing their name. The package sources that are included by default a Stork and Fedora 8 distribution packages. In this next step two packages are installed from the Fedora 8 distribution: {{{ $ cd raven_experiment $ echo > packages/emacs.name $ echo > packages/java.name $ raven publish building: helloworld2 already current: ./packages/helloworld2-0.1-0.noarch.rpm trusting: helloworld-1.0.tar.gz trusting: helloworld2-0.1-0.noarch.rpm trusting: ln_raven_experiment_ravenconfig-0.1-0.noarch.rpm trusting user: stork trusting user: fedora8 adding to packages.pacman: emacs adding to packages.pacman: helloworld adding to packages.pacman: helloworld2 adding to packages.pacman: java building: ln_raven_experiment_ravenconfig RPM version incremented to 0.2 re-building tpfiles due to change in tempest rpm trusting: helloworld-1.0.tar.gz trusting: helloworld2-0.1-0.noarch.rpm trusting: ln_raven_experiment_ravenconfig-0.2-0.noarch.rpm trusting user: stork trusting user: fedora8 adding to packages.pacman: emacs adding to packages.pacman: helloworld adding to packages.pacman: helloworld2 adding to packages.pacman: java building: ln_raven_experiment_ravenconfig RPM already up-to-date linking: helloworld-1.0.tar.gz linking: helloworld2-0.1-0.noarch.rpm linking: ln_raven_experiment_ravenconfig-0.2-0.noarch.rpm signing: ln_raven_experiment.tpfile signing: bbn_gusheval.stork.conf repository: https://stork-repository.cs.arizona.edu/REPOAPI/ uploading: ln_raven_experiment_ravenconfig-0.2-0.noarch.rpm True uploading: bbn_gusheval.27fcf8b05f7cbbedcd5ca6bd2ba63a683d779d5d.stork.conf True already-uploaded: helloworld2-0.1-0.noarch.rpm already-uploaded: helloworld-1.0.tar.gz uploading: ln_raven_experiment.27fcf8b05f7cbbedcd5ca6bd2ba63a683d779d5d.tpfile True }}} It is also possible to see stork tags in the PlanetLab Slice details: [[Image(2010-07-28_Provisioning-raven-tags.jpg)]] == OWL Support on SeattleGENI How-to == Pre-requisites: 1. Get an account at [https://seattlegeni.cs.washington.edu/geni/ SeattleGENI] portal. 2. Request key generation as part of the account creation. 3. Download userid.privatekey and userid.publickey and place keys in seatlleGENI install directory. 4. Login into [https://seattlegeni.cs.washington.edu/geni/html/login?next=/geni/html/profile Seattle GENI] 5. Request resources at [https://seattlegeni.cs.washington.edu/geni/html/myvessels My Vessels] Once the above steps are completed you may install and use the Owl slice monitoring package. Installed the [http://raven.cs.arizona.edu/projects/project/raw-attachment/wiki/OwlSeattle/owl-seattle-1.0.tar.gz OwlSeattle package] as instructed in the [http://raven.cs.arizona.edu/projects/project/wiki/OwlSeattle Owl support for Seattle] page: 1. Unpacked the Owl package in the same directory of the MillionNode Seattle GENI. 2. Used the repypp.py tool to create a preprocessed code file named owltest.pp.repy: {{{ $python ./repypp.py owltest.repy owltest.pp.repy }}} Set up key for seash which supports the following options: {{{ A target can be either a host:port:vesselname, %ID, or a group name. on target [command] -- Runs a command on a target (or changes the default) as keyname [command]-- Run a command using an identity (or changes the default). add [target] [to group] -- Adds a target to a new or existing group remove [target] [from group] -- Removes a target from a group show -- Displays shell state (use 'help show' for more info) set -- Changes the state of the targets (use 'help set') browse -- Find vessels I can control genkeys fn [len] [as identity] -- creates new pub / priv keys (default len=1024) loadkeys fn [as identity] -- loads filename.publickey and filename.privatekey list -- Update and display information about the vessels upload localfn (remotefn) -- Upload a file download remotefn (localfn) -- Download a file delete remotefn -- Delete a file reset -- Reset the vessel (clear the log and files and stop) run file [args ...] -- Shortcut for upload a file and start start file [args ...] -- Start an experiment stop -- Stop an experiment split resourcefn -- Split another vessel off of each vessel join -- Join vessels on the same node help [help | set | show ] -- help information exit -- exits the shell loadstate fn -- Load saved states from a local file. One must call 'loadkeys username' and 'as username' first before loading the states, so seash knows whose RSA keys to use in deciphering the state file. savestate fn -- Save the current state information to a local file. }}} In the following example, 3 vessels(nodes) had been requested from [https://seattlegeni.cs.washington.edu/geni/html/get_resources SeattleGENY->My Vessels] to run the owl script: [[Image(2010-07-23_seattleGENI-1.jpg)]] The Owl script can now be run: {{{ $ python ./seash.py !> genkeys lnevers Created identity 'lnevers' !> loadkeys lnevers !> as lnevers lnevers@ !> browse ['130.161.40.154:1224', '133.9.81.166:1224', '203.178.143.10:1224', '152.3.138.5:1224', '208.117.131.116:1224', '133.9.81.164:1224', '206.117.37.9:1224', '149.169.227.129:1224', '203.30.39.243:1224'] Added targets: %1(149.169.227.129:1224:v6), %3(203.30.39.243:1224:v8), %2(206.117.37.9:1224:v8) Added group 'browsegood' with 3 targets lnevers@ !> on browsegood lnevers@browsegood !> run owltest.pp.repy }}} Various details can be shown within the seash.py interface: {{{ lnevers@browsegood !> update lnevers@browsegood !> show info 149.169.227.129:1224:v6 {'nodekey': {'e': 65537L, 'n': 108700547230965815030281892518836265406880649144319241850548452379387629334687581413313579314495983534078661105889728950154775555961574530604793952431511451091235346680564399627840628247477370016596965218186643920418626097706670807761103530971073488453100773168860730126434481991452870333859427518267898057957L}, 'version': '0.1r', 'nodename': '149.169.227.129'} 206.117.37.9:1224:v8 {'nodekey': {'e': 65537L, 'n': 111036927216391924654743705931909443359542286095079239170551986946721053435455525436183696732328791173792811118797954615349469544359991891203741636400610278508657796796074694320840320112973045053175565741280334677219128496811085304825114209253189982049737395182708371542628957434089203105688361256978110320889L}, 'version': '0.1r', 'nodename': '206.117.37.9'} 203.30.39.243:1224:v8 {'nodekey': {'e': 65537L, 'n': 99870974570855030753474234333944335808140240269544700795681203201656260374719692200619947292539967608982021680255874385683566208312961321822579665850598146823642597999046217853884862228809734080873008121222523381717514460448115073333642563631881087440881392384369718908915671660586118345893265826964714853589L}, 'version': '0.1r', 'nodename': '203.30.39.243'} lnevers@browsegood !> list ID Own Name Status Owner Information %1 149.169.227.129:1224:v6 Started %2 206.117.37.9:1224:v8 Started %3 203.30.39.243:1224:v8 Started lnevers@browsegood !> show hostname 149.169.227.129 is known as planetlab1.eas.asu.edu 206.117.37.9 has unknown host information 203.30.39.243 has unknown host information lnevers@browsegood !> show location %1(149.169.227.129): Tempe, AZ, United States %2(206.117.37.9): Pasadena, CA, United States %3(203.30.39.243): Singapore, Singapore lnevers@browsegood !> show files Files on '206.117.37.9:1224:v8': 'owltest.pp.repy' Files on '149.169.227.129:1224:v6': 'owltest.pp.repy' Files on '203.30.39.243:1224:v8': 'owltest.pp.repy' lnevers@browsegood !> show log Log from '206.117.37.9:1224:v8': owl experiment start calling update calling update calling update calling update Log from '149.169.227.129:1224:v6': owl experiment start calling update calling update calling update calling update Log from '203.30.39.243:1224:v8': owl experiment start calling update calling update calling update calling update }}} Once the experiment is running, you can view the Owl data collected at the [http://owl.cs.arizona.edu/owl_beta/ Owl Slice Monitoring Service] website. At this location you will be presented with a list of databases: [[Image(2010-07-23_Provisioning-raven-owl-1.jpg)]] The database to choose is "seattle": [[Image(2010-07-23_Provisioning-raven-owl-2.jpg)]] and if one of the vessels is chosen: [[Image(2010-07-23_Provisioning-raven-owl-3.jpg)]]