wiki:JOhrenSandbox/GIMINotes8.1.2012

Version 21 (modified by Jeanne Ohren, 12 years ago) (diff)

--

Notes from Post-GEC14 testing of GIMI



Step Status
Establish test/experiment environment Color(green,Success)?
Obtain slice of GENI resources, install and configure GIMI I&M tools Color(green,Success)?
Run and orchestrate GIMI I&M tools and reference/actual experiment Color(green,Success)?
Push I&M measurement results to iRODS archive service Color(green,Success)?
Pull I&M measurement results from iRODS archive service to UW for processing Color(red,Failure)?
Observe I&M measurement results on GIMI presentation service Color(orange,Blocked)?
Release resources, and cleanup Color(green,Success)?

1. Establish test/experiment environment

User Workspace

  • Using the GEC14 Tutorial VM
  • Inputs:
    • GENI certificate
      • installed in $HOME/.ssl
      • location configured in omni_config
      • stripped the passphrase for automation
    • SSH key pair
      • installed in $HOME/.ssh
      • location configured in omni_config
      • key added to ssh-agent
    • iRODS account
      • username configured in $HOME/.irods/.irodsEnv
      • password stored using iinit
  • All credentials configured using credconfig.sh script
  • Outputs:
    • omni.py ready to use
    • i-commands ready to use
    • ready to log into nodes in the slice

iRODS server

  • iRODS server on emmy8 was down so I set up my own on pc41.emulab.net

OML server

  • Set up my own OML server on pc41.emulab.net
  • I was not able to get the oml2-server to use the hook script when started as a service (/etc/init.d/oml2-server start) - only when started on the command line:
    $ oml2-server --listen=3003 --data-dir=/users/johren/gimi --logfile=/users/johren/oml_server.log -H /users/johren/oml2-server-hook.sh
    
  • Configured hook script to use my iRODS server
    ***************
    *** 23,31 ****
      # THE SOFTWARE.
      #
      irodsUserName=rods
    ! irodsHost=emmy8.casa.umass.edu
      irodsPort=1247
    ! irodsZone=geniRenci
      export irodsUserName irodsHost irodsPort irodsZone
      
      # XXX: You might need to initialise the iRODS password for the UNIX user
    --- 23,31 ----
      # THE SOFTWARE.
      #
      irodsUserName=rods
    ! irodsHost=pc41.emulab.net
      irodsPort=1247
    ! irodsZone=bbnZone
      export irodsUserName irodsHost irodsPort irodsZone
      
      # XXX: You might need to initialise the iRODS password for the UNIX user
    
  • Configured /etc/omf-expctl-5.4/omf-expctl.yaml to point to my OML server (can also be specified on the command line)
          # URI to the OML server to use for this EC 
          # (can be overwritten on the EC command line)
          :omluri: 'tcp:pc41.emulab.net:3003'
    

2. Obtain slice of GENI resources, install and configure GIMI I&M tools

  • Used omni to create the slice and the sliver on ExoSM aggregate
  • Rspec includes an execute service for each node that downloads gimi-postboot.sh from the web server (on pc41.emulab.net) and runs it with the slicename and nodeid parameters
        <services>
          <execute command="wget -q -P /tmp http://pc41.emulab.net/gimi-postboot.sh ;chmod +x /tmp/gimi-postboot.sh;/tmp/gimi-postboot.sh %SLICENAME% NodeB > /tmp/gimi-postboot.log" shell="/bin/sh"/>
        </services>
    
  • Inputs:
    • rspec template - %SLICENAME% must be replaced by the slice name when using this script
  • Output:
    • slice manifest
    • slice login information
    • all nodes are ready to run the experiment
      • hostname is set
      • oml2-iperf installed
      • RC is configured and running

3. Run and orchestrate GIMI I&M tools and reference/actual experiment

  • Run the tcp_iperf.rb script using the EC on the UW VM
    $ omf-5.4 exec --no-cmc -S johGIM1208011452 tcp_iperf.rb -- --source1 johGIM1208011452-NodeA --sink johGIM1208011452-NodeC
    
  • Inputs:
    • slicename
    • tcp_iperf.rb
  • Outputs:
    • sq3 database file on the OML server

4. Push I&M measurement results to iRODS archive service

  • This is done by the oml2-server-hook.sh script
  • I had to do a lot of modification to this script because it is very tailored to the OML server configuration
    • Pulls filename and username from the path of the file so it is very dependent upon the filesystem configuration of the server (e.g. home directories are in /users instead of /home)
      !                                         NAME=${DBFILE:12:6};
      !                                         FILE=${DBFILE:12};
      -----------
      !                                         NAME=${DBFILE:7:6};
      !                                         FILE=${DBFILE:19};
      
    • Had to make some changes to the ichmod commands to get it to work with my iRODS server
      !                                             ichmod -M own $NAME /geniRenci/home/$NAME/$FILE
      !                                             iput -f ${DBFILE} /geniRenci/home/$NAME/$FILE;
      !                                             imeta add -d /geniRenci/home/$NAME/$FILE Date ${DATE}
      !                                             imeta add -d /geniRenci/home/$NAME/$FILE UserName $NAME
      !                                             imeta add -d /geniRenci/home/$NAME/$FILE SliceName $SLICE
      --------
      !                                             ichmod -M own rods /bbnZone/home/$NAME 
      !                                             iput -f ${DBFILE} /bbnZone/home/$NAME/$FILE;
      !                                             ichmod -M own $NAME /bbnZone/home/$NAME/$FILE 
      !                                             imeta add -d /bbnZone/home/$NAME/$FILE Date ${DATE}
      !                                             imeta add -d /bbnZone/home/$NAME/$FILE UserName $NAME
      !                                             imeta add -d /bbnZone/home/$NAME/$FILE SliceName $SLICE
      
  • The hook script currently expects the usernames to begin with "gimi". I had to modify my script to look specifically for my username. This will not currently work on emmy9 when experimenters use their own credentials
    !                                           if [[ $NAME =~ ^gimi[0-9] ]]; then
    --------
    !                                           if [[ $NAME =~ ^johren ]]; then
    
  • Inputs:
    • oml2-server-hook script running on the OML server
    • sq3 database file on the OML server
  • Outputs:
    • sq3 database file in experimenter's home location on the iRODS server
    • experimenter is able to iget the sq3 file

5. Pull I&M measurement results from iRODS archive service to UW/iREEL

  • Ran the tutorial_viz.sh command to pull the results from iRODS and run the R script on it to create the images
  • The R script is currently failing and, therefore, not producing the graphs:
    > mydata1 <- dbGetQuery(con, "select oml_sender_id,begin_interval,size,end_interval from iperf_transfer where oml_sender_id=1")
    > intervals1 <- mydata1$end_interval - mydata1$begin_interval
    > throughput1 <- abs(mydata1$size)/intervals1/1024/1024*8
    Error in abs(mydata1$size) :
      Non-numeric argument to mathematical function
    Execution halted
    spawn sudo cp johGIM1208011452.png /var/www/
    [sudo] password for geniuser:
    cp: cannot stat `johGIM1208011452.png': No such file or directory 
    
  • Inputs:
    • sq3 file is accessible in the experimenter's home location on the iRODS server
  • Outputs:
    • visualization images available on the web server on the UW

6. Observe I&M measurement results on GIMI presentation service

  • Currently blocked on failure in step 5.
  • Inputs:
    • None
  • Outputs:
    • visualization images viewable in the web server on the UW

7. Release resources, and cleanup

  • Used omni to delete the sliver
  • When should the data on the OML server be cleaned up?
  • Inputs:
    • slice name
  • Outputs:
    • sliver has been deleted

Release Management concerns

As GENI experimenters and operators start to use GIMI and the code evolves, there are some release management issues that we should consider.

  1. Code Delivery
    • How will updates be delivered to the end users?
      • Packages? Tarballs? Repository?
    • What should be included in these releases?
    • Where these be published?
  2. Versions
    • How will releases be named?
    • As users report problems and we create tickets, we will need to identify what version of the software was used so the issue can be reproduced and troubleshooted.
    • What versions do we need to track? What dependencies do we have? (e.g. OML server 2.8)
  3. Image Updates
    • How often will new static images be created?
    • How will updates be handled between static image releases? Postboot scripts?
    • How does the user identify which image they are running? Can they tell from within the running node?

Attachments (1)

Download all attachments as: .zip