Version 74 (modified by, 9 years ago) (diff)


Demo on Feb 5, 2013: Progress Towards GEMINI v2(beta) Release

February 5, 2013, 1pm - 4pm

Via webex

Agenda, with notes:



1) Service to service authentication and authorization, using proxy certs



2) Instrumentize script

Ezra and Hussam


Slice pre-configured by Hussam.

  • x4 VMs on emulab at UK
  • PCA, PCB, PCC and GN

Summary of demo steps

  • There will be two version of the demo due to state of the code being worked on.
  • Hussam's version of the instrumentize will have all code related to active measurement installation, configuration disabled
  • Ezra's version of the same instrumentation script will have the active measurements with the AA Stuff enabled.

Demo Plan

  • A slice will be pre-created.
    • GN is loaded with a custom image called “GEMINI_GN”
    • All MP Nodes are loaded with a custom image called “GEMINI_MP”
  • Show the scripts that comprise the whole instrumentize process
  • Start the Instrumentize Process
    • Start
    • includes "shell-in-a-box" to permit early ssh into nodes
    • Start
    • Show data collected on the GN in the form of rrds and HTML Tables.
  • Jump to archiving this data to irods using GSI as listed in Item 6 of this Agenda.

Executing the demo: Hussam & Ezra

Current interfaces

  • Both version of the instrumentation code will talk to the GENIDESKTOP Parser Service to obtain User information, Slice/Slice Credential Information and Manifest information thus making the instrumentation process somewhat independent of the Control Frameworks.
  • NO communication between AMs/SAs and the User's instrumentation code is required since this now the work of the Parser service.
  • A Detailed list of the Parser API and its return values is provided in this document ParserAPI.pdf.

Issue: where does Parser service run? in User Workspace, where it has access to certs/credentials? somewhere else, with a copy of credentials? (but, trying to avoid this)

  • But then, how can GN get access to Parser service, since the UW is not necessarily reachable?
  • Option: Parser in UW could push info to GN

Features available now (2/5)

  • All previous passive data collection functionality (INSTOOLS) still available.
  • Drupal CMS to display passive data collected on the GN available
  • Passive data archive to irods using GSI and proxy certs

Features expected to be available by GEC16 (3/19)

  • Convert Parser API backend to use OMNI inorder to make it compatible with different Control Frameworks and AMs
  • Add new Parser API call to allow the parser to store the AM list provided by the user. This is required since OMNI requires that user provide this list or else it polls every AM in its list obtained from the Clearing House, which would slow down the parser.

Issues that need to be resolved before GEC16 (3/19)

  • Deploy and test the complete merged Active and Passive instrumentation code in the User Workspace and from the GeniDesktop.

3) Steps for accessing iRODS from GN via GSI

Hussam, Wesley, Jeanne, Shu



  • iRODS server is built with GSI
  • Slice owner has an account on the iRODS server
  • Slice owner's account has aua from GENI certificate configured

Prior to setting up the slice, the user must contact the iRODS administrator (via GEMINI mailing list?) and provide their username and GENI certificate identity (give the openssl command). An account will be established for them in the "GEMINI zone" with an aua matching the provided certificate identity.

  1. Instrumentize will generate a proxy certificate from slice owner's GENI certificate. (Wesley will confirm this)
  2. Instrumentize will place proxy pem file in appropriate location on the GN (home directory of user executing the archive server)
  3. Instrumentize will configure irodsEnv for the user executing the archive server.
    1. host/port - set to the KY iRODS server
    2. zone/resource - a predetermined "GEMINI zone"
    3. username - provided by the user during instrumentize - this MUST match the iRODS account
    4. irods directories - determined from username and zone (e.g. /geminiZone/home/username)

iRODS servers with GSI

There are currently a couple of options for iRODS servers that will be used by the GEMINI experimenters.

  1. Continue to use the iRODS server in KY with its own iCAT.
  2. Federate the KY iRODS server with the GIMI iRODS server.
  3. Others?

Demo Plan

  • A slice will be pre-instrumentized.
    • Show the command line for the instrumentize
    • Show the following items that were configured by instrumentize:
      • proxy certificate (show output of 'openssl verify' or 'grid-proxy-info')
      • irodsEnv contents
    • Show the configuration on the iRODS server
      • user account created
      • aua's for the user
      • contents before the archive
  • The archive will be triggered
    • Show the iRODS contents after the archive

Current interfaces

  • Uses proxy certs generated during the instrumentize process and copied onto the GN Node.
  • Only interface available as of today is a command line to be run when logged into the GN Node to archive data.
  • To view the data archived the user can use the icommands from the User Workspace, or can go to the irods Server's web interface to view the data.

Features available now (2/5)

  • Current version archives (stores) passive measurement data (INSTOOLS) and associated metadata, from RRD on GN
  • Does the web-interface on iRODS support GSI authentication to view the data and metadata?
  • Irods account creation is outside the scope of the GEMINI Instrumentize process. The user is expected to have this account before hand.

Features expected to be available by GEC16 (3/19)

  • Script for GN, to archive data from MS and metadata from UNIS
  • Web-interface to initiate the archive script on GN

Issues that need to be resolved before GEC16 (3/19)

  • Need to define naming conventions, et.c, for storing measurement data and associated metadata, into iRODS, incluidng full path, and decide how to load path into GN from UW.

4) GENI Desktop, initing global node, displaying topology, instrumentize, displaying measurements

UKentucky Charles



  • Logon to GENI Desktop using credentials.
  • See list of existing slices.
  • Visit a new slice.
    • Page layout with map view.
    • Alternate views.
    • SSH to a node.
    • Send command to node(s).
  • Instrumentize a slice.
  • Visit an instrumentized slice.
    • Show passive graphs.
    • Show MC page.


Features availabel now (2/5)

  • Login with credentials.
  • View list user's slices.
  • View slice's topology. Logical only if no GN.
  • SSH to nodes.
  • Send commands to nodes.
  • View MC pages.
  • View passive graphs.

Features expected to be available by GEC16 (3/19)

  • Transfer files to nodes.
  • View active measurement graphs, from MS on GN
  • Refined user interface.

Issues to be resolved before GEC16 (3/19)

  • caching for quicker page view
  • more logical, cleaner, friendlier menu

5) Configuring GEMINI services, from GUI in GN, to UNIS, to services

Ahmed, Matt, Ezra

Blipp Components

+ configuration (figure)

Configuration Workflow

+ summary of demo steps

No demo is shown, because the configuration web interface was blocked on getting the BLiPP configuration format finalized.

+ current interfaces to other entities, and message flows (like “AA-workflow”)

perfAdmin has already AA integrated into it, little modifications are needed to make it work with the new AA mechanism.

+ features available now (2/5)

The old perfAdmin, no need to demo it again

+ features expected to be available by GEC16 (3/19)

Ability to configure BLiPP by adding new Probes, deleting Probes or modifying existing Probes.

+ issues that need to be resolved before GEC16 (3/19)

6) Measurements, BLiPP in nodes, to MS in GN, to GUI on GN


Blipp Components

Blipp Data Flow

Configuration Workflow

Summary of demo steps

Depends on what sort of interfaces we have. The plan is to port the existing perfAdmin GUI to work with the new configuration in the new UNIS. If that happens we can do a similar demo to last time as far as how it looks. All the names of different tools should look a bit more streamlined since everything is going through blipp, and there is only one measurement store, but since most of the work has been on generalizing the backend code, the demo shouldn't change much.

Now that the configuration structure has mostly settled, we can start working on interfaces to configure BLiPP. If we aren't able to port in time, the old tools are still available. It should also be pretty easy to write a quick client tool that can get and post new config from and to UNIS.

Current interfaces to other entities, and message flows (like “AA-workflow”)

BLiPP interacts with UNIS and the MS through their well defined APIs, it doesn't expose any interface itself, although this might be something to think about in the future so that it can be commanded to reload config rather than waiting for it to poll UNIS. See figures above for details.

Blipp does reload it's config file at the same time it reloads config from UNIS, so it can be reconfigured that way, but anything that's already in UNIS will override changes in the file.

Features available now (2/5)

+ generic scheduling
+ passive probes

+ cpu
+ memory
+ network

+ active probes

+ ping

+ more unit tests
+ measurement aggregation
+ flexible, cascading configuration structure

Features expected to be available by GEC16 (3/19)

+ metadata reuse, either from service summary, or caching locally
+ net probe to gather port info and merge with what's already in unis
+ bandwidth probe
+ one way ping probe
+ present GUI for graphing data pulled from MS, and metadata pulled from UNIS. Jeremey working on it, extending work from GEC15; has to rewrite. Issue with debcurl library.

Issues that need to be resolved before GEC16 (3/19)

+ AA (due in a few days)
+ Many more tests, for stable software
+ Configuration interface, using new interface

7) GEMINI v2(beta) Configuration for GEC16

Features needed by GEC16 (3/19)


+ Image for GN; need (soon) all necessary packages

+ Image for MP nodes; need all necessary packages, plus will pull from Python at load time

+ Basic host measurements, using BLiPP

+ Some active network measurements, using BLiPP

Issues that need to be resolved before GEC16 (3/19)


+ Usable on InstaGENI racks; also protoGENI? Hussma has tried, and found a problem that throws out GEMINI rspec extension

+ Approach to first tutorial session: all demo (agree?)

+ Small-group topics for second tutorial session

Attachments (17)