wiki:020513_GEMINI_Demo

Version 81 (modified by hmussman@bbn.com, 11 years ago) (diff)

--

Demo on Feb 5, 2013: Progress Towards GEMINI v2(beta) Release

February 5, 2013, 1pm - 4pm

Via webex

Agenda, with notes:

Summary of Tasks and Issues

  • Issue 2a: where does Parser service run? in User Workspace, where it has access to certs/credentials? somewhere else, with a copy of credentials? (but, trying to avoid this) But then, how can GN get access to Parser service, since the UW is not necessarily reachable? Option: Parser in UW could push info to GN
  • Task 3a: Script for GN, to archive data from MS and metadata from UNIS (Matt, Ezra, Hussam)
  • Task 3b: Web-interface to initiate the archive script on GN (Hussam, Charles)
  • Issue 3c: Need to define naming conventions, et.c, for storing measurement data and associated metadata, into iRODS, incluidng full path, and decide how to load path into GN from UW. (Harry, Jeanne)
  • Task 4a: View active measurement graphs, from MS on GN (Charles)
  • Task 6b: net probe to gather port info and merge with what's already in unis (Matt)
  • Task 6c: bandwidth probe (Matt)
  • Task 6d: one way ping probe (Matt)
  • Task 6e: present GUI for graphing data pulled from MS, and metadata pulled from UNIS. Jeremey working on it, extending work from GEC15; has to rewrite. Issue with debcurl library. (Jeremy, Matt, Ezra)
  • Task 6f: AA (due in a few days) (Matt)
  • Task 6h: Configuration interface, using new interface (Matt)
  • Task 7a: Usable on InstaGENI racks; also protoGENI? Hussma has tried, and found a problem that throws out GEMINI rspec extension (Hussam)
  • Task 7b: Approach to first tutorial session: all demo; agree? (all)
  • Task 7c: Small-group topics for second tutorial session; what are they? (all)

Introductions

1:00pm

1) Service to service authentication and authorization, using proxy certs

1:08pm
Ezra

GEMINI AA Demo

GEMINI AA Demo

Configuration

  • This is the gec15 demo topology. Four MP nodes in a full mesh and one GN.
  • Slice name: gemslice4
  • Slice URN: urn:publicid:IDN+emulab.net+slice+gemslice4
  • Slice UUID: 58665b24-6b2a-11e2-a39d-001143e453fe

Demo Steps

  • Slice is already fully instrumentized using gdesktop-init.py and gdesktop-instrumentize.py
  • AA-specific steps take place in gdesktop-instrumentize.py
  • An edited version of -instrumentize will be run to demonstrate the AA steps (see workflow below)
  • The UNIS log will be made visible to show the interaction with instrumentize.
  • Once the AA steps have completed, the MS on the GN will be started.
  • One or more BLiPP instances will be started on the MP nodes.
  • A browser (or unis_client) will be used to access metadata on UNIS and relevant data on the MS.
  • Only authorized users will have access via either the user or proxy certificates.

Interfaces and Workflow

Available Features (2/5)

  • UNIS, MS, and BLiPP are secured via PKI
  • UNIS, MS, BLiPP use GEMINI authorization (ABAC slice_admin role) to restrict access to network resource objects
    • Note: MS does not authorize read/write to /data
  • Instrumentize has been updated to generate proxy certificates and ABAC credentials
    • Certificates are automatically copied to nodes in slice
    • Credentials get pushed to UNIS to allow access for services on the nodes
  • RSpec manifest is converted to UNIS format and pushed securely to UNIS service
  • BLiPP service configuration is generated and pushed securely to UNIS

Available Features by GEC16 (3/19)

  • MS authorizes read/write access to /data
  • GENI/GEMINI Desktop support
    • Note: issue is with NSS versus OpenSSL for curl on Fedora images
    • Might be resolved with custom images, or re-compiled packages
  • RSpec Parser to return slice UUID and other information as JSON object

To be resolved by GEC16 (3/19)

  • Improved error handling during instrumentize
  • Improved creddy integration, reduce number of dependencies
  • Try to remove extra passphrase entry during instrumentize
  • Code changes fully merged (UNIS, MS, and BLiPP)

2) Instrumentize script

1:25pm
Ezra and Hussam

Configuration

Slice pre-configured by Hussam.

  • x4 VMs on emulab at UK
  • PCA, PCB, PCC and GN


Summary of demo steps

  • There will be two version of the demo due to state of the code being worked on.
  • Hussam's version of the instrumentize will have all code related to active measurement installation, configuration disabled
  • Ezra's version of the same instrumentation script will have the active measurements with the AA Stuff enabled.

Demo Plan

  • A slice will be pre-created.
    • GN is loaded with a custom image called “GEMINI_GN”
    • All MP Nodes are loaded with a custom image called “GEMINI_MP”
  • Show the scripts that comprise the whole instrumentize process
  • Start the Instrumentize Process
    • Start gdesktop-init.py
    • includes "shell-in-a-box" to permit early ssh into nodes
    • Start gdesktop-instrumentize.py
    • Show data collected on the GN in the form of rrds and HTML Tables.
  • Jump to archiving this data to irods using GSI as listed in Item 6 of this Agenda.

Executing the demo: Hussam & Ezra

Current interfaces


  • Both version of the instrumentation code will talk to the GENIDESKTOP Parser Service to obtain User information, Slice/Slice Credential Information and Manifest information thus making the instrumentation process somewhat independent of the Control Frameworks.
  • NO communication between AMs/SAs and the User's instrumentation code is required since this now the work of the Parser service.
  • A Detailed list of the Parser API and its return values is provided in this document ParserAPI.pdf.
  • Issue 2a: where does Parser service run? in User Workspace, where it has access to certs/credentials? somewhere else, with a copy of credentials? (but, trying to avoid this) But then, how can GN get access to Parser service, since the UW is not necessarily reachable? Option: Parser in UW could push info to GN

Features available now (2/5)

  • All previous passive data collection functionality (INSTOOLS) still available.
  • Drupal CMS to display passive data collected on the GN available
  • Passive data archive to irods using GSI and proxy certs

Features expected to be available by GEC16 (3/19)

  • Convert Parser API backend to use OMNI inorder to make it compatible with different Control Frameworks and AMs
  • Add new Parser API call to allow the parser to store the AM list provided by the user. This is required since OMNI requires that user provide this list or else it polls every AM in its list obtained from the Clearing House, which would slow down the parser.

Issues that need to be resolved before GEC16 (3/19)

  • Deploy and test the complete merged Active and Passive instrumentation code in the User Workspace and from the GeniDesktop.

3) Steps for accessing iRODS from GN via GSI

1:55pm
Hussam, Wesley, Jeanne, Shu

steps

Prerequisites

  • iRODS server is built with GSI
  • Slice owner has an account on the iRODS server
  • Slice owner's account has aua from GENI certificate configured

Prior to setting up the slice, the user must contact the iRODS administrator (via GEMINI mailing list?) and provide their username and GENI certificate identity (give the openssl command). An account will be established for them in the "GEMINI zone" with an aua matching the provided certificate identity.

  1. Instrumentize will generate a proxy certificate from slice owner's GENI certificate. (Wesley will confirm this)
  2. Instrumentize will place proxy pem file in appropriate location on the GN (home directory of user executing the archive server)
  3. Instrumentize will configure irodsEnv for the user executing the archive server.
    1. host/port - set to the KY iRODS server
    2. zone/resource - a predetermined "GEMINI zone"
    3. username - provided by the user during instrumentize - this MUST match the iRODS account
    4. irods directories - determined from username and zone (e.g. /geminiZone/home/username)

iRODS servers with GSI

There are currently a couple of options for iRODS servers that will be used by the GEMINI experimenters.

  1. Continue to use the iRODS server in KY with its own iCAT.
  2. Federate the KY iRODS server with the GIMI iRODS server.
  3. Others?

Demo Plan

  • A slice will be pre-instrumentized.
    • Show the command line for the instrumentize
    • Show the following items that were configured by instrumentize:
      • proxy certificate (show output of 'openssl verify' or 'grid-proxy-info')
      • irodsEnv contents
    • Show the configuration on the iRODS server
      • user account created
      • aua's for the user
      • contents before the archive
  • The archive will be triggered
    • Show the iRODS contents after the archive

Current interfaces

  • Uses proxy certs generated during the instrumentize process and copied onto the GN Node.
  • Only interface available as of today is a command line to be run when logged into the GN Node to archive data.
  • To view the data archived the user can use the icommands from the User Workspace, or can go to the irods Server's web interface to view the data.

Features available now (2/5)

  • Current version archives (stores) passive measurement data (INSTOOLS) and associated metadata, from RRD on GN
  • Does the web-interface on iRODS support GSI authentication to view the data and metadata?
  • Irods account creation is outside the scope of the GEMINI Instrumentize process. The user is expected to have this account before hand.

Features expected to be available by GEC16 (3/19)

  • Task 3a: Script for GN, to archive data from MS and metadata from UNIS (Matt, Ezra, Hussam)
  • Task 3b: Web-interface to initiate the archive script on GN (Hussam, Charles)

Issues that need to be resolved before GEC16 (3/19)

  • Issue 3c: Need to define naming conventions, et.c, for storing measurement data and associated metadata, into iRODS, incluidng full path, and decide how to load path into GN from UW. (Harry, Jeanne)

4) GENI Desktop, initing global node, displaying topology, instrumentize, displaying measurements

2:15pm
UKentucky Charles

Configuration


Demo

  • Logon to GENI Desktop using credentials.
  • See list of existing slices.
  • Visit a new slice.
    • Page layout with map view.
    • Alternate views.
    • SSH to a node.
    • Send command to node(s).
  • Instrumentize a slice.
  • Visit an instrumentized slice.
    • Show passive graphs.
    • Show MC page.

Interfaces

Features available now (2/5)

  • Login with credentials.
  • View list user's slices.
  • View slice's topology. Logical only if no GN.
  • SSH to nodes.
  • Send commands to nodes.
  • View MC pages.
  • View passive graphs.

Features expected to be available by GEC16 (3/19)

  • Transfer files to nodes.
  • Task 4a: View active measurement graphs, from MS on GN (Charles)
  • Refined user interface.

Issues to be resolved before GEC16 (3/19)

  • caching for quicker page view
  • more logical, cleaner, friendlier menu

5) Configuring GEMINI services, from GUI in GN, to UNIS, to services

2:45pm
Ahmed, Matt, Ezra

Blipp Components

+ configuration (figure)

Configuration Workflow

+ summary of demo steps

No demo is shown, because the configuration web interface was blocked on getting the BLiPP configuration format finalized.

+ current interfaces to other entities, and message flows (like “AA-workflow”)

perfAdmin has already AA integrated into it, little modifications are needed to make it work with the new AA mechanism.

+ features available now (2/5)

The old perfAdmin, no need to demo it again

+ features expected to be available by GEC16 (3/19)

Ability to configure BLiPP by adding new Probes, deleting Probes or modifying existing Probes.

+ issues that need to be resolved before GEC16 (3/19)

6) Measurements, BLiPP in nodes, to MS in GN, to GUI on GN

3:00pm
Matt

Blipp Components

Blipp Data Flow

Configuration Workflow

Summary of demo steps

Depends on what sort of interfaces we have. The plan is to port the existing perfAdmin GUI to work with the new configuration in the new UNIS. If that happens we can do a similar demo to last time as far as how it looks. All the names of different tools should look a bit more streamlined since everything is going through blipp, and there is only one measurement store, but since most of the work has been on generalizing the backend code, the demo shouldn't change much.

Now that the configuration structure has mostly settled, we can start working on interfaces to configure BLiPP. If we aren't able to port in time, the old tools are still available. It should also be pretty easy to write a quick client tool that can get and post new config from and to UNIS.

Current interfaces to other entities, and message flows (like “AA-workflow”)

BLiPP interacts with UNIS and the MS through their well defined APIs, it doesn't expose any interface itself, although this might be something to think about in the future so that it can be commanded to reload config rather than waiting for it to poll UNIS. See figures above for details.

Blipp does reload it's config file at the same time it reloads config from UNIS, so it can be reconfigured that way, but anything that's already in UNIS will override changes in the file.

Features available now (2/5)

+ generic scheduling
+ passive probes

+ cpu
+ memory
+ network

+ active probes

+ ping

+ more unit tests
+ measurement aggregation
+ flexible, cascading configuration structure

Features expected to be available by GEC16 (3/19)

  • metadata reuse, either from service summary, or caching locally (Matt)
  • Task 6b: net probe to gather port info and merge with what's already in unis (Matt)
  • Task 6c: bandwidth probe (Matt)
  • Task 6d: one way ping probe (Matt)
  • Task 6e: present GUI for graphing data pulled from MS, and metadata pulled from UNIS. Jeremey working on it, extending work from GEC15; has to rewrite. Issue with debcurl library. (Jeremy, Matt, Ezra)

Issues that need to be resolved before GEC16 (3/19)

  • Task 6f: AA (due in a few days) (Matt)
  • Many more tests, for stable software (Matt)
  • Task 6h: Configuration interface, using new interface (Matt)

7) GEMINI v2(beta) Configuration for GEC16




Features needed by GEC16 (3/19)

3:30pm
discussion

+ Image for GN; need (soon) all necessary packages

+ Image for MP nodes; need all necessary packages, plus will pull from Python at load time

+ Basic host measurements, using BLiPP

+ Some active network measurements, using BLiPP

Issues that need to be resolved before GEC16 (3/19)

3:45pm
discussion

  • Task 7a: Usable on InstaGENI racks; also protoGENI? Hussma has tried, and found a problem that throws out GEMINI rspec extension (Hussam)
  • Task 7b: Approach to first tutorial session: all demo (agree?) (all)
  • Task 7c: Small-group topics for second tutorial session (all)

Attachments (17)