wiki:InstMeasTopic_4.8PortalService

T8) GENI User Tools and Services

1) Goals

Work towards Max Ott's vision for experiment support

TridentCom Portal Paper

Vision Slides from GEC11

Testbed Portal System Requirements

Redmine Portal Service




NICTA Lab Notebook Service capabilities
– Wiki: Keep notes with references
– Schedule & monitor runs
– Analysis with integrated R server
– Repository for all artifacts

Provide a way for a GENI user (e.g., experimenter or operator) to access a wide variety of "GENI User Services", where each user service provides an interface (e.g., API or GUI) to the user. Those user services with a GUI (web) interface are often called "portal services".

Together, the "GENI User Services" should provide all of the functions the user needs to setup and run their experiment, then gather, analyze and present the measurement data.

These services should work together via APIs, etc., to streamline the experiment process.

2) Tasks

Based upon the configuration defined below, the implementation is split into:

1) A GENI User Workspace, which is a persistent Linux OS environment dedicated to the user, that serves as a container for multiple user tools

2) Multiple GENI User Tools, where each provides a service with an interface or a "portal" to the user.

Define, prototype, deploy and operate a GENI User Workspace. It can be hosted on a server dedicated to the user, or on a server hosting multiple user workspaces for multiple users.

Gather the various "user tools" that have been implemented to date, and fit into GENI User Workspace Service so that GENI I&M users can begin to conveniently conduct experiments or instrument infrastructure.

Optimize "user tools" and their interfaces to better meet the needs of GENI users (e.g., experimenters and operators).

3) Team

LEAD Jeanne Ohren (GPO)
Jim Griffioen and/or Charles Carpenter (INSTOOLS and GEMINI, U Kentucky)
Max Ott and/or Christoph Dwertmann (NICTA)
Chris Small (NetKarma, IU)
Ahmed El-Hassany (IU)
Giridhar Manepalli (CNRI)
Harry Mussman (GPO)
Vic Thomas (GPO)
Niky Riga (GPO)
Luisa Nevers (GPO)

4) Meetings

Review at GEC13

Summary of current status: (Jeanne Ohren)

slides

5) Issues

6) Uses Cases

User services use cases (Jeannie Ohren, GPO)

– User’s window into GENI
– Conveniently and easily manage GENI resources, I&M data, and experiment results
– A place to control the experiment
– Works for both short/small (weeks-months?) and long/large (months-years?) experiments
– A place to get to necessary GENI experimenter tools such as Flack, GUSH, OMNI
– A place to store and retrieve experiment results and analysis (Lab notebook?)
– MAP (Measurement, Analysis, Presentation)
– Manages moving I&M data to archive (iRODS?) with metadata (MDOD?)
– Allows the public to access published experiment results (or is this a separate interface?)

7) User Workspace Configuration

The current view of the User Services in the User Workspace is shown in the following figure.


Persistent Linux OS environment dedicated to a user:

  • On a server:
    • Dedicated to user, i.e., a desktop server
    • In an organization (e.g., BBN), shared by multiple users in the organization
    • In GENI infrastructure, shared by multiple users
      • How are partitions for different users assigned? using GENI resource assignment process?

– With a login for the user

  • Includes a file system for the user
    – Includes certificate and credential stores for the user
    – Includes various services/tools dedicated to this user, with programmatic and web interfaces
    • With interfaces/APIs between services tools
    • See Testbed Portal System Requirements above for an example
    • Prefer programmatic interface/API to allow coordination between services, and scripts
    • Prefer web interfaces to allow users to view topology, measurement data, events and trends
  • Persistent data available to user includes:
    • Keys, certificates and credentials
    • Experiment scripts
    • Experiment topology descriptions (RSpecs)
    • Measurement data and MDODs (metadata)

Included services/tools/capabilities:

  • User can load desired services/tools, and bind them together
  • Experiment Control services/tools (e.g., GUSH, OMNI, OMF Experiment Controller) used to:
    • Assign resources for application and I&M services
    • Load topology and load images into slice, including those required for I&M
    • Configure and orchestrate/manage experiment applications
    • Configure and orchestrate/manage I&M services
  • Measurement Collection (MC) services, e.g., OML Server
  • Measurement Analysis and Presentation (MAP) services
  • Measurement Data Object Descriptor (MDOD) creation and editing service, including annotation
  • Directory Archive (DA) service, which can bundle directories into an object, and push to Digital Object Archive (DOA) service
  • What specific services/tools will be available?

NOT included services/tools:

  • Shared Digital Object Archive (DOA) service, i.e., iRODS service

8) Current and planned GENI User Tools

a) Gush with OMNI, and OMF EC

For experiment setup and orchestration.

For I&M setup and orchestration.

b) INSTOOLS portal and web services (U Kentucky)

INSTOOLS project wiki

INSTOOLS GEC7 slides

INSTOOLS design document

INSTOOLS tutorial

INSTOOLS evaluation


INSTOOLS Portal service capabilities:

  • Provides a way for experimenter to find all web servers in experimenter's slice

INSTOOLS Web service capabilities:

  • One is resident in each Measurement Controller (MC)
  • Allows experimenter to configure measurements
  • Allows experimenter to view topology and view measured data
  • Allows experimenter to open a command line into any node, using vnc protocol

c) LAMP Periscope services (U Delaware+)

LAMP project wiki

LAMP tutorial

LAMP evaluation


Periscope services capabilities:

  • Slice Overview
    • Configuration Status
    • Registered Services
      • Measurement Tools (Daemons)
      • perfSONAR Services
  • Configuration
    • Enabled Services
    • Clock Synchronization
    • Scheduled Tests
  • Visualization
    • Throughput
    • One-way Latency
    • Ping Latency
    • Host Monitoring

d) Directory Archive (DA), MDOD Creator/Editor (MDODCE), Digital Object Archive (DOA) and Object Identifier (OI) services (CNRI)

Original prototype configuration:

Measurement Data Archive (MDA) service capabilities:

  • The MDA srvc allows users to store, retrieve, browse, search, share and archive measurement data files, including their associated metadata.
  • The MDA srvc is not limited to measurement data, and can be used for other types of files needed by a researcher.
  • The MDA srvc utilizes the mechanisms provided by the CF to authenticate and authorize users.

The prototype MDA services have been implemented by CNRI, using these components:

  • A hosted Linux OS server that includes multiple services
    • One User Workspace (UW) Srvc for each user, a Linux OS environment with a file system, etc.
    • A common Directory Archive (DA) service, which runs in the Linux server has has special privileges so that it can read and write the directories of each user.
  • A separate and common Digital Object Archive (DOA) Srvc.

More details are in design of prototype services

The (approximate) configuration of the prototype services is shown here:


Interfaces to these services:

Programmatic Linux OS interface to User Workspace service:

  • Users have accounts
  • Users can mount file system using SMB (Samba) protocol
  • Users may securely transfer files using SFTP
  • Users can access a secure shell (SSH) to modify permissions and group settings.
  • Users can load code and run processes.

The Directory Archive (DA) service runs on the same server, and has permission to read and write the directories of all users.

Web interface to Directory Archive (DA) service:

  • One or more directories can be designated to become a data object
  • A Measurement Data Object Descriptor (MDOD) (metadata) can be created as a text file, and then stored with the data object
  • Data objects can be searched, browsed and retrieved using web front end
  • Data objects can be found using keywords or timestamps
  • Users may examine metadata for a data object
  • Users may download data or metadata
  • Users may request that an object be sent to the persistent Digital Object Archive (DOA) service; a persistent identifier is returned

Programmatic HTTP interface to Directory Archive (DA) service:

  • Uses may GetID
  • Users may Archive an object
  • Users may retrieve an object or metadata

Web interface to Digital Object Archive (DOA) service:

  • Data objects can be searched, browsed and retrieved using web front end

Programmatic HTTP interface to Digital Object Archive (DOA) service:

  • Uses may GetID
  • Users may retrieve an object or metadata

Web interfaces to prototype DA and DOA services

Current view of configuration:

Several services in the original configuration are gathered together into the "Directory Archive (DA)" service in the User Workspace; among other functions, they define an object, and then push it to the DOA Service.

Also included in the User Workspace is an "MDOD Creator/Editor" service, that interacts with an "Object Identifier (OI)" service to get a unique persistent identifier (e.g., a handle). Several functions of this service were included in the CNRI prototype Digital Object Archive (DOA) service.


A DOA service, and an associated OI service, must be built to match this configuration. At first, the DOA service should be based on the CNRI prototype DOA service, and then it should be based on the iRODS service.

When the DA service pushes an object to the DOA service, an appropriate authorization arrangment must be used. For example, the DA service may have 'special privileges".

The OI service includes a registry between the "handle" of an object and the DOA service where the object is stored.

e) NICTA Lab Notebook services

Testbed Portal System Requirements

Redmine Portal Service




NICTA Lab Notebook Service capabilities
– Wiki: Keep notes with references
– Schedule & monitor runs
– Analysis with integrated R server
– Repository for all artifacts

f) GIMI Portal services

[ GIMI project wiki]


Experiment Control service capabilities:
– Assign resources to slice, including those required for I&M
– Load topology and load images into slice, including those required for I&M

  • Orchestrate (manage) experiment applications (processes)

Measurement Orchestration (MO) service capabilities:
– Orchestrate (manage) I&M services
– Create and edit Measurement Data Object Descriptor (MDOD), including annotation

OML Server and Results service capabilities:
– Collect data (MC function)
– Analyze and Present data (MAP functions)

  • Web interface
  • pdf file output

– Archive or retrieve SQL data object to iRODS archive service

g) GEMINI Portal services

Expected to include some or all of:

  • LAMP Periscope services
  • INSTOOLS Portal service
  • INSTOOLS web service
  • more?
Last modified 12 years ago Last modified on 04/04/12 11:51:31

Attachments (19)