_GIMI_NextSteps: 042512_GIMI_NextSteps.txt

[[PageOutline]]


= GIMI tasks, issues, next steps =

 - After discussion on 4/5/12:
 - After iRODS review with Shu on 4/12/12:
 - After status call with Mike and Max on 4/18/12:
 - After status call on 4/25/12:

== 1)  Status calls and reviews ==

 - on 4/25:  when is next status call?  5/2
 - on 4/25:  when is design review?  not 5/2;  week of 5/7?  week of 5/14?
 - ***  marks items that will be important to review on the next call
   - 3)  *** Architecture of GIMI tools for GEC14 tutorial
   - 4)  *** Introduce XML messaging service:  required for OMF messages, used to orchestrate services
   - 7)  *** Steps towards GIMI testing:  start by forwarding all code and plans for GEC13 demo;  by 4/25?
   - 8)  *** Steps towards GIMI tutorial at GEC14


== 2)  Target aggregates for GEC14 tutorial ==

a)  (Ilia) exoGENI racks;  all VMs  (Ilia)

b)  ( Ilia)  access to shell on all nodes by public IP address;  authentication/authorization using keys

c)  ( Ilia)  access to web interfaces (GUIs) on all nodes by public IP address;    authentication/authorization using keys?  certificates?  



== 3)  *** Architecture of GIMI tools for GEC14 tutorial  ==

a)  (Mike)  need to arrive at nominal configuration ASAP
 - on 4/18:  good discussion (see below);  still need a clear and complete description soon;  when? 
 - on 4/25:  discussion as noted below;  still need a clear and complete description soon, and a full design review;  when?
 
a')  on 4/20:  issue:  how to configure MPs/MCs?  from EC to RC in  MP/MC?

a!'')  on 4/20:  issue:  is there a MAP on common node?  at least to preview data?  is there a MAP in user workspace?  what?
 - on 4/25:  for 1st, no MAP on common node

b)  MPs with multiple possible measurements:  host info;  basic network performance with iperf, ping;  data from app
 - on 4/25:  need to list baseline measurements, and configurable parameters
 
c)  MC (OML server) in common node (node n+1)

d)  where is MAP service installed?  in common node?    in user workspace?    what is planned for this service?  how can analysis/presentation be configured?  is there an r script?  is there a GUI?  how is GUI accessed?
 - on 4/25:  for 1st, expect MAP only bin user workspace
 - on 4/25:  what is 1st MAP service?  what functions are provided?  can these be done using iREEL and r?

e)  movement of data from MC (OML server) to MAP service:   direct?   how?   via iRODS, i.e., MC to iRODS to MAP?  how?
 - on 4/18,  per Shu, and discussion:  could login to MC node;  establish first transfer from iRODS client on MC to to iRODS with iput;  use  iinit, so iRODS client remembers password;  then periodic transfers with no human intervention;  another push protocol, from MC to iRODS.
 - on 4/18,  per Shu, and discussion: next, can install iRODS client on user workspace, then pull data back to user workspace for analysis and presentation, and then push back to iRODS for archive
 
f)  1st:  common node in slice;  2nd:  common node in user workspace (out of slice)  (is this beyond GEC14?)
 - on 4/25:  for 1st, do common node in slice, but plan one per aggregate
 
g)  1st:  nodes on VMs, one aggregate;  2nd:  nodes on VMs, multiple aggregates

h)  1st:  Load image for common node, MP nodes;  2nd:  load MP code as package before/after app
 - on 4/25:  what is test app?  need to have something visible, e.g., video to a web page?

i)  Is there authentication/authorization when data pushed fom MP to MC?  1st:  none;  considered OK  2nd:  add ssl?

j)  Is there authentication/authorization when accessing GUI (MAP) on common node?  how?   does this use portal to GUIs?
 - on 4/25:  not in 1st

== 4)  *** Introduce XML messaging service:  required for OMF messages, used to orchestrate services ==

a)  (Max, Christoph)  public XMPP server must be available;  what?  where?

b)  (Max, Christoph)  signed OMF messages:  need summary of arrangement from NICTA.    what are names?  who configures pub/sub nodes?    what is structure?    how to generate and distribute keys?  is a public key registry required?    how would it be done?    
 - on 4/18:  Max explains plan to use OMF6.0 approach;  limited messages;  can use XMPP or another transport;  signed messages;  can start with no authentication/authorization, and then turn it on later;  
 - on 4/18:  need clear and complete description soon, particularly naming and establsihment of XMPP nodes. so that others can consider for use and Jeanne can begin testing.
 - on 4/18:  expect release of first OMF6.0 by mid-May, with firm RC and skeleton EC;  expect updated EC by mid-June;  is this too late for GEC14?
 - on 4/25:  need to start with an earlier version of OMF, e.g., OMF 5.3

== 5)  Tools in user workspace  ==

a)  includes stores for keys, certificates, credentials, rspecs

b)  (Jeanne A, David) Use Gush with OMNI to acquire resources, load images/packages, setup connectivity, manage processes

c)  (Christoph) Use OMF EC to configure/orchestrate services;  can EC drive scripts?

d)  (Ilia) FLUKES optional to create (or observe) rspec

e)  (Ilia)  what is best way to load images?  packages?

f)  (Christoph)  Add iREEL (experimenter portal service);  
 - on 4/25:  what can this do?  can it provide MAP functions?  can it drive scripts?  (need copy to run locally)  

g)  (From GEMINI;  under discussion)  Include portal service to find other GUIs?
 - on 4/25:  for 1st, not necessary  

h)  (Giridhar, Jeanne, Shu)  Include Directory Archive (DA) service ,which can push data to DOA service, using OI service  (all from CNRI, also GPO )  
 - on 4/25:  now replaced with iRODS client
 
i)   (Giridhar, Jeanne, Shu) Include MDOD creator/editor  How does this interface with iRODS, ot push MDOD to iRODS?  
  - on 4/18:  who on GIMI project will help define final MDOD schema?  Shu.
  - on 4/25:  need to get group together, to revise MDOD schema, and consider implementation of creator/editor

  


== 6)  Introduce iRODS service ==

After review with Shu on 4/12/12:

a)  (Shu)  define configuration of iRODS servers, etc.  also logins for users

b)  (Shu)  define functions, interfaces to iRODS, and iRODS client    can a host mount a file system in iRODS?

c)  (Shu)  define naming in iRODS

d) (Shu)  define use case for:  moving data from MC to iRODS, then from iRODS to user workspace    
 - use iRODS cleint in MC to push to iRODS?  how is authentication/authorization handled?
 - use iRODS client in user workspace to pull to user workspace?  how is authentication/authorization handled?

On 4/18:  use cases received:

GIMI iRODS Use Cases - Preparation

 Create an account for user: shuang
  iadmin mkuser shuang user_type
 Assign user to a group
  iadmin atg groupName shuang
 Admin can assign password to user (also supports GSI/Kerberos)
  iadmin moduser shuang password xxx
 A lot of other things can be done:
  iadmin –h
 Create an iRODS directory for the user OML measurement data
  imkdir results
  
GIMI iRODS Use Case 1
 Perform an OML-based measurement, result is saved in a file (a.sqlite3) under /home/shuang/results
 Put the file into iRODS
  iput a.sqlite3 (suppose ipwd is /geniRenci/home/shuang/results)
 Work on the local copy under /home/shuang/results
 Push the modified file back into iRODS (think it as a version control system)
  Irsync –r /home/shuang/result i:/geniRenci/home/shuang/results
  
GIMI iRODS Use Case 2
 Check out a file in iRODS to my local directory
  iget a.sqlite3
 Work on the local copy
 Push the modified file back into iRODS
  irsync a.sqlite3 i:a.sqlite3

GIMI iRODS Use Case 3
 Suppose there are many files under /home/shuang/data that need to be copied into iRODS, we can mount the local directory directly into iRODS
  imkdir /geniRenci/home/shuang/mountpoint
  imcoll –m filesystem /home/shuang/data /geniRenci/home/shuang/mountpoint
 Work on the local files
 Sync with iRODS
  Irsync –r /home/shuang/data i:/geniRenci/home/shuang/mountpoint

GIMI iRODS Use Case 4
 Data processing – e.g. filtering files under /geniRenci/home/shuang
  Write a rule called filter.r which filters files and save them under /geniRenci/home/shuang/filtered
  irule –F filter.r “*Path=‘/geniRenci/home/shuang’”

e)  (Shu)  consider interface to get unique handle;  need to get current handle minting service from CNRI

f)  (Shu)  consider storing and possibly formulating MDOD in iRODS    how?

g)  (Shu)  review above with GEMINI team


== 7)  *** Steps towards GIMI testing ==

a)  (Jeanne) Arrange user workspace 

c)  (Mike)  What is first configuration of tools code?  when?  who?
 - on 4/18:  plan to forward all code and plans for GEC13 to Jeanne
 - on 4/18:  code leads are David and Delip;  will join future calls

d)  (Jeanne)  What are first test scripts?  when?
 - on 4/25:  need to outline process for using tools in user workspace
 - on 4/25:  need to provide first test plan 

e)  (Jeanne)  When are first tests?



== 8)  *** Steps towards GIMI tutorial at GEC14 ==

a)  (Ilia) Start with GEC13 ORCA turorial? 

b)  (Mike)  Define covered use cases, and review with experiment support team (Mark, Niky)

c)  (Mike)  When is first demo?

d)  (Mike)  Need to establish high-level schedule:  freeze code by end of June?  then only testing, tutorial prep? 



== 9)  Use GIMI in WiMAX site  ==

a)  (Mike, Cong)  Can we do something by GEC14?  what?
 - on 4/25:  agree UMass Amherst WiMAX site should be up by GEC14
 - on 4/25:  agree should have plan to integrate GIMI with WiMAX by GEC14, but does not need to be done