GIMI tasks, issues, next steps: After discussion on 4/5/12: After iRODS review with Shu on 4/12/12: After status call with Mike and Max on 4/18/12: 1. When is next status call? 5/2; but reserving 4/25 for other topics. *** marks items that will be important to review on the next call 3. *** Architecture of GIMI tools for GEC14 tutorial; still need a clear and complete description soon; when? 5/2? earlier? various issues 4. *** Introduce XML messaging service: required for OMF messages, used to orchestrate services; expect release of first OMF6.0 by mid-May, with firm RC and skeleton EC; expect updated EC by mid-June; is this too late for GEC14? 7) *** Steps towards GIMI testing: start by forwarding all code and plans for GEC13 demo; by 4/25? 8) *** Steps towards GIMI tutorial at GEC14: need to establish high-level schedule; need to define and review use cases; by 5/2? 2. Target aggregates for GEC14 tutorial a) (Ilia) exoGENI racks; all VMs (Ilia) b) ( Ilia) access to shell on all nodes by public IP address; authentication/authorization using keys c) ( Ilia) access to web interfaces (GUIs) on all nodes by public IP address; authentication/authorization using keys? certificates? 3. *** Architecture of GIMI tools for GEC14 tutorial a) (Mike) need to arrive at nominal configuration ASAP on 4/18: good discussion (see below); still need a clear and complete description soon; when? 5/2? earlier? a') on 4/20: issue: how to configure tools? from EC to RC in tool? a'') on 4/20: issue: is there a MAP on common node? at least to preview data? is there a MAP in user workspace? what? b) MPs with multiple possible measurements: host info; basic network performance with iperf, ping; data from app c) MC (OML server) in common node (node n+1) d) where is MAP service installed? in common node? in user workspace? what is planned for this service? how can analysis/presentation be configured? is there an r script? is there a GUI? how is GUI accessed? e) movement of data from MC (OML server) to MAP service: direct? how? via iRODS, i.e., MC to iRODS to MAP? how? e') on 4/18, per Shu, and discussion: could login to MC node; establish first transfer from iRODS client on MC to to iRODS with iput; use iinit, so iRODS client remembers password; then periodic transfers with no human intervention; another push protocol, from MC to iRODS. e'') on 4/18, per Shu, and discussion: next, can install iRODS client on user workspace, then pull data back to user workspace for analysis and presentation, and then push back to iRODS for archive f) 1st: common node in slice; 2nd: common node in user workspace (out of slice) (is this beyond GEC14?) g) 1st: nodes on VMs, one aggregate; 2nd: nodes on VMs, multiple aggregates h) 1st: Load image for common node, MP nodes; 2nd: load MP code as package before/after app what is test app? i) Is there authentication/authorization when data pushed fom MP to MC? 1st: none; considered OK 2nd: add ssl? j) Is there authentication/authorization when accessing GUI (MAP) on common node? how? does this use portal to GUIs? 4. *** Introduce XML messaging service: required for OMF messages, used to orchestrate services a) (Max, Christoph) public XMPP server must be available; what? where? b) (Max, Christoph) signed OMF messages: need summary of arrangement from NICTA. what are names? who configures pub/sub nodes? what is structure? how to generate and distribute keys? is a public key registry required? how would it be done? On 4/18: Max explains plan to use OMF6.0 approach; limited messages; can use XMPP or another transport; signed messages; can start with no authentication/authorization, and then turn it on later; b') On 4/18: need clear and complete description soon, particularly naming and establsihment of XMPP nodes. so that others can consider for use and Jeanne can begin testing. b'') On 4/18: expect release of first OMF6.0 by mid-May, with firm RC and skeleton EC; expect updated EC by mid-June; is this too late for GEC14? 5) Tools in user workspace a) includes stores for keys, certificates, credentials, rspecs b) (Jeanne A, David) Use Gush with OMNI to acquire resources, load images/packages, setup connectivity, manage processes c) (Christoph) Use OMF EC to configure/orchestrate services can EC drive scripts? d) (Ilia) FLUKES optional to create (or observe) rspec e) (Ilia) what is best way to load images? packages? f) (Christoph) Add iREEL (experimenter portal service); what can this do? can it drive scripts? (need copy to run locally) g) (From GEMINI; under discussion) Include portal service to find other GUIs? h) (Giridhar, Jeanne, Shu) Include Directory Archive (DA) service ,which can push data to DOA service, using OI service (all from CNRI, also GPO ) Or is this replaced with iRODS client? i) (Giridhar, Jeanne, Shu) Include MDOD creator/editor How does this interface with iRODS, ot push MDOD to iRODS? j) (Mike) who on GIMI project will help define final MDOD schema? Shu. 6) Introduce iRODS service After review with Shu on 4/12/12: a) (Shu) define configuraion of iRODS servers, etc. also logins for users b) (Shu) define functions, interfaces to iRODS, and iRODS client can a host mount a file system in iRODS? c) (Shu) define naming in iRODS d) (Shu) define use case for: moving data from MC to iRODS, then from iRODS to user workspace use iRODS cleint in MC to push to iRODS? how is authentication/authorization handled? use iRODS client in user workspace to pull to user workspace? how is authentication/authorization handled? On 4/18: use cases received: GIMI iRODS Use Cases - Preparation Create an account for user: shuang iadmin mkuser shuang user_type Assign user to a group iadmin atg groupName shuang Admin can assign password to user (also supports GSI/Kerberos) iadmin moduser shuang password xxx A lot of other things can be done: iadmin –h Create an iRODS directory for the user OML measurement data imkdir results GIMI iRODS Use Case 1 Perform an OML-based measurement, result is saved in a file (a.sqlite3) under /home/shuang/results Put the file into iRODS iput a.sqlite3 (suppose ipwd is /geniRenci/home/shuang/results) Work on the local copy under /home/shuang/results Push the modified file back into iRODS (think it as a version control system) Irsync –r /home/shuang/result i:/geniRenci/home/shuang/results GIMI iRODS Use Case 2 Check out a file in iRODS to my local directory iget a.sqlite3 Work on the local copy Push the modified file back into iRODS irsync a.sqlite3 i:a.sqlite3 GIMI iRODS Use Case 3 Suppose there are many files under /home/shuang/data that need to be copied into iRODS, we can mount the local directory directly into iRODS imkdir /geniRenci/home/shuang/mountpoint imcoll –m filesystem /home/shuang/data /geniRenci/home/shuang/mountpoint Work on the local files Sync with iRODS Irsync –r /home/shuang/data i:/geniRenci/home/shuang/mountpoint GIMI iRODS Use Case 4 Data processing – e.g. filtering files under /geniRenci/home/shuang Write a rule called filter.r which filters files and save them under /geniRenci/home/shuang/filtered irule –F filter.r “*Path=‘/geniRenci/home/shuang’” e) (Shu) consider interface to get unique handle f) (Shu) consider storing and possibly formulating MDOD in iRODS how? g) (Shu) reivew above with GEMINI team 7) *** Steps towards GIMI testing a) (Jeanne) Arrange user workspace c) (Mike) What is first configuration of tools code? when? who? On 4/18: plan to forward all code and plans for GEC13 to Jeanne On 4/18: code leads are David and Delip; will join future calls d) (Jeanne) What are first test scripts? when? e) (Jeanne) When are first test? 8) *** Steps towards GIMI totorial at GEC14 a) (Ilia) Start with GEC13 ORCA turorial? b) (Mike) Define covered use cases, and review with experiment support team (Mark, Niky) c) (Mike) When is first demo? d) (Mike) Need to establish high-level schedule: freeze code by end of June? then only testing, tutorial prep? 9) Use GIMI in WiMAX site a) (Mike, Cong) Can we do something by GEC14? what?