[[PageOutline]] = GIMI tasks, issues, next steps = - After discussion on 4/5/12: - After iRODS review with Shu on 4/12/12: - After status call with Mike and Max on 4/18/12: - After status call on 4/25/12: - After status call on 5/2/12: == 1) Status calls and reviews == - on 5/2: when is next status call? 5/10 (Thurs) - on 5/2: when is design review? Thurs 5/17 in Cambridge? - *** marks items that will be important to review on the next call - +++ marks items for design review - 3) +++ Architecture of GIMI tools for GEC14 tutorial - 4) *** Introduce XML messaging service: required for OMF messages, used to orchestrate services - 7) *** Introduce IREEL service - 8) +++ Steps towards GIMI testing - 9) +++ Steps towards GIMI tutorial at GEC14 == 2) Target aggregates for GEC14 tutorial == a) (Ilia) exoGENI racks; all VMs (Ilia) b) ( Ilia) access to shell on all nodes by public IP address; authentication/authorization using keys c) ( Ilia) access to web interfaces (GUIs) on all nodes by public IP address; authentication/authorization using keys? certificates? == 3) +++ Architecture of GIMI tools for GEC14 tutorial == a) (Mike) need to arrive at nominal configuration ASAP - on 4/18: good discussion (see below); still need a clear and complete description soon; when? - on 4/25: discussion as noted below; still need a clear and complete description soon, and a full design review; when? - on 5/2: discussion about MAP as noted below; design review likely on 5/17. a') on 4/20: issue: how to configure MPs/MCs? from EC to RC in MP/MC? - on 5/2: if use IREEL to provide MAP functions, can 1st be used to configure the MPs/MCs, i.e., use IREEL and EC to configure the GIMI services, as if configuring a complete experiment. - on 5/2: later, 2nd, consider extending IREEL to configure the entire experiment, i.e., interface with Flukes/OMNI, etc., to aquire resources. a!'') on 4/20: issue: is there a MAP on common node? at least to preview data? is there a MAP in user workspace? what? - on 4/25: for 1st, no MAP on common node b) MPs with multiple possible measurements: host info; basic network performance with iperf, ping; data from app - on 4/25: need to list baseline measurements, and configurable parameters c) MC (OML server) in common node (node n+1) d) where is MAP service installed? in common node? in user workspace? what is planned for this service? how can analysis/presentation be configured? is there an r script? is there a GUI? how is GUI accessed? - on 4/25: for 1st, expect MAP only bin user workspace - on 4/25: what is 1st MAP service? what functions are provided? can these be done using iREEL and r? - on 5/2: 1st plan for MAP service: base it on iREEL 2.0, installed as a public service; where installed? not user workspace; not BBN; UMassAmherst? RENCI? NICTA too far away. e) movement of data from MC (OML server) to MAP service: direct? how? via iRODS, i.e., MC to iRODS to MAP? how? - on 4/18, per Shu, and discussion: could login to MC node; establish first transfer from iRODS client on MC to to iRODS with iput; use iinit, so iRODS client remembers password; then periodic transfers with no human intervention; another push protocol, from MC to iRODS. - on 4/18, per Shu, and discussion: next, can install iRODS client on user workspace, then pull data back to user workspace for analysis and presentation, and then push back to iRODS for archive - on 5/2: must be able to pull data from IRODS to IREEL for MAP functions. how? f) 1st: common node in slice; 2nd: common node in user workspace (out of slice) (is this beyond GEC14?) - on 4/25: for 1st, do common node in slice, but plan one per aggregate g) 1st: nodes on VMs, one aggregate; 2nd: nodes on VMs, multiple aggregates h) 1st: Load image for common node, MP nodes; 2nd: load MP code as package before/after app - on 4/25: what is test app? need to have something visible, e.g., video to a web page? i) Is there authentication/authorization when data pushed fom MP to MC? 1st: none; considered OK 2nd: add ssl? j) Is there authentication/authorization when accessing GUI (MAP) on common node? how? does this use portal to GUIs? - on 4/25: not in 1st == 4) *** Introduce XML messaging service: required for OMF messages, used to orchestrate services == a) (Max, Christoph) public XMPP server must be available; what? where? b) (Max, Christoph) signed OMF messages: need summary of arrangement from NICTA. what are names? who configures pub/sub nodes? what is structure? how to generate and distribute keys? is a public key registry required? how would it be done? - on 4/18: Max explains plan to use OMF6.0 approach; limited messages; can use XMPP or another transport; signed messages; can start with no authentication/authorization, and then turn it on later; - on 4/18: need clear and complete description soon, particularly naming and establsihment of XMPP nodes. so that others can consider for use and Jeanne can begin testing. - on 4/18: expect release of first OMF6.0 by mid-May, with firm RC and skeleton EC; expect updated EC by mid-June; is this too late for GEC14? - on 4/25: need to start with an earlier version of OMF, e.g., OMF 5.3 - on 5/2: need plan for using OMF 5.3 (or 5.4). == 5) Tools in user workspace == a) includes stores for keys, certificates, credentials, rspecs b) (Jeanne A, David) Use Gush with OMNI to acquire resources, load images/packages, setup connectivity, manage processes c) (Christoph) Use OMF EC to configure/orchestrate services; can EC drive scripts? d) (Ilia) FLUKES optional to create (or observe) rspec - on 5/2: how does this relate to IREEL, which will be sued to configure GIMI services? e) (Ilia) what is best way to load images? packages? f) (Christoph) Add iREEL (experimenter portal service); - on 4/25: what can this do? can it provide MAP functions? can it drive scripts? (need copy to run locally) - on 5/2: 1st plan for MAP service: base it on iREEL 2.0, installed as a public service; where installed? not user workspace; not BBN; UMassAmherst? RENCI? NICTA too far away. - on 5/2: must be able to pull data from IRODS to IREEL for MAP functions. how? - on 5/2: if use IREEL to provide MAP functions, can 1st be used to configure the MPs/MCs, i.e., use IREEL and EC to configure the GIMI services, as if configuring a complete experiment. - on 5/2: later, 2nd, consider extending IREEL to configure the entire experiment, i.e., interface with Flukes/OMNI, etc., to aquire resources. g) (From GEMINI; under discussion) Include portal service to find other GUIs? - on 4/25: for 1st, not necessary h) (Giridhar, Jeanne, Shu) Include Directory Archive (DA) service ,which can push data to DOA service, using OI service (all from CNRI, also GPO ) - on 4/25: now replaced with iRODS client i) (Giridhar, Jeanne, Shu) Include MDOD creator/editor How does this interface with iRODS, ot push MDOD to iRODS? - on 4/18: who on GIMI project will help define final MDOD schema? Shu. - on 4/25: need to get group together, to revise MDOD schema, and consider implementation of creator/editor == 6) Introduce iRODS service == After review with Shu on 4/12/12: a) (Shu) define configuration of iRODS servers, etc. also logins for users b) (Shu) define functions, interfaces to iRODS, and iRODS client can a host mount a file system in iRODS? c) (Shu) define naming in iRODS d) (Shu) define use case for: moving data from MC to iRODS, then from iRODS to user workspace - use iRODS cleint in MC to push to iRODS? how is authentication/authorization handled? - use iRODS client in user workspace to pull to user workspace? how is authentication/authorization handled? On 4/18: use cases received: GIMI iRODS Use Cases - Preparation Create an account for user: shuang iadmin mkuser shuang user_type Assign user to a group iadmin atg groupName shuang Admin can assign password to user (also supports GSI/Kerberos) iadmin moduser shuang password xxx A lot of other things can be done: iadmin –h Create an iRODS directory for the user OML measurement data imkdir results GIMI iRODS Use Case 1 Perform an OML-based measurement, result is saved in a file (a.sqlite3) under /home/shuang/results Put the file into iRODS iput a.sqlite3 (suppose ipwd is /geniRenci/home/shuang/results) Work on the local copy under /home/shuang/results Push the modified file back into iRODS (think it as a version control system) Irsync –r /home/shuang/result i:/geniRenci/home/shuang/results GIMI iRODS Use Case 2 Check out a file in iRODS to my local directory iget a.sqlite3 Work on the local copy Push the modified file back into iRODS irsync a.sqlite3 i:a.sqlite3 GIMI iRODS Use Case 3 Suppose there are many files under /home/shuang/data that need to be copied into iRODS, we can mount the local directory directly into iRODS imkdir /geniRenci/home/shuang/mountpoint imcoll –m filesystem /home/shuang/data /geniRenci/home/shuang/mountpoint Work on the local files Sync with iRODS Irsync –r /home/shuang/data i:/geniRenci/home/shuang/mountpoint GIMI iRODS Use Case 4 Data processing – e.g. filtering files under /geniRenci/home/shuang Write a rule called filter.r which filters files and save them under /geniRenci/home/shuang/filtered irule –F filter.r “*Path=‘/geniRenci/home/shuang’” e) (Shu) consider interface to get unique handle; need to get current handle minting service from CNRI f) (Shu) consider storing and possibly formulating MDOD in iRODS how? g) (Shu) review above with GEMINI team == 7) ***Introduce IREEL service == (Christoph) Add iREEL (experimenter portal service): - on 4/25: what can this do? can it provide MAP functions? can it drive scripts? (need copy to run locally) - on 5/2: 1st plan for MAP service: base it on iREEL 2.0, installed as a public service; where installed? not user workspace; not BBN; UMassAmherst? RENCI? NICTA too far away. - on 5/2: must be able to pull data from IRODS to IREEL for MAP functions. how? - on 5/2: if use IREEL to provide MAP functions, can 1st be used to configure the MPs/MCs, i.e., use IREEL and EC to configure the GIMI services, as if configuring a complete experiment. - on 5/2: later, 2nd, consider extending IREEL to configure the entire experiment, i.e., interface with Flukes/OMNI, etc., to aquire resources. == 8) +++ Steps towards GIMI testing == a) (Jeanne) Arrange user workspace c) (Mike) What is first configuration of tools code? when? who? - on 4/18: plan to forward all code and plans for GEC13 to Jeanne - on 4/18: code leads are David and Delip; will join future calls - on 5/2: have sent code, scripts from GEC13 demo - on 5/2: (Jeanne) for tracking, have setup process on geni.net d) (Jeanne) What are first test scripts? when? - on 4/25: need to outline process for using tools in user workspace - on 4/25: need to provide first test plan e) (Jeanne) When are first tests? == 9) +++ Steps towards GIMI tutorial at GEC14 == a) (Ilia) Start with GEC13 ORCA turorial? b) (Mike) Define covered use cases, and review with experiment support team (Mark, Niky) c) (Mike) When is first demo? d) (Mike) Need to establish high-level schedule: freeze code by end of June? then only testing, tutorial prep? == 10) Use GIMI in WiMAX site == a) (Mike, Cong) Can we do something by GEC14? what? - on 4/25: agree UMass Amherst WiMAX site should be up by GEC14 - on 4/25: agree should have plan to integrate GIMI with WiMAX by GEC14, but does not need to be done