| 1 | [[pageoutline]] |
| 2 | |
| 3 | |
| 4 | = GIMI tasks, issues, next steps = |
| 5 | After discussion on 4/5/12: |
| 6 | After iRODS review with Shu on 4/12/12: |
| 7 | After status call with Mike and Max on 4/18/12: |
| 8 | After status call on 4/25/12: |
| 9 | |
| 10 | == 1) Status calls and reviews == |
| 11 | on 4/25: when is next status call? 5/2 |
| 12 | on 4/25: when is design review? not 5/2; week of 5/7? week of 5/14? |
| 13 | *** marks items that will be important to review on the next call |
| 14 | 3) *** Architecture of GIMI tools for GEC14 tutorial |
| 15 | 4) *** Introduce XML messaging service: required for OMF messages, used to orchestrate services |
| 16 | 7) *** Steps towards GIMI testing: start by forwarding all code and plans for GEC13 demo; by 4/25? |
| 17 | 8) *** Steps towards GIMI tutorial at GEC14 |
| 18 | |
| 19 | |
| 20 | == 2) Target aggregates for GEC14 tutorial == |
| 21 | |
| 22 | a) (Ilia) exoGENI racks; all VMs (Ilia) |
| 23 | |
| 24 | b) ( Ilia) access to shell on all nodes by public IP address; authentication/authorization using keys |
| 25 | |
| 26 | c) ( Ilia) access to web interfaces (GUIs) on all nodes by public IP address; authentication/authorization using keys? certificates? |
| 27 | |
| 28 | |
| 29 | |
| 30 | == 3) *** Architecture of GIMI tools for GEC14 tutorial == |
| 31 | |
| 32 | a) (Mike) need to arrive at nominal configuration ASAP |
| 33 | on 4/18: good discussion (see below); still need a clear and complete description soon; when? |
| 34 | on 4/25: discussion as noted below; still need a clear and complete description soon, and a full design review; when? |
| 35 | |
| 36 | a') on 4/20: issue: how to configure MPs/MCs? from EC to RC in MP/MC? |
| 37 | |
| 38 | a'') on 4/20: issue: is there a MAP on common node? at least to preview data? is there a MAP in user workspace? what? |
| 39 | on 4/25: for 1st, no MAP on common node |
| 40 | |
| 41 | b) MPs with multiple possible measurements: host info; basic network performance with iperf, ping; data from app |
| 42 | on 4/25: need to list baseline measurements, and configurable parameters |
| 43 | |
| 44 | c) MC (OML server) in common node (node n+1) |
| 45 | |
| 46 | d) where is MAP service installed? in common node? in user workspace? what is planned for this service? how can analysis/presentation be configured? is there an r script? is there a GUI? how is GUI accessed? |
| 47 | on 4/25: for 1st, expect MAP only bin user workspace |
| 48 | on 4/25: what is 1st MAP service? what functions are provided? can these be done using iREEL and r? |
| 49 | |
| 50 | e) movement of data from MC (OML server) to MAP service: direct? how? via iRODS, i.e., MC to iRODS to MAP? how? |
| 51 | on 4/18, per Shu, and discussion: could login to MC node; establish first transfer from iRODS client on MC to to iRODS with iput; use iinit, so iRODS client remembers password; then periodic transfers with no human intervention; another push protocol, from MC to iRODS. |
| 52 | on 4/18, per Shu, and discussion: next, can install iRODS client on user workspace, then pull data back to user workspace for analysis and presentation, and then push back to iRODS for archive |
| 53 | |
| 54 | f) 1st: common node in slice; 2nd: common node in user workspace (out of slice) (is this beyond GEC14?) |
| 55 | on 4/25: for 1st, do common node in slice, but plan one per aggregate |
| 56 | |
| 57 | g) 1st: nodes on VMs, one aggregate; 2nd: nodes on VMs, multiple aggregates |
| 58 | |
| 59 | h) 1st: Load image for common node, MP nodes; 2nd: load MP code as package before/after app |
| 60 | on 4/25: what is test app? need to have something visible, e.g., video to a web page? |
| 61 | |
| 62 | i) Is there authentication/authorization when data pushed fom MP to MC? 1st: none; considered OK 2nd: add ssl? |
| 63 | |
| 64 | j) Is there authentication/authorization when accessing GUI (MAP) on common node? how? does this use portal to GUIs? |
| 65 | on 4/25: not in 1st |
| 66 | |
| 67 | == 4) *** Introduce XML messaging service: required for OMF messages, used to orchestrate services == |
| 68 | |
| 69 | a) (Max, Christoph) public XMPP server must be available; what? where? |
| 70 | |
| 71 | b) (Max, Christoph) signed OMF messages: need summary of arrangement from NICTA. what are names? who configures pub/sub nodes? what is structure? how to generate and distribute keys? is a public key registry required? how would it be done? |
| 72 | on 4/18: Max explains plan to use OMF6.0 approach; limited messages; can use XMPP or another transport; signed messages; can start with no authentication/authorization, and then turn it on later; |
| 73 | on 4/18: need clear and complete description soon, particularly naming and establsihment of XMPP nodes. so that others can consider for use and Jeanne can begin testing. |
| 74 | on 4/18: expect release of first OMF6.0 by mid-May, with firm RC and skeleton EC; expect updated EC by mid-June; is this too late for GEC14? |
| 75 | on 4/25: need to start with an earlier version of OMF, e.g., OMF 5.3 |
| 76 | |
| 77 | == 5) Tools in user workspace == |
| 78 | |
| 79 | a) includes stores for keys, certificates, credentials, rspecs |
| 80 | |
| 81 | b) (Jeanne A, David) Use Gush with OMNI to acquire resources, load images/packages, setup connectivity, manage processes |
| 82 | |
| 83 | c) (Christoph) Use OMF EC to configure/orchestrate services; can EC drive scripts? |
| 84 | |
| 85 | d) (Ilia) FLUKES optional to create (or observe) rspec |
| 86 | |
| 87 | e) (Ilia) what is best way to load images? packages? |
| 88 | |
| 89 | f) (Christoph) Add iREEL (experimenter portal service); |
| 90 | on 4/25: what can this do? can it provide MAP functions? can it drive scripts? (need copy to run locally) |
| 91 | |
| 92 | g) (From GEMINI; under discussion) Include portal service to find other GUIs? |
| 93 | on 4/25: for 1st, not necessary |
| 94 | |
| 95 | h) (Giridhar, Jeanne, Shu) Include Directory Archive (DA) service ,which can push data to DOA service, using OI service (all from CNRI, also GPO ) |
| 96 | on 4/25: now replaced with iRODS client |
| 97 | |
| 98 | i) (Giridhar, Jeanne, Shu) Include MDOD creator/editor How does this interface with iRODS, ot push MDOD to iRODS? |
| 99 | on 4/18: who on GIMI project will help define final MDOD schema? Shu. |
| 100 | on 4/25: need to get group together, to revise MDOD schema, and consider implementation of creator/editor |
| 101 | |
| 102 | |
| 103 | |
| 104 | |
| 105 | == 6) Introduce iRODS service == |
| 106 | |
| 107 | After review with Shu on 4/12/12: |
| 108 | |
| 109 | a) (Shu) define configuration of iRODS servers, etc. also logins for users |
| 110 | |
| 111 | b) (Shu) define functions, interfaces to iRODS, and iRODS client can a host mount a file system in iRODS? |
| 112 | |
| 113 | c) (Shu) define naming in iRODS |
| 114 | |
| 115 | d) (Shu) define use case for: moving data from MC to iRODS, then from iRODS to user workspace |
| 116 | use iRODS cleint in MC to push to iRODS? how is authentication/authorization handled? |
| 117 | use iRODS client in user workspace to pull to user workspace? how is authentication/authorization handled? |
| 118 | |
| 119 | On 4/18: use cases received: |
| 120 | |
| 121 | GIMI iRODS Use Cases - Preparation |
| 122 | |
| 123 | Create an account for user: shuang |
| 124 | iadmin mkuser shuang user_type |
| 125 | Assign user to a group |
| 126 | iadmin atg groupName shuang |
| 127 | Admin can assign password to user (also supports GSI/Kerberos) |
| 128 | iadmin moduser shuang password xxx |
| 129 | A lot of other things can be done: |
| 130 | iadmin –h |
| 131 | Create an iRODS directory for the user OML measurement data |
| 132 | imkdir results |
| 133 | |
| 134 | GIMI iRODS Use Case 1 |
| 135 | Perform an OML-based measurement, result is saved in a file (a.sqlite3) under /home/shuang/results |
| 136 | Put the file into iRODS |
| 137 | iput a.sqlite3 (suppose ipwd is /geniRenci/home/shuang/results) |
| 138 | Work on the local copy under /home/shuang/results |
| 139 | Push the modified file back into iRODS (think it as a version control system) |
| 140 | Irsync –r /home/shuang/result i:/geniRenci/home/shuang/results |
| 141 | |
| 142 | GIMI iRODS Use Case 2 |
| 143 | Check out a file in iRODS to my local directory |
| 144 | iget a.sqlite3 |
| 145 | Work on the local copy |
| 146 | Push the modified file back into iRODS |
| 147 | irsync a.sqlite3 i:a.sqlite3 |
| 148 | |
| 149 | GIMI iRODS Use Case 3 |
| 150 | Suppose there are many files under /home/shuang/data that need to be copied into iRODS, we can mount the local directory directly into iRODS |
| 151 | imkdir /geniRenci/home/shuang/mountpoint |
| 152 | imcoll –m filesystem /home/shuang/data /geniRenci/home/shuang/mountpoint |
| 153 | Work on the local files |
| 154 | Sync with iRODS |
| 155 | Irsync –r /home/shuang/data i:/geniRenci/home/shuang/mountpoint |
| 156 | |
| 157 | GIMI iRODS Use Case 4 |
| 158 | Data processing – e.g. filtering files under /geniRenci/home/shuang |
| 159 | Write a rule called filter.r which filters files and save them under /geniRenci/home/shuang/filtered |
| 160 | irule –F filter.r “*Path=‘/geniRenci/home/shuang’” |
| 161 | |
| 162 | e) (Shu) consider interface to get unique handle; need to get current handle minting service from CNRI |
| 163 | |
| 164 | f) (Shu) consider storing and possibly formulating MDOD in iRODS how? |
| 165 | |
| 166 | g) (Shu) review above with GEMINI team |
| 167 | |
| 168 | |
| 169 | == 7) *** Steps towards GIMI testing == |
| 170 | |
| 171 | a) (Jeanne) Arrange user workspace |
| 172 | |
| 173 | c) (Mike) What is first configuration of tools code? when? who? |
| 174 | on 4/18: plan to forward all code and plans for GEC13 to Jeanne |
| 175 | on 4/18: code leads are David and Delip; will join future calls |
| 176 | |
| 177 | d) (Jeanne) What are first test scripts? when? |
| 178 | on 4/25: need to outline process for using tools in user workspace |
| 179 | on 4/25: need to provide first test plan |
| 180 | |
| 181 | e) (Jeanne) When are first tests? |
| 182 | |
| 183 | |
| 184 | |
| 185 | == 8) *** Steps towards GIMI tutorial at GEC14 == |
| 186 | |
| 187 | a) (Ilia) Start with GEC13 ORCA turorial? |
| 188 | |
| 189 | b) (Mike) Define covered use cases, and review with experiment support team (Mark, Niky) |
| 190 | |
| 191 | c) (Mike) When is first demo? |
| 192 | |
| 193 | d) (Mike) Need to establish high-level schedule: freeze code by end of June? then only testing, tutorial prep? |
| 194 | |
| 195 | |
| 196 | |
| 197 | == 9) Use GIMI in WiMAX site == |
| 198 | |
| 199 | a) (Mike, Cong) Can we do something by GEC14? what? |
| 200 | on 4/25: agree UMass Amherst WiMAX site should be up by GEC14 |
| 201 | on 4/25: agree should have plan to integrate GIMI with WiMAX by GEC14, but does not need to be done |