_GIMI_NextSteps: 050212_GIMI_NextSteps.txt

File 050212_GIMI_NextSteps.txt, 11.8 KB (added by hmussman@bbn.com, 8 years ago)
Line 
1[[PageOutline]]
2
3
4= GIMI tasks, issues, next steps =
5
6 - After discussion on 4/5/12:
7 - After iRODS review with Shu on 4/12/12:
8 - After status call with Mike and Max on 4/18/12:
9 - After status call on 4/25/12:
10 - After status call on 5/2/12:
11
12== 1)  Status calls and reviews ==
13
14 - on 5/2:  when is next status call?  5/10 (Thurs)
15 - on 5/2:  when is design review?  Thurs 5/17 in Cambridge?
16 - ***  marks items that will be important to review on the next call
17 - +++ marks items for design review
18   - 3)  +++ Architecture of GIMI tools for GEC14 tutorial
19   - 4)  *** Introduce XML messaging service:  required for OMF messages, used to orchestrate services
20   - 7)  *** Introduce IREEL service
21   - 8)  +++ Steps towards GIMI testing
22   - 9)  +++ Steps towards GIMI tutorial at GEC14
23
24
25== 2)  Target aggregates for GEC14 tutorial ==
26
27a)  (Ilia) exoGENI racks;  all VMs  (Ilia)
28
29b)  ( Ilia)  access to shell on all nodes by public IP address;  authentication/authorization using keys
30
31c)  ( Ilia)  access to web interfaces (GUIs) on all nodes by public IP address;    authentication/authorization using keys?  certificates? 
32
33
34
35== 3)  +++ Architecture of GIMI tools for GEC14 tutorial  ==
36
37a)  (Mike)  need to arrive at nominal configuration ASAP
38 - on 4/18:  good discussion (see below);  still need a clear and complete description soon;  when?
39 - on 4/25:  discussion as noted below;  still need a clear and complete description soon, and a full design review;  when?
40 - on 5/2:  discussion about MAP as noted below;  design review likely on 5/17.
41 
42a')  on 4/20:  issue:  how to configure MPs/MCs?  from EC to RC in  MP/MC?
43 - on 5/2:  if use IREEL to provide MAP functions, can 1st be used to configure the MPs/MCs, i.e., use IREEL and EC to configure the GIMI services, as if configuring a complete experiment.
44 - on 5/2:  later, 2nd, consider extending IREEL to configure the entire experiment, i.e., interface with Flukes/OMNI, etc., to aquire resources.
45
46a!'')  on 4/20:  issue:  is there a MAP on common node?  at least to preview data?  is there a MAP in user workspace?  what?
47 - on 4/25:  for 1st, no MAP on common node
48
49b)  MPs with multiple possible measurements:  host info;  basic network performance with iperf, ping;  data from app
50 - on 4/25:  need to list baseline measurements, and configurable parameters
51 
52c)  MC (OML server) in common node (node n+1)
53
54d)  where is MAP service installed?  in common node?    in user workspace?    what is planned for this service?  how can analysis/presentation be configured?  is there an r script?  is there a GUI?  how is GUI accessed?
55 - on 4/25:  for 1st, expect MAP only bin user workspace
56 - on 4/25:  what is 1st MAP service?  what functions are provided?  can these be done using iREEL and r?
57 - on 5/2:  1st plan for MAP service:  base it on iREEL 2.0, installed as a public service;  where installed?  not user workspace;  not BBN;  UMassAmherst?  RENCI?  NICTA too far away. 
58
59e)  movement of data from MC (OML server) to MAP service:   direct?   how?   via iRODS, i.e., MC to iRODS to MAP?  how?
60 - on 4/18,  per Shu, and discussion:  could login to MC node;  establish first transfer from iRODS client on MC to to iRODS with iput;  use  iinit, so iRODS client remembers password;  then periodic transfers with no human intervention;  another push protocol, from MC to iRODS.
61 - on 4/18,  per Shu, and discussion: next, can install iRODS client on user workspace, then pull data back to user workspace for analysis and presentation, and then push back to iRODS for archive
62 - on 5/2:  must be able to pull data from IRODS to IREEL for MAP functions.  how?
63 
64f)  1st:  common node in slice;  2nd:  common node in user workspace (out of slice)  (is this beyond GEC14?)
65 - on 4/25:  for 1st, do common node in slice, but plan one per aggregate
66 
67g)  1st:  nodes on VMs, one aggregate;  2nd:  nodes on VMs, multiple aggregates
68
69h)  1st:  Load image for common node, MP nodes;  2nd:  load MP code as package before/after app
70 - on 4/25:  what is test app?  need to have something visible, e.g., video to a web page?
71
72i)  Is there authentication/authorization when data pushed fom MP to MC?  1st:  none;  considered OK  2nd:  add ssl?
73
74j)  Is there authentication/authorization when accessing GUI (MAP) on common node?  how?   does this use portal to GUIs?
75 - on 4/25:  not in 1st
76
77== 4)  *** Introduce XML messaging service:  required for OMF messages, used to orchestrate services ==
78
79a)  (Max, Christoph)  public XMPP server must be available;  what?  where?
80
81b)  (Max, Christoph)  signed OMF messages:  need summary of arrangement from NICTA.    what are names?  who configures pub/sub nodes?    what is structure?    how to generate and distribute keys?  is a public key registry required?    how would it be done?   
82 - on 4/18:  Max explains plan to use OMF6.0 approach;  limited messages;  can use XMPP or another transport;  signed messages;  can start with no authentication/authorization, and then turn it on later; 
83 - on 4/18:  need clear and complete description soon, particularly naming and establsihment of XMPP nodes. so that others can consider for use and Jeanne can begin testing.
84 - on 4/18:  expect release of first OMF6.0 by mid-May, with firm RC and skeleton EC;  expect updated EC by mid-June;  is this too late for GEC14?
85 - on 4/25:  need to start with an earlier version of OMF, e.g., OMF 5.3
86 - on 5/2:  need plan for using OMF 5.3 (or 5.4).
87
88== 5)  Tools in user workspace  ==
89
90a)  includes stores for keys, certificates, credentials, rspecs
91
92b)  (Jeanne A, David) Use Gush with OMNI to acquire resources, load images/packages, setup connectivity, manage processes
93
94c)  (Christoph) Use OMF EC to configure/orchestrate services;  can EC drive scripts?
95
96d)  (Ilia) FLUKES optional to create (or observe) rspec
97 - on 5/2:  how does this relate to IREEL, which will be sued to configure GIMI services?
98
99e)  (Ilia)  what is best way to load images?  packages?
100
101f)  (Christoph)  Add iREEL (experimenter portal service); 
102 - on 4/25:  what can this do?  can it provide MAP functions?  can it drive scripts?  (need copy to run locally)
103 - on 5/2:  1st plan for MAP service:  base it on iREEL 2.0, installed as a public service;  where installed?  not user workspace;  not BBN;  UMassAmherst?  RENCI?  NICTA too far away.
104 - on 5/2:  must be able to pull data from IRODS to IREEL for MAP functions.  how?
105 - on 5/2:  if use IREEL to provide MAP functions, can 1st be used to configure the MPs/MCs, i.e., use IREEL and EC to configure the GIMI services, as if configuring a complete experiment.
106 - on 5/2:  later, 2nd, consider extending IREEL to configure the entire experiment, i.e., interface with Flukes/OMNI, etc., to aquire resources.
107 
108 
109g)  (From GEMINI;  under discussion)  Include portal service to find other GUIs?
110 - on 4/25:  for 1st, not necessary 
111
112h)  (Giridhar, Jeanne, Shu)  Include Directory Archive (DA) service ,which can push data to DOA service, using OI service  (all from CNRI, also GPO ) 
113 - on 4/25:  now replaced with iRODS client
114 
115i)   (Giridhar, Jeanne, Shu) Include MDOD creator/editor  How does this interface with iRODS, ot push MDOD to iRODS? 
116  - on 4/18:  who on GIMI project will help define final MDOD schema?  Shu.
117  - on 4/25:  need to get group together, to revise MDOD schema, and consider implementation of creator/editor
118
119 
120
121
122== 6)  Introduce iRODS service ==
123
124After review with Shu on 4/12/12:
125
126a)  (Shu)  define configuration of iRODS servers, etc.  also logins for users
127
128b)  (Shu)  define functions, interfaces to iRODS, and iRODS client    can a host mount a file system in iRODS?
129
130c)  (Shu)  define naming in iRODS
131
132d) (Shu)  define use case for:  moving data from MC to iRODS, then from iRODS to user workspace   
133 - use iRODS cleint in MC to push to iRODS?  how is authentication/authorization handled?
134 - use iRODS client in user workspace to pull to user workspace?  how is authentication/authorization handled?
135
136On 4/18:  use cases received:
137
138GIMI iRODS Use Cases - Preparation
139
140 Create an account for user: shuang
141  iadmin mkuser shuang user_type
142 Assign user to a group
143  iadmin atg groupName shuang
144 Admin can assign password to user (also supports GSI/Kerberos)
145  iadmin moduser shuang password xxx
146 A lot of other things can be done:
147  iadmin –h
148 Create an iRODS directory for the user OML measurement data
149  imkdir results
150 
151GIMI iRODS Use Case 1
152 Perform an OML-based measurement, result is saved in a file (a.sqlite3) under /home/shuang/results
153 Put the file into iRODS
154  iput a.sqlite3 (suppose ipwd is /geniRenci/home/shuang/results)
155 Work on the local copy under /home/shuang/results
156 Push the modified file back into iRODS (think it as a version control system)
157  Irsync –r /home/shuang/result i:/geniRenci/home/shuang/results
158 
159GIMI iRODS Use Case 2
160 Check out a file in iRODS to my local directory
161  iget a.sqlite3
162 Work on the local copy
163 Push the modified file back into iRODS
164  irsync a.sqlite3 i:a.sqlite3
165
166GIMI iRODS Use Case 3
167 Suppose there are many files under /home/shuang/data that need to be copied into iRODS, we can mount the local directory directly into iRODS
168  imkdir /geniRenci/home/shuang/mountpoint
169  imcoll –m filesystem /home/shuang/data /geniRenci/home/shuang/mountpoint
170 Work on the local files
171 Sync with iRODS
172  Irsync –r /home/shuang/data i:/geniRenci/home/shuang/mountpoint
173
174GIMI iRODS Use Case 4
175 Data processing – e.g. filtering files under /geniRenci/home/shuang
176  Write a rule called filter.r which filters files and save them under /geniRenci/home/shuang/filtered
177  irule –F filter.r “*Path=‘/geniRenci/home/shuang’”
178
179e)  (Shu)  consider interface to get unique handle;  need to get current handle minting service from CNRI
180
181f)  (Shu)  consider storing and possibly formulating MDOD in iRODS    how?
182
183g)  (Shu)  review above with GEMINI team
184
185
186== 7)  ***Introduce IREEL service ==
187
188
189(Christoph)  Add iREEL (experimenter portal service):
190 - on 4/25:  what can this do?  can it provide MAP functions?  can it drive scripts?  (need copy to run locally)
191 - on 5/2:  1st plan for MAP service:  base it on iREEL 2.0, installed as a public service;  where installed?  not user workspace;  not BBN;  UMassAmherst?  RENCI?  NICTA too far away.
192 - on 5/2:  must be able to pull data from IRODS to IREEL for MAP functions.  how?
193 - on 5/2:  if use IREEL to provide MAP functions, can 1st be used to configure the MPs/MCs, i.e., use IREEL and EC to configure the GIMI services, as if configuring a complete experiment.
194 - on 5/2:  later, 2nd, consider extending IREEL to configure the entire experiment, i.e., interface with Flukes/OMNI, etc., to aquire resources.
195
196
197
198== 8)  +++ Steps towards GIMI testing ==
199
200a)  (Jeanne) Arrange user workspace
201
202c)  (Mike)  What is first configuration of tools code?  when?  who?
203 - on 4/18:  plan to forward all code and plans for GEC13 to Jeanne
204 - on 4/18:  code leads are David and Delip;  will join future calls
205 - on 5/2:  have sent code, scripts from GEC13 demo
206 - on 5/2:  (Jeanne)  for tracking, have setup process on geni.net
207
208d)  (Jeanne)  What are first test scripts?  when?
209 - on 4/25:  need to outline process for using tools in user workspace
210 - on 4/25:  need to provide first test plan
211
212e)  (Jeanne)  When are first tests?
213
214
215
216== 9)  +++ Steps towards GIMI tutorial at GEC14 ==
217
218a)  (Ilia) Start with GEC13 ORCA turorial?
219
220b)  (Mike)  Define covered use cases, and review with experiment support team (Mark, Niky)
221
222c)  (Mike)  When is first demo?
223
224d)  (Mike)  Need to establish high-level schedule:  freeze code by end of June?  then only testing, tutorial prep?
225
226
227
228== 10)  Use GIMI in WiMAX site  ==
229
230a)  (Mike, Cong)  Can we do something by GEC14?  what?
231 - on 4/25:  agree UMass Amherst WiMAX site should be up by GEC14
232 - on 4/25:  agree should have plan to integrate GIMI with WiMAX by GEC14, but does not need to be done
233
234
235