GIMI: 041812_GIMI_NextSteps.txt

File 041812_GIMI_NextSteps.txt, 8.4 KB (added by hmussman@bbn.com, 7 years ago)
Line 
1GIMI tasks, issues, next steps:
2After discussion on 4/5/12:
3After iRODS review with Shu on 4/12/12:
4After status call with Mike and Max on 4/18/12:
5
61.  When is next status call?  5/2;  but reserving 4/25 for other topics.
7
8***  marks items that will be important to review on the next call
9 3.  *** Architecture of GIMI tools for GEC14 tutorial;  still need a clear and complete description soon;  when?  5/2?  earlier?  various issues
10 4.  *** Introduce XML messaging service:  required for OMF messages, used to orchestrate services;  expect release of first OMF6.0 by mid-May, with firm RC and skeleton EC;  expect updated EC by mid-June;  is this too late for GEC14?
11 7)  *** Steps towards GIMI testing:  start by forwarding all code and plans for GEC13 demo;  by 4/25?
12 8)  *** Steps towards GIMI tutorial at GEC14:  need to establish high-level schedule;  need to define and review use cases;  by 5/2?
13
14
152.  Target aggregates for GEC14 tutorial
16
17a)  (Ilia) exoGENI racks;  all VMs  (Ilia)
18
19b)  ( Ilia)  access to shell on all nodes by public IP address;  authentication/authorization using keys
20
21c)  ( Ilia)  access to web interfaces (GUIs) on all nodes by public IP address;    authentication/authorization using keys?  certificates? 
22
23
24
253.  *** Architecture of GIMI tools for GEC14 tutorial
26
27a)  (Mike)  need to arrive at nominal configuration ASAP
28
29on 4/18:  good discussion (see below);  still need a clear and complete description soon;  when?  5/2?  earlier?
30
31a')  on 4/20:  issue:  how to configure tools?  from EC to RC in  tool?
32
33a'')  on 4/20:  issue:  is there a MAP on common node?  at least to preview data?  is there a MAP in user workspace?  what?
34
35b)  MPs with multiple possible measurements:  host info;  basic network performance with iperf, ping;  data from app
36
37c)  MC (OML server) in common node (node n+1)
38
39d)  where is MAP service installed?  in common node?    in user workspace?    what is planned for this service?  how can analysis/presentation be configured?  is there an r script?  is there a GUI?  how is GUI accessed?
40
41e)  movement of data from MC (OML server) to MAP service:   direct?   how?   via iRODS, i.e., MC to iRODS to MAP?  how?
42
43e')  on 4/18,  per Shu, and discussion:  could login to MC node;  establish first transfer from iRODS client on MC to to iRODS with iput;  use  iinit, so iRODS client remembers password;  then periodic transfers with no human intervention;  another push protocol, from MC to iRODS.
44
45e'')  on 4/18,  per Shu, and discussion: next, can install iRODS client on user workspace, then pull data back to user workspace for analysis and presentation, and then push back to iRODS for archive
46 
47f)  1st:  common node in slice;  2nd:  common node in user workspace (out of slice)  (is this beyond GEC14?)
48
49g)  1st:  nodes on VMs, one aggregate;  2nd:  nodes on VMs, multiple aggregates
50
51h)  1st:  Load image for common node, MP nodes;  2nd:  load MP code as package before/after app   what is test app?
52
53i)  Is there authentication/authorization when data pushed fom MP to MC?  1st:  none;  considered OK  2nd:  add ssl?
54
55j)  Is there authentication/authorization when accessing GUI (MAP) on common node?  how?   does this use portal to GUIs?
56
57
584.  *** Introduce XML messaging service:  required for OMF messages, used to orchestrate services
59
60a)  (Max, Christoph)  public XMPP server must be available;  what?  where?
61
62b)  (Max, Christoph)  signed OMF messages:  need summary of arrangement from NICTA.    what are names?  who configures pub/sub nodes?    what is structure?    how to generate and distribute keys?  is a public key registry required?    how would it be done?   
63
64On 4/18:  Max explains plan to use OMF6.0 approach;  limited messages;  can use XMPP or another transport;  signed messages;  can start with no authentication/authorization, and then turn it on later; 
65
66b')  On 4/18:  need clear and complete description soon, particularly naming and establsihment of XMPP nodes. so that others can consider for use and Jeanne can begin testing.
67
68b'')  On 4/18:  expect release of first OMF6.0 by mid-May, with firm RC and skeleton EC;  expect updated EC by mid-June;  is this too late for GEC14?
69
705)  Tools in user workspace
71
72a)  includes stores for keys, certificates, credentials, rspecs
73
74b)  (Jeanne A, David) Use Gush with OMNI to acquire resources, load images/packages, setup connectivity, manage processes
75
76c)  (Christoph) Use OMF EC to configure/orchestrate services  can EC drive scripts?
77
78d)  (Ilia) FLUKES optional to create (or observe) rspec
79
80e)  (Ilia)  what is best way to load images?  packages?
81
82f)  (Christoph)  Add iREEL (experimenter portal service);  what can this do?  can it drive scripts?  (need copy to run locally) 
83
84g)  (From GEMINI;  under discussion)  Include portal service to find other GUIs? 
85
86h)  (Giridhar, Jeanne, Shu)  Include Directory Archive (DA) service ,which can push data to DOA service, using OI service  (all from CNRI, also GPO )  Or is this replaced with iRODS client?
87
88i)   (Giridhar, Jeanne, Shu) Include MDOD creator/editor  How does this interface with iRODS, ot push MDOD to iRODS? 
89
90j)  (Mike)  who on GIMI project will help define final MDOD schema?  Shu.
91
92 
93
94
956)  Introduce iRODS service
96
97After review with Shu on 4/12/12:
98
99a)  (Shu)  define configuraion of iRODS servers, etc.  also logins for users
100
101b)  (Shu)  define functions, interfaces to iRODS, and iRODS client    can a host mount a file system in iRODS?
102
103c)  (Shu)  define naming in iRODS
104
105d) (Shu)  define use case for:  moving data from MC to iRODS, then from iRODS to user workspace   
106use iRODS cleint in MC to push to iRODS?  how is authentication/authorization handled?
107use iRODS client in user workspace to pull to user workspace?  how is authentication/authorization handled?
108
109On 4/18:  use cases received:
110
111GIMI iRODS Use Cases - Preparation
112
113 Create an account for user: shuang
114  iadmin mkuser shuang user_type
115 Assign user to a group
116  iadmin atg groupName shuang
117 Admin can assign password to user (also supports GSI/Kerberos)
118  iadmin moduser shuang password xxx
119 A lot of other things can be done:
120  iadmin –h
121 Create an iRODS directory for the user OML measurement data
122  imkdir results
123 
124GIMI iRODS Use Case 1
125 Perform an OML-based measurement, result is saved in a file (a.sqlite3) under /home/shuang/results
126 Put the file into iRODS
127  iput a.sqlite3 (suppose ipwd is /geniRenci/home/shuang/results)
128 Work on the local copy under /home/shuang/results
129 Push the modified file back into iRODS (think it as a version control system)
130  Irsync –r /home/shuang/result i:/geniRenci/home/shuang/results
131 
132GIMI iRODS Use Case 2
133 Check out a file in iRODS to my local directory
134  iget a.sqlite3
135 Work on the local copy
136 Push the modified file back into iRODS
137  irsync a.sqlite3 i:a.sqlite3
138
139GIMI iRODS Use Case 3
140 Suppose there are many files under /home/shuang/data that need to be copied into iRODS, we can mount the local directory directly into iRODS
141  imkdir /geniRenci/home/shuang/mountpoint
142  imcoll –m filesystem /home/shuang/data /geniRenci/home/shuang/mountpoint
143 Work on the local files
144 Sync with iRODS
145  Irsync –r /home/shuang/data i:/geniRenci/home/shuang/mountpoint
146
147GIMI iRODS Use Case 4
148 Data processing – e.g. filtering files under /geniRenci/home/shuang
149  Write a rule called filter.r which filters files and save them under /geniRenci/home/shuang/filtered
150  irule –F filter.r “*Path=‘/geniRenci/home/shuang’”
151
152
153
154
155e)  (Shu)  consider interface to get unique handle
156
157f)  (Shu)  consider storing and possibly formulating MDOD in iRODS    how?
158
159g)  (Shu)  reivew above with GEMINI team
160
161
1627)  *** Steps towards GIMI testing
163
164a)  (Jeanne) Arrange user workspace
165
166c)  (Mike)  What is first configuration of tools code?  when?  who?
167
168On 4/18:  plan to forward all code and plans for GEC13 to Jeanne
169
170On 4/18:  code leads are David and Delip;  will join future calls
171
172d)  (Jeanne)  What are first test scripts?  when? 
173
174e)  (Jeanne)  When are first test?
175
176
177
1788)  *** Steps towards GIMI totorial at GEC14
179
180a)  (Ilia) Start with GEC13 ORCA turorial?
181
182b)  (Mike)  Define covered use cases, and review with experiment support team (Mark, Niky)
183
184c)  (Mike)  When is first demo?
185
186d)  (Mike)  Need to establish high-level schedule:  freeze code by end of June?  then only testing, tutorial prep?
187
188
189
1909)  Use GIMI in WiMAX site
191
192a)  (Mike, Cong)  Can we do something by GEC14?  what?
193
194
195