GEMINI_TopicsIssuesTasks: 041912_GEMINI_TopicsIssuesTasks.txt

File 041912_GEMINI_TopicsIssuesTasks.txt, 21.0 KB (added by hmussman@bbn.com, 12 years ago)
Line 
1[[PageOutline]]
2
3= GEMINI Topics, Issues and Tasks =
4
5Notes and tasks from 3/22/12 GEMINI status call:  [[BR]]
6Additions and corrections after call on 3/27/12 with Martin:  [[BR]]
7Additions and corrections after team call on 3/29/12, and call on 3/30/12 with Jim:  [[BR]]
8Additions and changes after call with Martin and Jim on 4/4/12:  [[BR]]
9Addtions and changes after call with Jim and Charles on 4/17/12:  [[BR]]
10Additions and changes after call with Martin on 4/18/12:  [[BR]]
11Additions and changes after team call 4/19/12:   (sections are rearranged) [[BR]]
12
13* 4/5/12 topics :  [[BR]]
14 2d)  ABAC  [[BR]]
15 4d)  Old UNIS vs new UNIS  [[BR]]
16 5j')  Use of Gush  [[BR]]
17
18+ 4/12/12 topics suggested by Harry:  [[BR]]
19 3g)  review access to http on VM (Hussam+) [[BR]]
20 5i)  who will help formulate final MDOD schema?   [[BR]]
21 5j')  report on Gush  (Jeanne) [[BR]]
22 7h)  review current status of baseline configuration to gather host metrics (Guilherme, and all) [[BR]]
23 8)  agree on steps towards GEMINI tutorials at GEC14  (all) [[BR]]
24 9)  agree on configuration to collect network measurements (all) [[BR]]
25 9l)  review preliminary test plan for colelcting network measurements  (Jeanne)  [[BR]]
26
27+ 4/19/12 topics suggested by Harry: [[BR]]
28 3g)  review access to http on VM (Hussam+) [[BR]]
29 4d)  Old UNIS vs new UNIS: understand transition plan  (?) [[BR]]
30 5i)  who will help formulate final MDOD schema? (Martin on4/18:  Ezra and Omer)  [[BR]]
31 5j')  report on Gush  (Jeanne)   (delayed) [[BR]]
32 6d)  provide a more complete view of GEMINI portal service  (Jim, Charles on 4/18/12)  [[BR]]
33 7h)  review current status of baseline configuration to gather host metrics (Guilherme, and all) [[BR]]
34 8)  agree on steps towards GEMINI tutorials at GEC14  (all) [[BR]]
35 9)  agree on configuration to make active network measurements (all) [[BR]]
36 9l)  review updated test plan for making active network measurements  (Jeanne)  [[BR]]
37
38+ 4/26/12 topics suggested by Harry: [[BR]]
39 3g)  review access to http on VM (Hussam+) [[BR]]
40 5j')  report on Gush  (Jeanne)   (delayed) [[BR]]
41 6d)  provide a more complete view of GEMINI portal service  (Jim, Charles on 4/18/12)  [[BR]]
42 7l)  review updated test plan for making active network measurements  (Jeanne)  [[BR]]
43 8a)  review current status of baseline configuration to gather host metrics (Guilherme, and all) [[BR]]
44 9)  agree on steps towards GEMINI tutorials at GEC14  (all) [[BR]]
45 9a)  agree on plan to coordinate configurations to make active network measurements  and to collect basic host metrics in one slice (all) [[BR]]
46 9d)  agree on schedule  [[BR]]
47
48 
49 
50== 1)  Authentication and authorization:  multiple actor options: ==
51
52a)  tool (outside slice) to !AggMgr srvc;  AM API;  XMl-RPC + ssl   [protoGENI cert + GENI credential]
53
54b)  tool (outside slice) to host (Slice A);  ssh, scp    [private/public keys]
55
56c)  tool (outside slice) to I&M srvc (Slice A);  http(s)   [in LAMP, browser to GUI, https with protoGENI cert]  [can private/public keys be used for access to a GUI?]  [in OMF, signed messages using private/public keys;  more details?]
57
58d)  I&M srvc (Slice A) to I&M srvc (Slice A);  http(s)   [in LAMP, service to service, https with LAMP cert, from LAMP CA]    [in GIMI/OML, not using http;  what is done there?]
59
60e)  I&M srvc (Slice A) to I&M srvc (Slice B);  http(s)   [in European perfSONAR, SOAP interface with security tokens]  [can delegated GENI credentials be used?]  [can credentials based on ABAC be used?]
61
62f)  I&M srvc (Slice A) to UNIS srvc;  http(s)   [in LAMP, service to UNIS, https with protoGENI cert]
63
64g)  tool (outside slice) to iRODS archive srvc;  what is interface to iRODS?  ftp(s)?  can it be http(s)?  how is authentication/authorization handled?   [need info from Shu]  Note;  iRODS review call on 4/12, 9:30am. 
65
66h)  option:  I&M srvc (Slice A) to iRODS archive srvc;  is there any way to move data from MC direct to iRODS?  perhaps mount iRODS on node with MC?  [need info from Shu]
67
68
69
70
71== 2)  Authentication and authorization:  multiple methods: ==
72
73a)  [for ssh, ssl, etc.]  private/public keys   
74
75a')  [in OMF]  signed messages using private/public keys
76
77b)  user certificates
78
79c)  GENI credentials  (user and slice)
80
81c')  [in IMF, GENI credentials included with XML messages, for authorization?  how?  reuse?]
82
83d)  ABAC  [Harry:  GPO believes that ABAC may eventually be used for resource assignment, but not soon]  [What code is available from ISI?  Jim is checking with Teb Faber;  waiting for a response]
84
85        ABAC references: [[BR]]
86        Deter web site: http://abac.deterlab.net/ [[BR]]
87        Authorization storyboard from Jeff Chase:  http://groups.geni.net/geni/wiki/AuthStoryBoard [[BR]]
88        Slides on credential store from Jeff Chase:  http://groups.geni.net/geni/attachment/wiki/AuthStoryBoard/certstore.ppt [[BR]]
89        Slides on future of authorization in GENI from Tom Mitchell:  http://groups.geni.net/geni/attachment/wiki/GEC13Agenda/Authorization/AuthFuture.pdf  [note options without and with credential store] [[BR]]
90        Summary of GENI authorization discussion at GEC13 (and before):  http://groups.geni.net/geni/wiki/GeniAuthorization  [[BR]]
91
92*4/5/12 notes from Jeanne: [[BR]]
93Jim got code from Ted Faber.  Looking through it.  Looking at example code.   [[BR]]
94ABAC is not currently implemented (at ISI?) as a service.  This needs to be done.  Ted thinks this should be trivial.  Looks like (via papers) ORCA has implemented as a server (ORCA Pod?) with RESTful interface. [[BR]]
95Jim contacted Jeff Chase to get code.  Making some progress, still some unknowns. [[BR]]
96Martin: Thinks we perhaps can use UNIS for source of constraints or reference (URL) to constraints.  Use libabac to prove the chain of assertions. [[BR]]
97All agree that we should have a central location for rules. [[BR]]
98Guilherme:  Don’t want the rules to be exposed. [[BR]]
99Is the proving done at the service or at the authenticating application? [[BR]]
100Task:  Harry suggests drawing up a proposal for using ABAC.  Jim:  Jim and Martin to discuss, learn more, and come up with a proposal.   [[BR]]
101
102== 3)  Target protoGENI environments: ==
103
104a)  servers:  relatively few;  public IP available
105
106b)  VMs:  OpenVZ;  expect move to LXC;  internal to an aggregate, private host name, private IP addresses, need more details
107
108c)  To date, all LAMP/periscope has been on servers
109
110c')  Task:  try to run all LAMP nodes (or just measurement nodes) on VMs   (Matt Jaffe)
111
112d)  To date, all INSTOOLS has been with MC on server, and MPs on VMs
113
114e)  Task:  Try to run INSTOOLS MC in a VM;  Nasir on 3/30:  still runs, although might need some small code changes; but would need to open http port, perhaps with extension to rspec;  need to discuss with protoGENI (Jonathan Deurig) about adding to mapping agent;  Jim had discussed with Jonathan and Rob earlier, quite doable, but would have to restart mapping agent;  perhaps could "piggyback" on opening ssh port? 
115
116f)  Task:  can ssh into public host name (or public IP) , with special 5+ digit port number (from manifest) from port map
117
118g)  Task:  (see e) above) how to access http interface?  tunnel through ssh?  port map, like ssh?  perhaps could "piggyback" on opening ssh port?  setup a separate proxy?   [[BR]]
119
120 See [http://groups.geni.net/geni/wiki/RspecExtensionProposal_PortMapping  Port mapping proposal from Hussam]  [[BR]]
121 Jim on 4/12:  Talked with Rob Ricci.  He is willing to work on it, since many needs to proxy to VMs.   (In INSTOOLS, used VNC) [[BR]]
122 On 4/12:  protoGENI considering using OPS server to provide persistent proxy. [[BR]]
123 Jim on 4/19:  Long term solution:  Gary Wong at Utah is considering a proxy service per testbed, advertised in manifest.  [[BR]]
124 Jim of 4/19:  Short term solution:  user can get to http port via ssh and port forwarding;  Hussam has script.  [[BR]]
125 
126h)  Task:  what about vnc tunnels?  how were they done in INSTOOLS?   which port on host?  (who?)
127
128i)  Task:  what happens when VMs are on one aggregate?  can reach each other via private IP addresses
129
130i')  Task:  what happens when VMs are on multiple aggregates?   (who?)
131
132
133
134
135
136
137== 4)  UNIS service ==
138
139Per call with Martin on 3/27/12:
140
141a)  Question:  In LAMP, is there a local UNIS, or not??   
142 (Martin on 3/27:) Not yet;  needs to be, with push from local UNIS to global UNIS.
143
144b)  How does UNIS authenticate/authorize when receiving data?   
145 (Martin on 3/27:)  [in LAMP, service to UNIS, https with protoGENI cert]
146
147c)  Question:  Use web interface on common node to configure services, tests;  how does this push config to UNIS?  What authentication/authorization steps are included?
148
149d)  Old UNIS vs new UNIS: [[BR]]
150
151 4/5/12 notes from Jeanne:  What is the transition plan?   
152 Both can run in parallel until full functionality is available with new UNIS.  Then turn down old UNIS.  [[BR]]
153 Local vs. global UNIS hierarchy:  Will new UNIS have local and global configuration?  Yes, probably not by GEC14. [[BR]]
154  4/18/12 notes from Harry:  Martin indicates that Old UNIS and New UNIS can run in parallel, and in sync, until Old UNIS is eventually turned down.  [[BR]]
155 Guilherme on 4/19:  Expect Old UNIS and New UNIS to contain smae information, but no 1:1 mapping  [[BR]]
156 Guilherme on 4/19:  Could write a wrapper for perfSONAR services to talk with New UNIS, but significant effort.  [[BR]]
157
158== 5)  User workspace service ==
159
160Current view:    (Harry)
161
162a)  Persistent Linux environment, with file system, key/certificate/credential store, dedicated to the user;  also rspec store, etc.
163
164b)  Place for tools, e.g., Gush and OMNI, and EC, and scripts;  can easily call one another;  not in slice;  could deal with multiple slices
165
166c)  Place for "portals";  but what are they? (see below)
167
168d)  Task:  Setup user workspace using server (or VM) in BBN Cambridge lab;  begin to include tools, etc .    (Jeanne) 
169On 3/30/12:  Done on VM in BBN Cambridge lab, ubuntu 10.04, internal to BBN. 
170**Next:  external to BBN
171
172e)  Task: Consider VM to distribute user workspace  (Matt);  e.g, ubuntu on virtual box [similar to what has been done at GEC tutorials]
173
174f)  Task:  What is required to secure keys/certificates/credentials?  passphrase?  other?  [Per Tom Mitchell, OMNI does not require passphrase, but FLACK does currently require passpharase]  [Per Jim protoGENI cert does require passphrase] [Vic to check with Steve Schwab;  need to balance security and ability ot use scripts.]
175
176g)  Start with CNRI:  Directory Archive (DA) service, which can push data to DOA service, using OI service [[BR]]
177Then replace DOA with iRODS [[BR]]
178[Have iRODS at IU for NetKarma;  Jim and Wesley talking with Ilia and Shu] [[BR]]
179
180g')  Or, do we use IRODS client to push/pull date to/from IRODS?  [[BR]]
181
182h)  Include MDOD creator/editor  (CNRI, GPO)
183
184i)  Task:  Need help with final formulation of MDOD   (Ezra and Omer from GEMINI)  (Shu from GIMI)
185
186j)  Task:  Define view of user workspace service (Jeanne, Matt, Harry, Jim, Martin, Niky)  [[BR]]
187 [Jeanne to add security policy into view]  [[BR]]
188
189j')  Use of GUSH:  [[BR]]
190*4/5/12 notes from Jeanne:  [[BR]]
191What does Gush provide vs. Flack?  Why would user use Gush?  [[BR]]
192Working with VMs.   According to Vic, Jeannie A. says Gush will work with anything that allows SSH.  [[BR]]
193Jeanne O. has experienced some issues with VMs in Gush.  Investigate further.  [[BR]]
194Issues with hostnames?  Need to investigate this further.  [[BR]]
195Harry:  Suggest Jeanne talk with Luisa about Gush information.  She has worked with it a lot.  [[BR]]
196Jim asks Martin:  How does Gush integrate with UNIS?  [[BR]]
197Discussion of using UNIS to store/access information about the slices for the experiment rather than passing around rspecs.  [[BR]]
198How do we keep this UNIS information up-to-date?  [[BR]]
199Guilherme suggests things that are outside of slice introspection, user needs to push to UNIS.  [[BR]]
200What types of changes can we make to the slice in Gush/Omni/other that I&M and others need to discover from UNIS?  [[BR]]
201Task:  Things to investigate regarding Gush (Jeanne will report next week):  [[BR]]
2021.   Tridentcom paper says gush has ability to add and remove nodes from a slice.  How is this done?  Under what circumstances does this work? [GENI AM API does not support updateSliver]  [[BR]]
2032.   How does Gush work with protogeni VMs?  [[BR]]
204
2054/12: Jeanne reports:    [[BR]]
206Certain commands (slice add, renew, update) apply only to PlanetLab.  [[BR]]
207Issue:  Gush takes hostname and goes to physical node, not VMs;  considering how to rectify.  [[BR]]
208In ORCA, this works.  hostname in manifest is not the same as the hostname reported for the node.  [[BR]]
209
2104/12:  Jim asks:  How would be run shell scripts on nodes?  scp between nodes?  [[BR]]
211
212== 6)  Portal services ==
213
214a)  Option 1:  "portal to UIs".  [Is this close to Jim's proposal?]
215
216b)  Option 2:  a more complete tool for managing I&M services, as is implied in the SOW, so that it can view, orchestrate, edit MDOD, manage archiving, etc.  [Is this close to Max's proposal]
217
218c)  Task:  understand options for authentication and authorization at a web interface.  (who?)
219
220d)  Task: provide a more complete view of GEMINI portal service   (Jim and Charles) [[BR]]
221
222Task:  Jim and Charles plan to provide in a week or two. [[BR]]
223       
224Task:  Charles needs to find a name for the service   [[BR]]
225
226After discussion on 3/31/12 with Jim, Harry feels that this is very close to Option 1:  "portal to UIs".   [[BR]]
227       
228Jim expects User to have a capable browser, e.g., one that runs HTML-5 [[BR]]
229       
230Jim expects portal to manage windowing to various GUIs. [[BR]]
231       
232Jim expect all interactions to be via browser, so there are window(s) to login to shell(s), etc. [[BR]]
233       
234Jim does not specify whether browser is looking at GUI in slice, or a tool;  tools are not in a specified place.
235       
236Harry feels that portal and other tools are in a "user workspace",  in a persistent Linux environment, with file system, key/certificate/credential store, dedicated to the user;  could also have rspec store, etc. ;  then, all tools have ready access to required info, and can readily call one another. [[BR]]
237       
238Harry thinks of "persistent Linux environment" on infrastructure, e.g., a server under your desk or in the lab;  not your laptop;  Jim agrees, and has thought portal would be hosted on infrastructure at Kentucky [[BR]]
239       
240Harry feels that this is just a strucutre, that there is much more work to define tools, interfaces, etc.;  Jim agrees, was concerned it was the final configuration. [[BR]]
241       
242Task:  Harry will modify drawing to reflect discussion with Jim, and then two perspectives can be compared. [[BR]]
243
244Done on 4/4;  agree that portal can be in user workspace, or somewhere else. [[BR]]
245See updated drawing.  [[BR]]
246
247Jim and Charles on 4/17:  showed early demo;  portal is very thin, and runs in a server;  user logs into portal with browser, typically using HTML5 that runs in browser;  portal includes plugins that interface to various parts of the system, parts of slice and/or tools;  tools can run in user workspace;  portal remembers parameters, but is NOT persistent;  common denominator is  a web GUI, even when interfacing to a CLI;  expect user's identity to pull credentials from store  ("ABAC" approach);  expect various rspecs to be stored in UNIS.
248
249Guiherme on 4/19:  saw demo of portal
250
251Jim and Charles on 4/17/12:  next step is to further define portal, and write it up.
252
253e)  Task:  Understand NICTA's iREEL portal service;  is this a more complete tool for managing I&M services?   
254
255  - Get login, and survey  (Jeanne)
256       
257  - Provide more info (NICTA, e.g., Christoph)
258
259
260== 7)  Configuration 1:  make active network measurements  ==
261
262a)  follows p15 Operator A slice, like LAMP project. 
263
264a') for GEC14, consider: use LAMP code with perhaps minor modifications
265
266a!'')  Start:  each node on a server, with an available public IP address  single aggregate or multiple aggregates
267
268a!''')  Second:  all nodes on VMs, or all nodes on VMs except common node on server
269
270b)  One common node (e.g., node n+1) to:  configure measurements, collect data, present data
271
272c)  Multiple measurement nodes (e.g., nodes 1, 2, ..., n)
273
274d)  Start:  one slice, classic pS interface, with data pulled from measurement node, authorized with GENI credential or ?
275
276e)  Global old UNIS as shown;  include local UNIS on common node?
277
278f)  Load software onto common node with image;  as option, load package
279
280g)  Load software onto measurement node with image;  as option, load package after app
281
282g')  Use Gush to manage processes, so that can easily switch between app and active network measurements;  good for establishing continuity and available bandwidth at beginning of experiment.
283
284h)  Use web interface on common node to configure services, tests, like LAMP;  how does this push config to UNIS??  How do we let only user do this with keys, etc.
285
286i)  Use web interface on common node to present/observe data, like LAMP  How do we let only user do this with keys, etc.
287
288j)  Demo all of the perfSONAR network performance tools, tests, as was done in LAMP project
289
290k)  Extension:  pull data from one slice to another, as shown in p15 from Operator A to Operator B;  authorize using GENI credentials;  how is this done?
291
292l)  Provide regression tests of various configurations, features, etc., driven by scripts  (Jeanne)
293
294m)  Formulate tutorial for users at GEC14.  What is content of the tutorial, i.e., what are the use cases?  what is different between the test plan and the tutorial?
295
296n)  Coordinate tutorial content with GPO (Mark, Niky)  when? starting in mid-May   how?
297
298
299
300== 8)  Configuration 2:  gather basic host metrics  ==
301
302a)  follows p15 Experimenter C slice 
303
304a')  Need baseline configuration ASAP   [[BR]]
305  4/12:  (Guilherme)  considering overall framework, working on interfaces, considering UNIS functions and schema.   [[BR]]
306  4/19:  (Guilherme) Need baseline config for GEC14 tutorials ASAP  [[BR]]
307 
308a!'')  Need MP to gather host metrics   (Guilherme)
309  - easy for raw servers;  hard for VMs
310  - considering BLiPP (Matt) to gather all utilization metrics;  extend via libvirt to gather metrics from VMs
311  - Could still use SNMP daemon from INSTOOLS  (Jim, Hussam)
312  -  Use netflow to gather flow metrics;  need to be able to turn on/off since uses many resources
313  -  Could associate ntflow with both hosts and switches.
314 
315a!''')  Need to define which host metrics to gather
316 - 4/12:  For those gathered by INSTOOLS, see [http://groups.geni.net/geni/attachment/wiki/GEMINI_TopicsIssuesTasks/012312_INSTOOLS_Measurements.txt   list] by Hussam
317 - Talking to Dan about use cases for gathering host metrics.
318 - 4/19:  see list on github at:   [https://github.iu.edu/GEMINI/GEMINI/wikis/user-stories-for-instrumenting-an-experimenters-slice list]
319 - 4/19:  start with utilization metrics
320 - 4/19:  later, add netflow metrics
321
322
323b)  MP pushes to Measurement Store (MS)
324  - Use http?  POST to port?  what about authentication and authorization?
325  - Use XSP, for streaming?
326       
327c)  Need to realize new MS
328  - How many options?
329  - One per Aggregate?
330  - Can also be arranged for publish/subscribe;  how?
331
332d)  Uses new UNIS
333 - Uses RESTful interface, replaces older UNIS with SOAP interface
334 - Allows drawing topology
335 - Used to configure services?
336 - Prototype underway (Ahmed)
337 
338e)  Need to realize MAP service
339
340 - Start with modified Periscope service
341 - Later:  include druple from INSTOOLS?
342 - Uses new MS
343 - Uses new UNIS
344
345f)  Later:  Extend to gathering data from an application
346
347g)  Task:  Prototype soon   when? (Guilherme)
348
349h)  Backup option:  Use what capabilities are in LAMP?
350
351i)  Provide regression tests of various configurations, features, etc., driven by scripts  (Jeanne)
352
353j)  Formulate tutorial for users at GEC14.  What is content of the tutorial, i.e., what are the use cases?  what is different between the test plan and the tutorial?
354
355k)  Coordinate tutorial content with GPO (Mark, Niky)  when?  how?
356
357
358== 9)  Steps toward GEC14 tutorials ==
359
360a)  Need plan to coordinate configurations to provide make active network measurements and to collect basic host metrics in the same slice
361 - 4/19:  For GEC13, LAMP code and INSTOOLS code were both loaded;  portal was pointed to both GUIs
362 - 4/19:  Plan to load both types of code
363 - 4/19:  Perhaps UK could work on unified deployment plan.  Note:  Cannot use UpdateSlice.
364 
365b)  Long term strategy:  unified set of tools
366
367c)  Start with protoGENI tutorial?  LAMP tutorial?  INSTOOLS tutorial?
368
369d)  Arrange user workspace (GPO, Jeanne)
370
371e)  Need to agree on schedule for GEC14 tools:
372 - Design review week of 5/7?  includes arch, configuration, use cases
373 - Development complete (freeze) on 6/15
374 - Testing and bug fixes:  +2 weeks
375 - Prepare and test turotrial:  +1 week
376
377
378
379== 10)  GEMINI project documentation ==
380
381a)  Code on IU github  Good:  all relevant code appears to be here, including Kentuck code
382
383b)  Jira  Good:  being used by IU to track project  Concern:  Kentucky effort not reflected here
384
385c)  GENI trac for GEMINI  Include TopicsTasksIssues  Include drawings