[[PageOutline]] = GIMI v2 Plan, Tasks and Status = == 1) Plan == === 1.1) Goals === a) GIMI (also GEMINI) to work with both ExoGENI and InstaGENI [[BR]] b) Working toward consistent user experience, both racks, both sets of I&M tools [[BR]] c) Implement basic workflow per http://groups.geni.net/geni/wiki/GeniExperiments#a5StepsinBasicGENIExperimentTutorialTestWorkflowHarry and http://groups.geni.net/geni/wiki/TestTutorialExperimentWorkflow [[BR]] Issue: what range of workflows must be supported? small (classroom exercise); medium (research experiment); large (long-running service, with opt-in) [[BR]] Answer: eventually all of these, but with an emphasis on small and medium at this time; should have a way to extend medium to large; Sol 4 calls for shakedown experiments that are large [[BR]] d) Easy to setup basic set of measurements (towards unified experiment environment) [[BR]] Issue: what are basic set of measurements? [[BR]] Answer: at least "passive host measurements", and "mesh of pings between nodes" [[BR]] e) Easy to mix-and-match tools, as go from step to step, with automatic transfer of info between steps/tools [[BR]] f) Easy to configure and then visualize (graph) measurements, starting from a graphical (or tabular) view of the topology of resources [[BR]] Issue: But, starting from a starting from a graphical (or tabular) view of the topology of resources to configure desired measurements is not always best. For example, for a classroom exercise, may want a pre-defined experiment, with a pre-define measurement configuration. For example, for a large experiment with 100s of nodes, it is not practical or useful to drive measurements from a graphical view of the topology; want to do everything with a script. [[BR]] Answer: Should be easy to configure and then visualize (graph) measurements, but the mechanism will likely differ depending on the type of experiment: small (predefined), medium, large. Nonetheless, should provide a method to serve all three that is not cumbersome for the experimenter. For a large experiment, will need to be able to use a script. [[BR]] h) Ability to simultaneously run multiple measurement sessions (e.g., passive host, mesh of pings, intermittent iperf, and experiment specific), each with multiple measurements and multiple graphs, during an experiment [[BR]] i) Stay consistent with Spiral 5 GIMI SOW [[BR]] === 1.2) Slice with GIMI I&M Tools Configuration === See [http://groups.geni.net/geni/attachment/wiki/GIMIv2tasks/Visio-041713_GIMI_Figures.pdf figures] [[br]] * Fig 4a) GIMI I&M tools Config Spiral 5 [[br]] * Fig 5) GIMI Portal Config [[br]] * Fig 7) OML Result Isolation [[br]] * Fig 8) IRODS Config [[br]] * Fig 1b) A&A: to iRODS with GSI [[br]] * Fig 1c) A&A: ti iRODS with ticket [[br]] * Gif 4b) WiMAX site config Spiral 5 [[br]] === 1.3) Detailed Workflow === See [http://groups.geni.net/geni/attachment/wiki/GIMIv2tasks/Visio-040313_GEMINI_Workflow.pdf sample detailed workflow] [[Br]] === 1.4) GIMI Portal Configuration === === 1.5) GIMI Portal Implementation === === 1.6) Acceptance tasks === == 2) Team == Mike Zink (UMass Amherst) Cong Wang (UMass Amherst) Max Ott (NICTA) Ilia Baldine (RENCI) Shu Huang (RENCI) Fraida Fund (NYU Poly) Deniz Gurkan (UH) Harry Mussman (BBN) Jeanne Ohren (BBN) == 3) Planning and Status Meetings/Conference Calls == 040113; 4pm EDT 040413; 4:30pm EDT 041513; 4pm EDT 042213; 4pm EDT; discuss iticket for iRODS 042913; 4pm EDT; discuss planning meeting 5/2- 5/3 for GIMI Portal; discuss interop status; discuss GSAS status 050213 - 050313; define GIMI Portal configuration 050913; review workflow, and plan to define GIMI Portal 051313; 2-4pm EDT; demo of interop, review workflow 052013 - 052213; design GIMI Portal 052713; 4pm EDT 061013; 2-4pm EDT; demo; bulk of work done on v2 GIMI Portal 062413; 4pm EDT 070813; 2-4pm EDT; demo; v2 ready for testing 072113 - 072313 is GEC17; v2 software released after testing; integrated tutorials presented == 4) GIMI Tasks Through GEC17 == * Started 032013 * Revised 032913 * Revised 040213 after 040113 call, and after GEMINI call on 040213 * Revised 040513 after discussion with Mike on 040413 * Revised after 041513 and 042213 calls, and discussions with Mike. * Revised after 042913 call 1) By 4/15: /cleanup/checkin/test/document current code [[BR]] What needs work? [[BR]] 2) Agree on goals: [[BR]] a) GIMI (also GEMINI) to work with both ExoGENI and InstaGENI [[BR]] b) Consistent user experience, both racks, both sets of I&M tools [[BR]] c) Implement basic workflow per http://groups.geni.net/geni/wiki/GeniExperiments#a5StepsinBasicGENIExperimentTutorialTestWorkflowHarry and http://groups.geni.net/geni/wiki/TestTutorialExperimentWorkflow [[BR]] Issue: what range of workflows must be supported? small (classroom exercise); medium (research experiment); large (long-running service, with opt-in) [[BR]] Answer: eventually all of these, but with an emphasis on small and medium at this time; should have a way to extend medium to large; Sol 4 calls for shakedown experiments that are large [[BR]] d) Easy to setup basic set of measurements (towards unified experiment environment) [[BR]] Issue: what are basic set of measurements? [[BR]] Answer: at least "passive host measurements", and "mesh of pings between nodes" [[BR]] e) Easy to mix-and-match tools, as go from step to step, with automatic transfer of info between steps/tools [[BR]] f) Easy to configure and then visualize (graph) measurements, starting from a graphical (or tabular) view of the topology of resources [[BR]] Issue: But, starting from a starting from a graphical (or tabular) view of the topology of resources to configure desired measurements is not always best. For example, for a classroom exercise, may want a pre-defined experiment, with a pre-define measurement configuration. For example, for a large experiment with 100s of nodes, it is not practical or useful to drive measurements from a graphical view of the topology; want to do everything with a script. [[BR]] Answer: Should be easy to configure and then visualize (graph) measurements, but the mechanism will likely differ depending on the type of experiment: small (predefined), medium, large. Nonetheless, should provide a method to serve all three that is not cumbersome for the experimenter. For a large experiment, will need to be able to use a script. [[BR]] h) Ability to simultaneously run multiple measurement sessions (e.g., passive host, mesh of pings, intermittent iperf, and experiment specific), each with multiple measurements and multiple graphs, during an experiment [[BR]] i) Stay consistent with Spiral 5 GIMI SOW [[BR]] 3) Build workflow for GIMI to work on both InstaGENI and ExoGENI [[BR]] a) Start with browser, GENI Desktop, FLACK and OMNI, for InstaGENI [[BR]] b) Start with User Workspace, Flukes and OMNI for ExoGENI [[BR]] c) When can InstaGENI be used with ExoGENI? also GENI Desktop? [[BR]] d) Which browsers are supported? [[BR]] e) Which credentials are supported? [[BR]] f) Need detailed plan from team: Jeanne (Lead), Luisa, Ezra, Hussam, Ilia (or rep) and Cong [[BR]] g) Current InstaGENI configuration per attached figure; changes? [[BR]] h) Current ExoGENI configuration? [[BR]] i) Plan to complete: [[br]] By 4/15: Jeanne will have met with team, and provide report of DRAFT plan [[BR]] 4/16: Sub-team has met, made plan; see http://groups.geni.net/geni/wiki/IMInteroperability [[BR]] 4) Define configuration of v2 (multi-session) GIMI Portal service. [[BR]] a) On 4/2: Reconfirm that will continue as a multi-user persistent service, with account for each user, to retain advantages of a persistent sink for measurement data. However, encourage users to move completed measurements to iRODS, so that we do not end up with long-term storage of measurement data within GIMI Portal Service. [[BR]] b) Continue to include OML Server and Lab wiki (for orchestration, results, and documentation) [[BR]] c) Separate OML results, by experiment and user, per the defined method [[BR]] d) For each user, introduce multiple "windows" to support multiple measurement sessions (e.g., passive host, mesh of pings, intermittent iperf, and experiment specific) [[BR]] e) For each user, be able to include session with Lab wiki to use OMF script to orchestrate experiment services (independent of any measurements) [[BR]] f) Store results in iRODS, by user, following defined structure [[BR]] g) Realize workable authentication/authorization to iRODS; start with current proxycert arrangement [[BR]] On 4/15: Fraida suggests iticket, used in iarchive command; Shu agrees to review and test iticket functions [[Br]] On 4/22: Shu reviews iticket functions; see link. Looks like a good mechanism for a multi-session GIMI Portal h) Do we want to support link from GENI Desktop? or somehow include Lab wiki as a plug-in in the GENI Desktop? These could be a step towards a "consistent user interface". [[BR]] On 4/19: Jim Griffioen says not hard to add link from browser in GENI Desktop (new tab, or a floating window) to GIMI Portal [[br]] i) Plan to define configuration: On 4/4: We need a workable solution, not the “best” solution. We can’t wait for future authentication/authorization mechanisms. [[BR]] On 4/4: We need to have the bulk of the work done in 6-8 weeks (no later than 6/11), then cleanup towards GEC17. [[BR]] On 4/4: A small team will be organized to work on the problem, define multiple solutions, and pick the best we can do in the time allotted: Jeanne, Cong, Mike (lead), Shu, (also web services expert from RENCI?), Max. [[BR]] By 4/15, Mike will have met with team, and will provide a report, including: evaluation of current Lab wiki code; available options for authentication of user to iRODS (starting with current proxycert arrangement); overview of current structure, and early view of possible options [[BR]] On 4/19, Mike agrees to have small group (Mike, Cong, Divya, Jeanne and Harry) meet in Cambridge 5/2 - 5/3 to define multi-session GIMI Portal, including: [[BR]] 1) Multiple sessions, but no multiple accounts [[BR]] 2) How info is held in portal for each user, and configured there for the user (DB?) [[BR]] 3) How user authenticates with the portal (like GENI CH Portal to GENI Desktop) [[BR]] 4) How the portal is authorized to write to the user's account in iRODS (based on iticket) [[BR]] 5) All captured in descriptive text, drawings, and a detailed workflow [[BR]] 5) Design GIMI Portal service [[br]] a) Who? On 4/19, Mike agrees to have small group (Mike, Cong, Divya, Max, Jeanne and Harry) meet in Cambridge 5/20 - 5/21 (also 5/22?) Suggest: have Tom/Aaron present to represent GENI CH Portal [[BR]] b) Follows: definition defined 5/2 - 5/3 [[BR]] c) Includes: [[BR]] Thin server (Apache server?) [[BR]] OML Server [[BR]] User DB [[BR]] LabWiki [[BR]] 6) Add features to iRODS [[BR]] a) Add rules to process incoming descriptors in XML files, ans store info in iCAT [[BR]] b) Add rules to move object to public archive directory, and manage any changes [[BR]] c) Integrate archive service with handle service [[BR]] d) Plan: [[br]] By 4/15, Shu needs to summarize planned extensions, per his earlier document. [[BR]] By 4/29, All GENI Storage and Archive Service plans summarized on wiki page at http://groups.geni.net/geni/wiki/GSAS [[BR]] By 4/29, All reviews of descriptor and iRODS work for GENI Storage and Archive Service completed, and descritor document completed. [[BR]] On 4/29, Giridhar agrees to provide update to schema. [[BR]] On 4/29, Shu agrees to provide latest task list, and schedule. [[BR]] 7) Include easy way to configure measurements, for all types of experiments (small, medium and large) [[BR]] Issue: Currently, involves: resource request (with ORCA or AM API); post-boot script; OMF script. But these are not linked. So, all is easy for a pre-defined (small) experiment, but script writing is reuired for a medium experiment, and info must be manually transferred from resource request/post boot script to the OMF script. [[BR]] a) Consider best way to "add GIMI extensions" (modify request to add software image or packages, and add any necessary nodes) [[BR]] b) Consider best way to "instrumentize with GIMI" (use script to setup proxy certs, configure services, connect via XMPP, and then start services) [[BR]] c) Consider best way to drive OMF script (on a medium experiment) from the slice information, e.g., number of nodes, names of nodes, topology of nodes. [[BR]] Option 1: consider use of GENI Desktop, with plug-ins to "add GIMI extensions" and "instrumentize with GIMI"; this would be a step towards a "consistent user interface". [[BR]] Option 2: consider some way to automatically write OMF script based on parameters forwarded from resource assignment. Maybe, manifest info could be transferred via iRODS. If this could be done for a medium experiment, perhaps an extension to a large experiment would be identified. [[BR]] Option 3: particularly for a large experiment, perhaps could automatically discover the nodes that are up, and the measurements that have been configured in each node. This could verify an OMF script. [[BR]] Option 4: although we do not need to drive measurement setup from graphical view of topology, we could discover the measurements that have been setup, and present a graphical view of the measurements, along with a graphical view of the resource topology. This could be used to verify the OMF script mathces the slioce resources, etc. [[BR]] d) Plan: [[br]] On 040413, put off discussion of this topic, until other topics are understood; may have BBN summer interns work on some of these topics. [[BR]] 8) Key task list: [[br]] || '''ID''' || '''Description''' || '''Assignee''' || '''Due''' || '''Status''' || '''Notes''' || || 1 || Define, prototype, and integrate push from GENI Portal to iRODs to create user account || Tom/Aaron,Shu || || || || || 2 || Define, prototype, and integrate link from GENI Portal to iRODS with login prompt || Tom/Aaron, Shu || || || || || 3 || Define, prototype, and integrate link from GENI Portal to GIMI Portal, SSO || Tom/Aaron, NICTA || || || || || 4 || Define and prototype interface to config GIMI Portal server || Cong, NICTA, Jeanne || || || || || 5 || Add measure_extensions to rspec || Mike, Jeanne, Ilia || || || || || 6 || Test Flack with ExoGENI || Divya, Jeanne, Luisa || || || || || 7 || Design and write scripts for setup, release, and archive || Cong, Jeanne || || || || || 8 || Define GIMI Portal config and workflow || GIMI arch team || 5/15 || || || || 9 || Design GIMI Portal || GIMI design team || 5/21 || || || || 10 || Define directory and file structure for all info in iRODS || GIMI arch team || || || || v2 GIMI Init Script task list: [[br]] || '''ID''' || '''Description''' || '''Who''' || '''Due''' || '''Status''' || '''Demos''' || '''Notes''' || || 1 || Establish GENI policy on sharing research results || || || || || || || 1a || DRAFT policy ( 2 pages) || Larry/Giridhar || 6/15/13 || || || || || 1b || Review with GPO || Larry/Giridhar || 6/20/13 || || || || || 1c || Review with community || Larry/Giridhar || GEC17 (how?)|| || || || || 1d || || || || || || || || 1e || || || || || || || v2 GIMI Portal task list: [[br]] || '''ID''' || '''Description''' || '''Who''' || '''Due''' || '''Status''' || '''Demos''' || '''Notes''' || || 1 || Establish GENI policy on sharing research results || || || || || || || 1a || DRAFT policy ( 2 pages) || Larry/Giridhar || 6/15/13 || || || || || 1b || Review with GPO || Larry/Giridhar || 6/20/13 || || || || || 1c || Review with community || Larry/Giridhar || GEC17 (how?)|| || || || || 1d || || || || || || || || 1e || || || || || || ||