[[PageOutline]] = GIMI v2 Plan, Tasks and Status = == 1) Plan == === 1.1) Goals === a) GIMI (also GEMINI) to work with both ExoGENI and InstaGENI [[BR]] b) Working toward consistent user experience, both racks, both sets of I&M tools [[BR]] c) Implement basic workflow per http://groups.geni.net/geni/wiki/GeniExperiments#a5StepsinBasicGENIExperimentTutorialTestWorkflowHarry and http://groups.geni.net/geni/wiki/TestTutorialExperimentWorkflow [[BR]] Issue: what range of workflows must be supported? small (classroom exercise); medium (research experiment); large (long-running service, with opt-in) [[BR]] Answer: eventually all of these, but with an emphasis on small and medium at this time; should have a way to extend medium to large; Sol 4 calls for shakedown experiments that are large [[BR]] d) Easy to setup basic set of measurements (towards unified experiment environment) [[BR]] Issue: what are basic set of measurements? [[BR]] Answer: at least "passive host measurements", and "mesh of pings between nodes" [[BR]] e) Easy to mix-and-match tools, as go from step to step, with automatic transfer of info between steps/tools [[BR]] f) Easy to configure and then visualize (graph) measurements, starting from a graphical (or tabular) view of the topology of resources [[BR]] Issue: But, starting from a starting from a graphical (or tabular) view of the topology of resources to configure desired measurements is not always best. For example, for a classroom exercise, may want a pre-defined experiment, with a pre-define measurement configuration. For example, for a large experiment with 100s of nodes, it is not practical or useful to drive measurements from a graphical view of the topology; want to do everything with a script. [[BR]] Answer: Should be easy to configure and then visualize (graph) measurements, but the mechanism will likely differ depending on the type of experiment: small (predefined), medium, large. Nonetheless, should provide a method to serve all three that is not cumbersome for the experimenter. For a large experiment, will need to be able to use a script. [[BR]] h) Ability to simultaneously run multiple measurement sessions (e.g., passive host, mesh of pings, intermittent iperf, and experiment specific), each with multiple measurements and multiple graphs, during an experiment [[BR]] i) Stay consistent with Spiral 5 GIMI SOW [[BR]] === 1.2) Slice with GIMI I&M Tools Configuration === See [http://groups.geni.net/geni/attachment/wiki/GIMIv2tasks/Visio-041713_GIMI_Figures.pdf figures] [[br]] * Fig 4a) GIMI I&M tools Config Spiral 5 [[br]] * Fig 5) GIMI Portal Config [[br]] * Fig 7) OML Result Isolation [[br]] * Fig 8) IRODS Config [[br]] * Fig 1b) A&A: to iRODS with GSI [[br]] * Fig 1c) A&A: ti iRODS with ticket [[br]] * Gif 4b) WiMAX site config Spiral 5 [[br]] === 1.3) Detailed Workflow === See [http://groups.geni.net/geni/attachment/wiki/GIMIv2tasks/Visio-040313_GEMINI_Workflow.pdf sample detailed workflow] [[Br]] === 1.4) GIMI Portal Configuration === === 1.5) GIMI Portal Implementation === === 1.6) Acceptance tasks === == 2) Team == Mike Zink (UMass Amherst) Cong Wang (UMass Amherst) Max Ott (NICTA) Ilia Baldine (RENCI) Shu Huang (RENCI) Fraida Fund (NYU Poly) Deniz Gurkan (UH) Harry Mussman (BBN) Jeanne Ohren (BBN) == 3) Planning and Status Meetings/Conference Calls == 040113; 4pm EDT 040413; 4:30pm EDT 041513; 4pm EDT 042213; 4pm EDT; discuss iticket for iRODS 042913; 4pm EDT; discuss planning meeting 5/2- 5/3 for GIMI Portal; discuss interop status; discuss GSAS status 050213 - 050313; define GIMI Portal configuration 050913; review workflow, and plan to define GIMI Portal 051313; 2-4pm EDT; demo of interop, review workflow 052013 - 052213; design GIMI Portal 052713; 4pm EDT 061013; 2-4pm EDT; demo; bulk of work done on v2 GIMI Portal 062413; 4pm EDT 070813; 2-4pm EDT; demo; v2 ready for testing 072113 - 072313 is GEC17; v2 software released after testing; integrated tutorials presented == 4) GIMI Tasks Through GEC17 == * Started 032013 * Revised 032913 * Revised 040213 after 040113 call, and after GEMINI call on 040213 * Revised 040513 after discussion with Mike on 040413 * Revised after 041513 and 042213 calls, and discussions with Mike. * Revised after 042913 call 1) By 4/15: /cleanup/checkin/test/document current code [[BR]] What needs work? [[BR]] 2) Agree on goals: [[BR]] a) GIMI (also GEMINI) to work with both ExoGENI and InstaGENI [[BR]] b) Consistent user experience, both racks, both sets of I&M tools [[BR]] c) Implement basic workflow per http://groups.geni.net/geni/wiki/GeniExperiments#a5StepsinBasicGENIExperimentTutorialTestWorkflowHarry and http://groups.geni.net/geni/wiki/TestTutorialExperimentWorkflow [[BR]] Issue: what range of workflows must be supported? small (classroom exercise); medium (research experiment); large (long-running service, with opt-in) [[BR]] Answer: eventually all of these, but with an emphasis on small and medium at this time; should have a way to extend medium to large; Sol 4 calls for shakedown experiments that are large [[BR]] d) Easy to setup basic set of measurements (towards unified experiment environment) [[BR]] Issue: what are basic set of measurements? [[BR]] Answer: at least "passive host measurements", and "mesh of pings between nodes" [[BR]] e) Easy to mix-and-match tools, as go from step to step, with automatic transfer of info between steps/tools [[BR]] f) Easy to configure and then visualize (graph) measurements, starting from a graphical (or tabular) view of the topology of resources [[BR]] Issue: But, starting from a starting from a graphical (or tabular) view of the topology of resources to configure desired measurements is not always best. For example, for a classroom exercise, may want a pre-defined experiment, with a pre-define measurement configuration. For example, for a large experiment with 100s of nodes, it is not practical or useful to drive measurements from a graphical view of the topology; want to do everything with a script. [[BR]] Answer: Should be easy to configure and then visualize (graph) measurements, but the mechanism will likely differ depending on the type of experiment: small (predefined), medium, large. Nonetheless, should provide a method to serve all three that is not cumbersome for the experimenter. For a large experiment, will need to be able to use a script. [[BR]] h) Ability to simultaneously run multiple measurement sessions (e.g., passive host, mesh of pings, intermittent iperf, and experiment specific), each with multiple measurements and multiple graphs, during an experiment [[BR]] i) Stay consistent with Spiral 5 GIMI SOW [[BR]] 3) Build workflow for GIMI to work on both InstaGENI and ExoGENI [[BR]] a) Start with browser, GENI Desktop, FLACK and OMNI, for InstaGENI [[BR]] b) Start with User Workspace, Flukes and OMNI for ExoGENI [[BR]] c) When can InstaGENI be used with ExoGENI? also GENI Desktop? [[BR]] d) Which browsers are supported? [[BR]] e) Which credentials are supported? [[BR]] f) Need detailed plan from team: Jeanne (Lead), Luisa, Ezra, Hussam, Ilia (or rep) and Cong [[BR]] g) Current InstaGENI configuration per attached figure; changes? [[BR]] h) Current ExoGENI configuration? [[BR]] i) Plan to complete: [[br]] By 4/15: Jeanne will have met with team, and provide report of DRAFT plan [[BR]] 4/16: Sub-team has met, made plan; see http://groups.geni.net/geni/wiki/IMInteroperability [[BR]] 4) Define configuration of v2 (multi-session) GIMI Portal service. [[BR]] a) On 4/2: Reconfirm that will continue as a multi-user persistent service, with account for each user, to retain advantages of a persistent sink for measurement data. However, encourage users to move completed measurements to iRODS, so that we do not end up with long-term storage of measurement data within GIMI Portal Service. [[BR]] b) Continue to include OML Server and Lab wiki (for orchestration, results, and documentation) [[BR]] c) Separate OML results, by experiment and user, per the defined method [[BR]] d) For each user, introduce multiple "windows" to support multiple measurement sessions (e.g., passive host, mesh of pings, intermittent iperf, and experiment specific) [[BR]] e) For each user, be able to include session with Lab wiki to use OMF script to orchestrate experiment services (independent of any measurements) [[BR]] f) Store results in iRODS, by user, following defined structure [[BR]] g) Realize workable authentication/authorization to iRODS; start with current proxycert arrangement [[BR]] On 4/15: Fraida suggests iticket, used in iarchive command; Shu agrees to review and test iticket functions [[Br]] On 4/22: Shu reviews iticket functions; see link. Looks like a good mechanism for a multi-session GIMI Portal h) Do we want to support link from GENI Desktop? or somehow include Lab wiki as a plug-in in the GENI Desktop? These could be a step towards a "consistent user interface". [[BR]] On 4/19: Jim Griffioen says not hard to add link from browser in GENI Desktop (new tab, or a floating window) to GIMI Portal [[br]] i) Plan to define configuration: On 4/4: We need a workable solution, not the “best” solution. We can’t wait for future authentication/authorization mechanisms. [[BR]] On 4/4: We need to have the bulk of the work done in 6-8 weeks (no later than 6/11), then cleanup towards GEC17. [[BR]] On 4/4: A small team will be organized to work on the problem, define multiple solutions, and pick the best we can do in the time allotted: Jeanne, Cong, Mike (lead), Shu, (also web services expert from RENCI?), Max. [[BR]] By 4/15, Mike will have met with team, and will provide a report, including: evaluation of current Lab wiki code; available options for authentication of user to iRODS (starting with current proxycert arrangement); overview of current structure, and early view of possible options [[BR]] On 4/19, Mike agrees to have small group (Mike, Cong, Divya, Jeanne and Harry) meet in Cambridge 5/2 - 5/3 to define multi-session GIMI Portal, including: [[BR]] 1) Multiple sessions, but no multiple accounts [[BR]] 2) How info is held in portal for each user, and configured there for the user (DB?) [[BR]] 3) How user authenticates with the portal (like GENI CH Portal to GENI Desktop) [[BR]] 4) How the portal is authorized to write to the user's account in iRODS (based on iticket) [[BR]] 5) All captured in descriptive text, drawings, and a detailed workflow [[BR]] 5) Design GIMI Portal service [[br]] a) Who? On 4/19, Mike agrees to have small group (Mike, Cong, Divya, Max, Jeanne and Harry) meet in Cambridge 5/20 - 5/21 (also 5/22?) Suggest: have Tom/Aaron present to represent GENI CH Portal [[BR]] b) Follows: definition defined 5/2 - 5/3 [[BR]] c) Includes: [[BR]] Thin server (Apache server?) [[BR]] OML Server [[BR]] User DB [[BR]] LabWiki [[BR]] 6) Add features to iRODS [[BR]] a) Add rules to process incoming descriptors in XML files, ans store info in iCAT [[BR]] b) Add rules to move object to public archive directory, and manage any changes [[BR]] c) Integrate archive service with handle service [[BR]] d) Plan: [[br]] By 4/15, Shu needs to summarize planned extensions, per his earlier document. [[BR]] By 4/29, All GENI Storage and Archive Service plans summarized on wiki page at http://groups.geni.net/geni/wiki/GSAS [[BR]] By 4/29, All reviews of descriptor and iRODS work for GENI Storage and Archive Service completed, and descritor document completed. [[BR]] On 4/29, Giridhar agrees to provide update to schema. [[BR]] On 4/29, Shu agrees to provide latest task list, and schedule. [[BR]] 7) Include easy way to configure measurements, for all types of experiments (small, medium and large) [[BR]] Issue: Currently, involves: resource request (with ORCA or AM API); post-boot script; OMF script. But these are not linked. So, all is easy for a pre-defined (small) experiment, but script writing is reuired for a medium experiment, and info must be manually transferred from resource request/post boot script to the OMF script. [[BR]] a) Consider best way to "add GIMI extensions" (modify request to add software image or packages, and add any necessary nodes) [[BR]] b) Consider best way to "instrumentize with GIMI" (use script to setup proxy certs, configure services, connect via XMPP, and then start services) [[BR]] c) Consider best way to drive OMF script (on a medium experiment) from the slice information, e.g., number of nodes, names of nodes, topology of nodes. [[BR]] Option 1: consider use of GENI Desktop, with plug-ins to "add GIMI extensions" and "instrumentize with GIMI"; this would be a step towards a "consistent user interface". [[BR]] Option 2: consider some way to automatically write OMF script based on parameters forwarded from resource assignment. Maybe, manifest info could be transferred via iRODS. If this could be done for a medium experiment, perhaps an extension to a large experiment would be identified. [[BR]] Option 3: particularly for a large experiment, perhaps could automatically discover the nodes that are up, and the measurements that have been configured in each node. This could verify an OMF script. [[BR]] Option 4: although we do not need to drive measurement setup from graphical view of topology, we could discover the measurements that have been setup, and present a graphical view of the measurements, along with a graphical view of the resource topology. This could be used to verify the OMF script mathces the slioce resources, etc. [[BR]] d) Plan: [[br]] On 040413, put off discussion of this topic, until other topics are understood; may have BBN summer interns work on some of these topics. [[BR]] 8) Key task list: [[br]] || '''ID''' || '''Description''' || '''Assignee''' || '''Due''' || '''Status''' || '''Notes''' || || 1 || Define, prototype, and integrate push from GENI Portal to iRODs to create user account || Tom/Aaron,Shu || || || || || 2 || Define, prototype, and integrate link from GENI Portal to iRODS with login prompt || Tom/Aaron, Shu || || || || || 3 || Define, prototype, and integrate link from GENI Portal to GIMI Portal, SSO || Tom/Aaron, NICTA || || || || || 4 || Define and prototype interface to config GIMI Portal server || Cong, NICTA, Jeanne || || || || || 5 || Add measure_extensions to rspec || Mike, Jeanne, Ilia || || || || || 6 || Test Flack with ExoGENI || Divya, Jeanne, Luisa || || || || || 7 || Design and write scripts for setup, release, and archive || Cong, Jeanne || || || || || 8 || Define GIMI Portal config and workflow || GIMI arch team || 5/15 || || || || 9 || Design GIMI Portal || GIMI design team || 5/21 || || || || 10 || Define directory and file structure for all info in iRODS || GIMI arch team || || || || === 4.1) v2 GIMI Design Issues and Tasks === || '''ID''' || '''Description''' || '''Who''' || '''Due''' || '''Status''' || '''Demos''' || '''Notes''' || || I1.1 || Could initialize script be run from within GENI CH, like OMNI is now run? || || || || || || || I1.2 || Experiment is named and started by initialize script, in EME; should there be an “Experiment Authority” to track experiment_ids, and if so where? || || || || || || || I1.3 || Is there a way to associate multiple Users with an experiment, to share control and data? should this be tied to users associated with project in CH? || || || || || || || || || || || || || || || || || || || || || || === 4.2) v2 GIMI Interoperability Issues and Tasks === || '''ID''' || '''Description''' || '''Who''' || '''Due''' || '''Status''' || '''Demos''' || '''Notes''' || || T2.1 || Add measure_extensions to request_rspec, and use to select I&M software || Mike, Jeanne, Ilia || || || || how is selection done? || || T2.2 || Integrate Flack and GIMI with InstaGENI || Divya, Jeanne, Luisa || || GEC17 || || S1 || || || T2.3 || Integrate Flack and GIMI with ExoGENI || Divya, Jeanne, Luisa || || GEC17 || || S1 || || || || || || || || || || || || || || || || || || === 4.3) v2 GIMI Init Script in Exper Mgmt Envir Issues and Tasks === || '''ID''' || '''Description''' || '''Who''' || '''Due''' || '''Status''' || '''Demos''' || '''Notes''' || || 1 || Identity and authentication of User in EME || || || || || || || 1a || Download [2] user_credentials (cert + priv key)from GENI CH Portal, store locally, verify can act as User || Cong || GEC17 || || Sprint x || || || 1b || Use script to install cred's so can use iclient to access iRODS || Cong || GEC17 || || Sprint x || || || 1c || Use iinit with username/password to access iRODS || Cong || GEC17 || || Sprint x || || || 1d || || || || || || || || 1e || || || || || || || || 2 || v2 Init Script for User in EME || || || || || || || 2a || Provide interactive script, to allow selection at run time || Cong || GEC17 || || Sprint x || || || 2b || Provide options for a new experiment, or an update. || Cong || GEC17 || || Sprint x || Could this be automatic?|| || 2c || Enter exp_name, derive [10] exper_id || Cong || GEC17 || || Sprint x || || || 2d || Get [9] project_id || Cong || GEC17 || || Sprint x || || || 2e || Acting as User, get list of slice_id's, select [3] slice_id for this experiment || Cong || GEC17 || || Sprint x || || || 2f || Acting as User, using OMNI, get [7] sliver_manifest's || Cong || GEC17 || || Sprint x || || || 2g || Parse [7] to get [4] slice_node_names || Cong || GEC17 || || Sprint x || || || 2h || Register slice, nodes to XMPP server || Cong || GEC17 || || Sprint x || || || 2i || Use default GIMI Portal, or override || Cong || GEC17 || || Sprint x || || || 2j || Acting as User, using iclient, setup GSAS (iRODS) structure, derive defaults for [12] and [13], or override; derive and load descriptors [11]; load tempalte OMF scripts; get iticket for experiment, for GIMI Portal Agent || Cong || GEC17 || || Sprint x || || || 2k || Acting as User, using iclient, push [7] sliver_manifest_rspecs and [8] descriptors to iRODS || Cong || GEC17 || || Sprint x || || || 2l || Acting as User, using RESTful client, push configs to GIMI Portal || Cong || GEC17 || || Sprint x || || || 2m || || Cong || GEC17 || || Sprint x || || || 2n || || Cong || GEC17 || || Sprint x || || || 2o || || Cong || GEC17 || || Sprint x || || || 2z || Merge Init Script into CH, and run from button(s) on CH Portal || Cong, Tom/Aaron || GEC18 || || || || v2 GIMI Portal task list: [[br]] || '''ID''' || '''Description''' || '''Who''' || '''Due''' || '''Status''' || '''Demos''' || '''Notes''' || || 3 || v2 GIMI Portal and Services || || || || || || || 3a || Include LabWiki, with EC, GUI (web) User interface based on OMF web service, and underlying state machines || Max || GEC17 || || Sprint x || || || 3b || Include OML Server, with inbound interface, and underlying postgreSQL DB || Max || GEC17 || || Sprint x || || || 3c || Include Experiment Registry, with RESTful config interface, and underlying tuple-store (based on "OMF_SFA AM") || Max || GEC17 || || Sprint x || || || 3d || On first config message from Init Script in EME, use presented GENI cred's to register user in Experiment Registry || Max || GEC17 || || Sprint x || || || 3e || On first and subsequent config messages from Init Script in EME, put all exper info into Exper Registry || Max || GEC17 || || Sprint x || || || 3f || Provide navigation from GENI CH Portal to GUI User interface, identifying User, login and verify against Exper Resgistry || Max, Tom/Aaron || GEC17 || || Sprint x || || || 3g || On login to GUI User Interface, present LabWiki page, and allow User to select from available experiments || Max || GEC17 || || Sprint x || || || 3h || On LabWiki page, allow user to select available or define new measurement_task, with descriptor, and load template/stored script from iRODS || Max || GEC17 || || Sprint x || || || 3i || On LabWiki page, allow User to add parameters to scripts || Max || GEC17 || || Sprint x || How can we link from [4] node names, or other info in silver_manifest ?|| || 3j || On LabWiki page, execute script to configure OML Clients, pushing IP address of this GIMI Portal OML server, exper-id and oml_id || Max || GEC17 || || Sprint x || || || 3k || Accept OML measurements into OML server, identified by exper_id and oml_id, and put into postgreSQL DB || Max || GEC17 || || Sprint x || || || 3l || On LabWiki page, execute script to pull data from postgreSQL DB, and provide graphs || Max || GEC17 || || Sprint x || || || 3m || On LabWiki page, allow User to pull meas data from postgreSQL DB for this experiment, meas_task, and push to iRODS as table, adding a descriptor; use iRODS target and ticket from Exper Registry || Max || GEC17 || || Sprint x || || || 3n || On LabWiki page, allow User to push to iRODS scripts and other info for this experiment, meas_task || Max || GEC17 || || Sprint x || || || 3o || On LabWiki page, allow user to select available or define new measurement task for new page, without stopping current tasks || Max || GEC17 || || Sprint x || || || 3 || x || Max || GEC17 || || Sprint x || || || 3 || x || Max || GEC17 || || Sprint x || || || 3 || x || Max || GEC17 || || Sprint x || || || 3x || On LabWiki page, create new scripts || Max || GEC18 || || Sprint x || || || 3y || On LabWiki page, drag and drop scripts || Max || GEC18 || || Sprint x || || || 3z || Navigation between LabWiki Pages || Max || GEC18 || || Sprint x || ||