[[PageOutline]] = GIMI v2 Goals, Plan, Tasks and Status = == 1) GIMI v2 Goals == a) GIMI (also GEMINI) to work with both ExoGENI and InstaGENI [[BR]] b) Working toward consistent user experience, both racks, both sets of I&M tools [[BR]] c) Implement basic workflow per http://groups.geni.net/geni/wiki/GeniExperiments#a5StepsinBasicGENIExperimentTutorialTestWorkflowHarry and http://groups.geni.net/geni/wiki/TestTutorialExperimentWorkflow [[BR]] Issue: what range of workflows must be supported? small (classroom exercise); medium (research experiment); large (long-running service, with opt-in) [[BR]] Answer: eventually all of these, but with an emphasis on small and medium at this time; should have a way to extend medium to large; Sol 4 calls for shakedown experiments that are large [[BR]] d) Easy to setup basic set of measurements (towards unified experiment environment) [[BR]] Issue: what are basic set of measurements? [[BR]] Answer: at least "passive host measurements", and "mesh of pings between nodes" [[BR]] e) Easy to mix-and-match tools, as go from step to step, with automatic transfer of info between steps/tools [[BR]] f) Easy to configure and then visualize (graph) measurements, starting from a graphical (or tabular) view of the topology of resources [[BR]] Issue: But, starting from a starting from a graphical (or tabular) view of the topology of resources to configure desired measurements is not always best. For example, for a classroom exercise, may want a pre-defined experiment, with a pre-define measurement configuration. For example, for a large experiment with 100s of nodes, it is not practical or useful to drive measurements from a graphical view of the topology; want to do everything with a script. [[BR]] Answer: Should be easy to configure and then visualize (graph) measurements, but the mechanism will likely differ depending on the type of experiment: small (predefined), medium, large. Nonetheless, should provide a method to serve all three that is not cumbersome for the experimenter. For a large experiment, will need to be able to use a script. [[BR]] h) Ability to simultaneously run multiple measurement sessions (e.g., passive host, mesh of pings, intermittent iperf, and experiment specific), each with multiple measurements and multiple graphs, during an experiment [[BR]] i) Stay consistent with Spiral 5 GIMI SOW [[BR]] == 2) GIMI v2 Team == Mike Zink (UMass Amherst) Cong Wang (UMass Amherst) Divya Bhat (UMass Amherst) Max Ott (NICTA) Ilia Baldine (RENCI) Shu Huang (RENCI) Fraida Fund (NYU Poly) Deniz Gurkan (UH) Harry Mussman (BBN) Jeanne Ohren (BBN) Keleigh O'Neil (BBN) == 3) GIMI v2 Meetings/Conference Calls == 040113; 4pm EDT 040413; 4:30pm EDT 041513; 4pm EDT 042213; 4pm EDT; discuss iticket for iRODS 042913; 4pm EDT; discuss planning meeting 5/2- 5/3 for GIMI Portal; discuss interop status; discuss GSAS status 050213 - 050313; define GIMI Portal configuration 050913; review workflow, and plan to define GIMI Portal 051313; 2-4pm EDT; demo of interop, review workflow 052013 - 052113; review goals; review and update workflow; define entities and interfaces; define init script; define GIMI portal 052913; 4pm EDT; review documentation of workflow and designs; agree on tasks 061013; 2-4pm EDT; demo; end of sprint 1 (S1) 062413; 4pm EDT; demo; end of sprint 2 (S2) 070813; 2-4pm EDT; demo; end of sprint 3 (S2) 072113 - 072313 is GEC17; v2 software released after testing; integrated tutorials presented == 4) GIMI v2 Plan == * Started 032013 * Revised 032913 * Revised 040213 after 040113 call, and after GEMINI call on 040213 * Revised 040513 after discussion with Mike on 040413 * Revised after 041513 and 042213 calls, and discussions with Mike. * Revised after 042913 call * Revised after 052113 meeting === 4.1) Reference Figures === See [http://groups.geni.net/geni/attachment/wiki/GIMIv2tasks/052813_GIMI_Figures.vsd Figures in visio] [[br]] See [http://groups.geni.net/geni/attachment/wiki/GIMIv2tasks/Visio-052813_GIMI_Figures.pdf Figures in pdf] [[br]] * 1) GIMI v2 entities, interfaces [[br]] * 2a) Detailed GIMI v2 workflow (pp1/2) [[br]] * 2b) Detailed GIMI v2 workflow (pp2/2) [[br]] * 3) GIMI Portal [[br]] * 3b) GIMI Portal Config (multi account) [[br]] * 4) v2 GIMI I&M Tools Config [[br]] * 4b) GIMI I&M Tools Config Spiral 5 [[br]] * 5) OML Result Isolation [[br]] * 6) iRODS Config [[br]] * 7a) A&A: to iRODS with GSI [[br]] * 7b) A&A: to iRODS with ticket [[br]] * 7c) A&A: CH_Portal to GENI Desktop [[br]] * 8) WiMAX site config Spiral 5 [[br]] === 4.2) GIMI v2 Key Parameters === * [1] user_identity * [2] user_credentials * [3] slice_id * [4] slice_node_names * [5] slice_node_meas_extens * [6] slice_request_rspec * [7] sliver_manifest_rspecs * [8] sliver_manifest_descriptors * [9] project_id * [10] experiment_id * [11] proj_exper_step_descriptors * [12] user_irods_home_directory * [13] user_irods_target_directory * [14] user_irods_target_iticket * [15] measure_task_id * [16] meas_task_omf_script * [17] meas_task_omf_script_param * [18] meas_task_descriptor * [19] meas_data_object * [20] meas_data_descriptor === 4.3) Build workflow for GIMI to work on both InstaGENI and ExoGENI === a) Start with browser, GENI Desktop, FLACK and OMNI, for InstaGENI [[BR]] b) Start with User Workspace, Flukes and OMNI for ExoGENI [[BR]] c) When can InstaGENI be used with ExoGENI? also GENI Desktop? [[BR]] d) Which browsers are supported? [[BR]] e) Which credentials are supported? [[BR]] f) Need detailed plan from team: Jeanne (Lead), Luisa, Ezra, Hussam, Ilia (or rep) and Cong [[BR]] g) Current InstaGENI configuration per attached figure; changes? [[BR]] h) Current ExoGENI configuration? [[BR]] i) Plan to complete: [[br]] By 4/15: Jeanne will have met with team, and provide report of DRAFT plan [[BR]] 4/16: Sub-team has met, made plan; see http://groups.geni.net/geni/wiki/IMInteroperability [[BR]] === 4.4) Define configuration of v2 (multi-session) GIMI Portal service === a) On 4/2: Reconfirm that will continue as a multi-user persistent service, with account for each user, to retain advantages of a persistent sink for measurement data. However, encourage users to move completed measurements to iRODS, so that we do not end up with long-term storage of measurement data within GIMI Portal Service. [[BR]] b) Continue to include OML Server and Lab wiki (for orchestration, results, and documentation) [[BR]] c) Separate OML results, by experiment and user, per the defined method [[BR]] d) For each user, introduce multiple "windows" to support multiple measurement sessions (e.g., passive host, mesh of pings, intermittent iperf, and experiment specific) [[BR]] e) For each user, be able to include session with Lab wiki to use OMF script to orchestrate experiment services (independent of any measurements) [[BR]] f) Store results in iRODS, by user, following defined structure [[BR]] g) Realize workable authentication/authorization to iRODS; start with current proxycert arrangement [[BR]] On 4/15: Fraida suggests iticket, used in iarchive command; Shu agrees to review and test iticket functions [[Br]] On 4/22: Shu reviews iticket functions; see link. Looks like a good mechanism for a multi-session GIMI Portal h) Do we want to support link from GENI Desktop? or somehow include Lab wiki as a plug-in in the GENI Desktop? These could be a step towards a "consistent user interface". [[BR]] On 4/19: Jim Griffioen says not hard to add link from browser in GENI Desktop (new tab, or a floating window) to GIMI Portal [[br]] i) Plan to define configuration: On 4/4: We need a workable solution, not the “best” solution. We can’t wait for future authentication/authorization mechanisms. [[BR]] On 4/4: We need to have the bulk of the work done in 6-8 weeks (no later than 6/11), then cleanup towards GEC17. [[BR]] On 4/4: A small team will be organized to work on the problem, define multiple solutions, and pick the best we can do in the time allotted: Jeanne, Cong, Mike (lead), Shu, (also web services expert from RENCI?), Max. [[BR]] By 4/15, Mike will have met with team, and will provide a report, including: evaluation of current Lab wiki code; available options for authentication of user to iRODS (starting with current proxycert arrangement); overview of current structure, and early view of possible options [[BR]] On 4/19, Mike agrees to have small group (Mike, Cong, Divya, Jeanne and Harry) meet in Cambridge 5/2 - 5/3 to define multi-session GIMI Portal, including: [[BR]] 1) Multiple sessions, but no multiple accounts [[BR]] 2) How info is held in portal for each user, and configured there for the user (DB?) [[BR]] 3) How user authenticates with the portal (like GENI CH Portal to GENI Desktop) [[BR]] 4) How the portal is authorized to write to the user's account in iRODS (based on iticket) [[BR]] 5) All captured in descriptive text, drawings, and a detailed workflow [[BR]] === 4.5) Design GIMI Portal service === a) Who? On 4/19, Mike agrees to have small group (Mike, Cong, Divya, Max, Jeanne and Harry) meet in Cambridge 5/20 - 5/21 (also 5/22?) Suggest: have Tom/Aaron present to represent GENI CH Portal [[BR]] b) Follows: definition defined 5/2 - 5/3 [[BR]] c) Includes: [[BR]] Thin server (Apache server?) [[BR]] OML Server [[BR]] User DB [[BR]] !LabWiki [[BR]] === 4.6) Add features to iRODS === a) Add rules to process incoming descriptors in XML files, and store info in iCAT [[BR]] b) Add rules to move object to public archive directory, and manage any changes [[BR]] c) Integrate archive service with handle service [[BR]] d) Plan: [[br]] By 4/15, Shu needs to summarize planned extensions, per his earlier document. [[BR]] By 4/29, All GENI Storage and Archive Service plans summarized on wiki page at http://groups.geni.net/geni/wiki/GSAS [[BR]] By 4/29, All reviews of descriptor and iRODS work for GENI Storage and Archive Service completed, and descritor document completed. [[BR]] On 4/29, Giridhar agrees to provide update to schema. [[BR]] On 4/29, Shu agrees to provide latest task list, and schedule. [[BR]] === 4.7) Include easy way to configure measurements, for all types of experiments (small, medium and large) === Issue: Currently, involves: resource request (with ORCA or AM API); post-boot script; OMF script. But these are not linked. So, all is easy for a pre-defined (small) experiment, but script writing is reuired for a medium experiment, and info must be manually transferred from resource request/post boot script to the OMF script. [[BR]] a) Consider best way to "add GIMI extensions" (modify request to add software image or packages, and add any necessary nodes) [[BR]] b) Consider best way to "instrumentize with GIMI" (use script to setup proxy certs, configure services, connect via XMPP, and then start services) [[BR]] c) Consider best way to drive OMF script (on a medium experiment) from the slice information, e.g., number of nodes, names of nodes, topology of nodes. [[BR]] Option 1: consider use of GENI Desktop, with plug-ins to "add GIMI extensions" and "instrumentize with GIMI"; this would be a step towards a "consistent user interface". [[BR]] Option 2: consider some way to automatically write OMF script based on parameters forwarded from resource assignment. Maybe, manifest info could be transferred via iRODS. If this could be done for a medium experiment, perhaps an extension to a large experiment would be identified. [[BR]] Option 3: particularly for a large experiment, perhaps could automatically discover the nodes that are up, and the measurements that have been configured in each node. This could verify an OMF script. [[BR]] Option 4: although we do not need to drive measurement setup from graphical view of topology, we could discover the measurements that have been setup, and present a graphical view of the measurements, along with a graphical view of the resource topology. This could be used to verify the OMF script mathces the slioce resources, etc. [[BR]] d) Plan: [[br]] On 040413, put off discussion of this topic, until other topics are understood; may have BBN summer interns work on some of these topics. [[BR]] == 5) GIMI v2 Tasks and Status == === 5.1) v2 GIMI Design Issues and Tasks === See [http://groups.geni.net/gimi GIMI ticket tracking wiki] [[br]] || '''ID''' || '''Description''' || '''Who''' || '''Due''' || '''Status''' || '''Demos''' || '''Notes''' || || I1.1 || Experiment is named and started by initialize script, in EME; should there be an “Experiment Authority” to track experiment_ids, and if so where? in the CH? || || || || || || || I1.2 || Is there a way to associate multiple Users with an experiment, to share control and data? should this be tied to users associated with project in CH? needs to be carefully considered || || || || || || || || || || || || || || || || || || || || || || === 5.2) v2 GIMI Interoperability Issues and Tasks === See [http://groups.geni.net/geni/wiki/IMInteroperability Interoperability of I&M Tools on GENI Racks wiki page] [[br]] See [http://groups.geni.net/gimi GIMI ticket tracking wiki] [[br]] === 5.3) v2 GIMI Init Script in Exper Mgmt Envir (EME) Issues and Tasks === See [http://groups.geni.net/gimi GIMI ticket tracking wiki] [[br]] || '''ID''' || '''Description''' || '''Who''' || '''Due''' || '''Status''' || '''Demos''' || '''Notes''' || || T3.1 || Download [2] user_creds (cert + priv key)from GENI CH Portal (web interface), store locally in EME, verify can act as User || Cong, Jeanne, Keleigh || GEC17 || Completed || S1 || || || T3.3 || Use iinit with username/password to access iRODS, and setup iRODS env file (option 2 of 2) || Cong, Shu || GEC17 || ticket [http://groups.geni.net/gimi/ticket/5 #5] || S1 || || || T3.4 || Build GIMI v2 Init Script for User, to run in EME || Cong, Jeanne, Keleigh || GEC17 || ticket [http://groups.geni.net/gimi/ticket/4 #4] || S1, S2, S3 || See Fig 2a) and 2b) || || T3.5 || Make init script interactive, to allow selection by User at run time; how? || Cong, Jeanne, Keleigh || GEC17 || ticket [http://groups.geni.net/gimi/ticket/4 #4] || S1 || || || T3.6 || Provide options with init script for a new experiment, or an update. || Cong, Jeanne, Keleigh || GEC17 || || S2 || Could this be automatic?|| || T3.7 || Enter exp_name, derive [10] exper_id = user_name-exper_name-unix_timestamp, at start of experiment || Cong, Jeanne, Keleigh || GEC17 || ticket [http://groups.geni.net/gimi/ticket/4 #4] || S1 || See Fig 5) for OML Results Isolation || || T3.8 || Get [9] project_id from CH (how?), or enter || Cong, Jeanne, Keleigh, Tom/Aaron || GEC17 || || S2 || || || T3.9 || Acting as User, get list of slice_id's from CH SA, then select [3] slice_id for this experiment || Cong, Jeanne, Keleigh, Tom/Aaron || GEC17 || ticket [http://groups.geni.net/gimi/ticket/7 #7] || S1 || || || T3.10 || Acting as User, using OMNI, get [7] sliver_manifest_rspecs from AMs || Cong, Jeanne, Keleigh, Tom/Aaron || GEC17 || ticket [http://groups.geni.net/gimi/ticket/8 ticket #8] || S1 || || || T3.11 || Parse [7] to get [4] slice_node_names || Cong, Jeanne, Keleigh || GEC17 || || S1 || Do we want to parse [7] and get many more parameters? like GENI Desktop parser is doing? and then push them to Experiment Registry in GIMI Portal?|| || T3.12 || Register slice, nodes to XMPP server (if needed) || Cong, Jeanne, Keleigh || GEC17 || || S1 || Not needed if using OMF ver 5.4.2 || || T3.13 || Select default GIMI Portal, or override || Cong, Jeanne, Keleigh || GEC17 || || S2 || Now using default || || T3.14 || Acting as User, using iclient, setup GSAS (iRODS) structure for this experiment, derive defaults for [12] user_irods_home_directory and [13] user_irods_target_directory, or override; derive and load [11] proj_exper_step_descriptors; load template OMF scripts (where are these?); get iticket for experiment, assigned to selected GIMI Portal Agent || Cong, Jeanne, Keleigh, Shu, Antoine || GEC17 || tickets [http://groups.geni.net/gimi/ticket/9 #9], [http://groups.geni.net/gimi/ticket/10 #10], [http://groups.geni.net/gimi/ticket/11 #11]|| S1, S2 || Need summary of IRODS structure; see Fig 7b) for use of iticket || || T3.15 || Acting as User, using iclient, push [7] sliver_manifest_rspecs and [8] descriptors to iRODS || Cong, Keleigh || GEC17 || ticket [http://groups.geni.net/gimi/ticket/12 #12] || S1 || || || T3.16 || Acting as User, using RESTful client, push configs to GIMI Portal, including: [1] user_identity (from [2]), [10] experiment_id, [9] project_id, [3] slice_id, [4] slice_node_names, [7] sliver_manifest_rspecs, [12] user_irods_home_directory, [13] user_irods_target(exper)_directory, [14] user_irods_target(exper)_iticket || Cong, Jeanne, Keleigh, Max, Jack || GEC17 || ticket [http://groups.geni.net/gimi/ticket/13 #13] || S1, S2, S3 || Start with default values, and then replace with gathered values as available|| || T3.17 || Build v2 release script || Cong, Jeanne, Keleigh || GEC17 || ticket [http://groups.geni.net/gimi/ticket/6 #6] || S3 || See Fig 2b) || || || || || || || || || || || || || || || || || || T3.2 || Use script to install [2] user_cred's, so can use iclient to access iRODS (option 1 of 2) || Cong, Jeanne, Keleigh || || || || || || || Merge init script into CH, and run from button(s) on CH Portal, with available selections || Cong, Tom/Aaron || || || || || || || Build v2 bag script, to create object for archive || Cong, Jeanne, Keleigh, Shu || || || || || || || Build v2 archive script, to move object to public archive || Cong, Jeanne, Keleigh, Shu || || || || || === 5.4) v2 GIMI Portal: Experiment Registry === See [http://groups.geni.net/gimi GIMI ticket tracking wiki] [[br]] See [https://mytestbed.net/projects/labwiki/issues LabWiki ticket tracking site] [[br]] See [https://omf.mytestbed.net/projects/omf/issues OMF ticket tracking site] [[br]] See [https://oml.mytestbed.net/projects/oml/issues OML ticket tracking site] [[br]] || '''ID''' || '''Description''' || '''Who''' || '''Due''' || '''Status''' || '''Demos''' || '''Notes''' || || T4.1 || Include Experiment Registry in GIMI Portal, with RESTful config interface, and underlying tuple-store, starting with current "OMF_SFA AM" code || Cong, Max, Jack || GEC17 || ticket [http://groups.geni.net/gimi/ticket/14 #14] || S1 || See Fig 2a) and 3) Need to get latest code from Max. || || T4.2 || Define RESTful config interface/API , and integrate with interface in EME used by init script || Cong, Jeanne || GEC17 || ticket [http://groups.geni.net/gimi/ticket/15 #15] || S1 || || || T4.3 || On receipt of config messages from init script in EME, put user and experiment info (tuple) into Exper Registry || Cong, Max, Jack || GEC17 || ticket [http://groups.geni.net/gimi/ticket/16 #16] || S1, S2, S3 || || || T4.4 || Define Exper Registry interface/API to !LabWiki || GEC17 || Cong, Max, Jack || || S1 || || || || || || || || || || === 5.5) v2 GIMI Portal: OML Server === See [https://mytestbed.net/projects/labwiki/issues LabWiki ticket tracking site] [[br]] See [https://omf.mytestbed.net/projects/omf/issues OMF ticket tracking site] [[br]] See [https://oml.mytestbed.net/projects/oml/issues OML ticket tracking site] [[br]] || '''ID''' || '''Description''' || '''Who''' || '''Due''' || '''Status''' || '''Demos''' || '''Notes''' || || T5.1 || Include OML Server in GIMI Portal, with inbound OML Interface, and underlying postgreSQL DB || Steve, Olivier || GEC17 || || S1 || Now OML ver 2.9; consider 2.10? || || T5.2 || Accept OML measurements into OML server, into inbound OML Interface, identified by exper_id and oml_id, and put into postgreSQL DB || Steve, Olivier || GEC17 || || S1 || See Fig 5) on OML Results Isolation|| || || || || || || || || || || || || || || || || === 5.6) v2 GIMI Portal: !LabWiki === See [https://mytestbed.net/projects/labwiki/issues LabWiki ticket tracking site] [[br]] See [https://omf.mytestbed.net/projects/omf/issues OMF ticket tracking site] [[br]] See [https://oml.mytestbed.net/projects/oml/issues OML ticket tracking site] [[br]] || '''ID''' || '''Description''' || '''Who''' || '''Due''' || '''Status''' || '''Demos''' || '''Notes''' || || T6.1 || Include !LabWiki in GIMI Portal, with EC, with GUI (web) User interface based on OMF web service, and underlying state machines || Max, Jack || GEC17 || || S1, S2, S3 || Start with !LabWiki ver ?, based on OMF ver 5.4, and OMF Web ver ? || || T6.2 || Provide navigation from GENI CH Portal to GUI User Interface, identifying User, forwarding other information (what?). and allowing SSO; then login User, and then verify against User entry in Exper Registry || Max, Jack, Tom/Aaron || GEC17 || || S2 || Need to be able to select GIMI Portal from GENI CH Portal; how? || || T6.3 || On login to GUI User Interface, present !LabWiki page, and allow User to select from available experiments, as listed in Exper Registry || Max, Jack || GEC17 || || S1 || || || T6.4 || On !LabWiki page, for a given exper, allow User to select available or define new measurement_task, add descriptor, and then load template/stored script from iRODS, and then edit script || Max, Jack || GEC17 || || S1, S2, S3 || How can we link from [4] node names, or other info in silver_manifest? || || T6.5 || On !LabWiki page, allow User to add parameters to scripts || Max, Jack || GEC17 || || S1 || How can we link from [4] node names, or other info in silver_manifest? || || T6.6 || On !LabWiki page, allow User to execute script to configure OML Clients, pushing IP address of this GIMI Portal OML server, exper-id and oml_id || Max, Jack || GEC17 || || S1 || || || T6.7 || On !LabWiki page, allow User to execute script to orchestrate measurement_task, then pull data from postgreSQL DB, and provide graphs || Max, Jack, Olivier || GEC17 || || S1, S2, S3 || || || T6.8 || On !LabWiki page, allow User to pull, on demand, meas data from postgreSQL DB for this experiment and this meas_task, format as a table, add a descriptor; then use iclient to push artifact and descriptor files from GIMI Portal Agent to iRODS, using iRODS target and ticket from Exper Registry; GIMI Portal Agent is a registered iRODS user, is authenticated with cert/priv key, and can access the GENI User directories with the iticket. || Max, Jack, Shu || GEC17 || || S2, S3 || See Fig 7b) on use of iticket || || T6.9 || On !LabWiki page, allow User to push to iRODS scripts and other info for this experiment, meas_task, as stored in Experiment Registry || Max, Jack, Shu || GEC17 || || S3 || || || T6.10 || On !LabWiki page, allow user to select available or define new measurement task for new page, without stopping current tasks || Max, Jack || GEC17 || || S3 || || || T6.11 || On !LabWiki page, allow User to select, on demand, a particular experiment and meas_task, and stop them, without erasing measurement data in the DB || Max, Jack || GEC17 || || S2 || || || T6.12 || On !LabWiki page, allow User to select, on demand, a particular experiment and meas_task, that have been stopped, and erase measurement data in the DB || Max, Jack || GEC17 || || S3 || || || || || || || || || || || || || || || || || || || || Consider best way to drive OMF script (on a medium experiment) from the slice information, e.g., number of nodes, names of nodes, topology of nodes. || || || || || || || || On !LabWiki page, create new scripts, and then edit || Max, Jack || || || || || || || On !LabWiki page, drag and drop scripts, and then edit || Max, Jack || || || || || || || Navigation between !LabWiki Pages || Max, Jack || || || || || || || On !LabWiki page, allow user to retrieve all info on an experiment from iRODS, and again view results || Max, Jack || || || || || === 5.7) iRODS for v2 GIMI === Go to [http://groups.geni.net/geni/wiki/GSAS GENI Storage and Archive Service (GSAS) wiki page] [[br]] See [http://groups.geni.net/gimi GIMI ticket tracking wiki] [[br]]