[[PageOutline]] = I&M Design Topics at GEC16 = Tuesday, March 19, 10:30am - 12noon [[BR]] Room: City Creek [[BR]] Session leader: Harry Mussman (GENI Project Office, Raytheon BBN Technologies) [[BR]] == Description == This I&M session will focus on three key issues of interest to both the GIMI and GEMINI projects, as they continue to work towards a comprehensive set of I&M tools for GENI experimenters and for infrastructure measurement. [[BR]] We will first review a goal to move towards a "unified experiment environment". [[BR]] Then, we will discuss these topics: * Support for GIMI and GEMINI I&M tools on both ExoGENI and InstaGENI racks * Use of an instrumentize script, driven by rspec extension, to configure/init all I&M tools * Options for further measurement data analysis and presentation For each topic, we will have a review of current practice, and a discussion of how to move forward. [[BR]] It is hoped that some of the discussions at this session may inspire proposals that will be submitted to GENI Solicitation 4. Finally, we will form teams to continue the discussions. [[BR]] == Pre-Requisites == None [[BR]] == Agenda == This is a tentative agenda, which may change. [[BR]] == 1) Introductions == 8:30am [[BR]] == 2) Towards a Unified Experiment Environemnt == 8:35am [[BR]] Marshall Brinn (GPO) [[BR]] 10min [http://groups.geni.net/geni/attachment/wiki/GEC16Agenda/IMDesignTopics/GEC16%20Strawman%20Combined%20Tool.pptx slides:] What does it mean to move towards a "unified experiment environment"? [[BR]] == 3) Support GIMI and GEMINI I&M tools on both ExoGENI and InstaGENI racks == 8:45am === Introduction === Harry Mussman (GPO) [[BR]] GENI experiment management Step 3: Obtain slice, install software * Now: GIMI loaded (and tested) on ExoGENI, with Flukes * Now: GEMINI loaded (and tested) on InstaGENI, with FLACK * Next: be able to use GEMINI on ExoGENI, and GIMI on InstaGENI * Suggest: set goal to begin testing all combinations by 5/1; good support by GEC17 === Lessons learned from using GIMI tools on ExoGENI racks === Cong Wang (UMass Amherst) [[BR]] 5min [http://groups.geni.net/geni/attachment/wiki/GEC16Agenda/IMDesignTopics/GIMI-gec16.pptx slides]: current config; key techniques; known challenges [[BR]] Configuration: [[BR]] * No extra node * Persistent XMPP server, with reachable IP address * Presistent OML server (p/o GIMI portal), with reachable IP address === Lessons learned from using GEMINI tools on InstaGENI racks === Hussam Nasir (Univ Kentucky) [[BR]] 5min [http://groups.geni.net/geni/attachment/wiki/GEC16Agenda/IMDesignTopics/Lessons%20learned%20from%20using%20GEMINI%20tools%20on%20Instageni.pptx slides]: current config; key techniques; known challenges [[BR]] Configuration: [[BR]] * Only support for VMs; largest to date 20 nodes; to date, all on one aggregate * One extra VM for "Global Node" * Add desired measurements to request rspec * Compatible image for quick setup === Discussion === Cong Wang (UMass Amherst; Hussam Nasir (Univ Kentucky); Louisa Nevers (GPO); Jeanne Ohren (GPO) [[BR]] 15min How do we move towards using GEMINI on ExoGENI, and GIMI on InstaGENI? [[BR]] Discuss: * Suggested approaches * Key techniques * Likely challenges * Goal to begin testing all combinations by 5/1; good support by GEC17; are there any roadblocks? 1) Issues: * images * rspecs, since slight differences between ExoGENI and InstaGENI * workflow differences 2) Helpful: * list of reference experiments * repository of rspecs 3) Start testing soon! Team to continue discussion: Cong Wang (UMass Amherst; Hussam Nasir (Univ Kentucky); Ilia Baldine (RENCI); Rob Ricci (Univ Utah); Louisa Nevers (GPO); Jeanne Ohren (GPO) == 4) Use of instrumentize script, driven by rspec extension, to configure/initialize I&M tools == 9:10am === Overview === Harry Mussman (GPO) [[BR]] GENI experiment management Step 4: Configure/init I&M and exp app * Now: GIMI using post boot scripts, OMF * Now: GEMINI using instrumentize script, with choices based on rspec extensions (ignored by AggMgr's); also used to add GN (an extra node, one per Agg, for common GEMINI services) * Next: move towards common approach, to better unify experiment environment * Suggest: both move towards script, driven by rspec ext === Current GIMI use of post boot scripts, OMF, to configure/inti services === Cong Wang (UMass Amherst) [[BR]] 5min [http://groups.geni.net/geni/attachment/wiki/GEC16Agenda/IMDesignTopics/GIMI-gec16.pptx slides]: key techniques; known challenges [[BR]] Post-boot scripts can do anything a shell script can do Prime use: connect OMF RC on node into messaging system, so that OMF EC can talk to node === Current GEMINI use of instrumentize script, to configure/init services === Hussam Nasir (Univ Kentucky) [[BR]] 5min [http://groups.geni.net/geni/attachment/wiki/GEC16Agenda/IMDesignTopics/GEMINI%20Scripts.pptx slides]: key techniques; known challenges [[BR]] Now: Python scripts, that ssh into nodes === Discussion === Cong Wang (Umass Amherst) ; Hussam Nasir (Univ Kentucky); Marshall Brinn (GPO); Jeanne Ohren (GPO) [[BR]] 15min [[BR]] How do we move towards a common approach to configure and initialize I&M tools, to better unify experiment environment? [[BR]] Strawman: * move towards instrumentize script, driven by rspec ext Conclusions: 1) Can do equivalent of post-boot script in InstaGENI 2) Instrumentize script uses ssh, checks config, then sets config 3) Would like to be able to easily setup a basic set of measurements 4) Not clear how to resolve; next: * draw out both flows * discuss * try to unify Team for further discussion: Cong Wang (UMass Amherst); Mike Zink (UMass Amherst); Hussam Nasir (Univ Kentucky); Harry Mussman (GPO); Jeanne Ohren (GPO) == 5) Options for further measurement data analysis and presentation == 9:35am === Introduction === Harry Mussman (GPO) [[BR]] GENI experiment management Step 6: Analyze * Now: Both GEMINI and GIMI push measurement data objects (files) to GENI Storage Service (iRODS) * Now: GEMINI provides dump of mongo DB, plus separate file of metadata * Now: GIMI (will) provide SQLdump of PostgreSQL DB, which includes metadata * Now: Both use portal on GENI Storage Service (iRODS) to curate files (objects) Both: Pull measurement data files from GENI Storage Service (iRODS) back to user workspace, where DBs can be reconstituted, and data pulled for further analysis and presentation. [[BR]] Both: Many options for further analysis and presentation, not being developed as part of tools. [[BR]] Next: Consider possible ways to unify data formats, analysis and presentation; what commonly available tools might be useful? [[BR]] === Current GIMI measurement data object schema === Stephen Glass (NICTA) [[BR]] 10min GIMI measurement data object schema: [[BR]] * OML moving to OML 6.0, which includes metadata * OML server has DB backend, SQlite or PostgreSGL * Version of DB stored in iRODS === Current GEMINI measurement data object schema === Ezra Kissel (Indiana Univ) [[BR]] 10min [http://groups.geni.net/geni/attachment/wiki/GEC16Agenda/IMDesignTopics/20130319-gemini-metadata.pptx slides]: GEMINI measurement data object schema [[BR]] GEMINI measurement data object schema: * maesurement data stored in Mongo DB * topology and metadata stored in UNIS * versions of DB, topology and metadata stored in iRODS === Discussion === Stephen Glass (NICTA); Ezra Kissel (Indiana Univ); Marshall Brinn (GPO); Fraida Fund (NYU Poly) [[BR]] 10min How could we unify further analysis and presentation? [[BR]] Conclusions: 1) Both GIMI and GEMINI store measurement data into iRODS as a DB, but with different metadata formats 2) Want an open format; do we need to merge into one open format, or jst accept two open formats? 3) Earlier effort: Emulab workbench; ref? 4) Current effort: Research Data Alliance, formed to unify stored results; ref? Team to continue discussion: Stephen Glass (NICTA); Max Ott (NICTA); Ezra Kissel (Indiana Univ); Marshall Brinn (GPO); Fraida Fund (NYU Poly) == 6) Adjourn == 12noon [[BR]] === I&M BoF Dinner === Please join us for a [http://groups.geni.net/geni/wiki/GEC16Agenda/BoF_WiMAX I&M BoF dinner] on Wednesday evening. Please indicate on this poll, if you will be attending. [[BR]]