[[PageOutline]] = EG-ADM-7: Documentation Review Test = This page captures status for the test case EG-ADM-7, which verifies the availability of Documentation for the ExoGENI rack. For overall status see the [wiki:GENIRacksHome/ExogeniRacks/AcceptanceTestStatus ExoGENI Acceptance Test Status] page. ''Last Updated: 2013-02-26'' = Test Status = This section captures the status for each step in the acceptance test plan. || '''Step''' || '''State''' || ''' Ticket ''' || '''Comments''' || ||1. ExoGENI Rack Installation Documentation || [[Color(green,Pass)]] || || || ||2. ExoGENI Rack Part List Documentation || [[Color(green,Pass)]]|| || || ||3. ExoGENI Software Components Identification Documentation || [[Color(orange,Partial)]] || exoticket:144 || || ||4. ExoGENI and OS Upgrade Plans ||[[Color(green,Pass)]] || exoticket:146 || || ||5. ExoGENI Software Components Functional Description and Access to Source code || [[Color(orange,Partial)]]|| || || ||6. ExoGENI Rack Resources Usage and Policy Documentation || [[Color(red,Fail)]]|| || || ||7. ExoGENI Rack Resources Administration || [[Color(orange,Partial)]] || || || ||8. ExoGENI Rack Administrative procedure for user accounts || [[Color(red,Fail)]] ||exoticket:145 || || ||9. ExoGENI Rack Administrative procedure for rack operations || [[Color(red,Fail)]]|| || || ||10. ExoGENI Rack Administrative procedure for rack scheduled shutdown || [[Color(red,Fail)]]|| || || ||11. ExoGENI Rack Administrative procedure for Emergency Stop. ||[[Color(#63B8FF,In Progress)]] ||exoticket:143|| || ||12. ExoGENI Rack Administrative procedure for health status || [[Color(orange,Partial)]] || || || ||13. ExoGENI Rack Site Operations documentation || [[Color(red,Fail)]]|| || || ||14. Site Specific ExoGENI Deployment Data || [[Color(orange,Partial)]] || || [[BR]] || '''State Legend''' || '''Description''' || || [[Color(green,Pass)]] || Documentation review completed and met all criteria || || [[Color(#98FB98,Pass: most criteria)]] || Documentation review completed and met most criteria. Exceptions documented || || [[Color(red,Fail)]] || Documentation review completed and failed to meet criteria || || [[Color(yellow,Complete)]] || Documentation review completed but will require re-execution due to expected changes || || [[Color(orange,Partial)]] || Minimal Documentation was found || || [[Color(#63B8FF,In Progress)]] || Currently under review || [[BR]] = ExoGENI Documentation Tests = The ExoGENI Documentation Tests focus on documentation review, and apply the following criteria for each document: * The document has been provided in a public location, where public locations are https://wiki.exogeni.net, and https://geni-orca.renci.org * The document contains the required information. * The documented information appears to be accurate. Documents to be reviewed are listed below. == 1. ExoGENI Rack Installation Documentation == '''The goal of this evaluation:''' To find pre-installation document that lists specific minimum requirements for all site-provided services for potential rack sites (e.g. space, number and type of power plugs, number and type of power circuits, cooling load, public addresses, NLR or Internet2 layer2 connections, etc.). This document should also list all standard expected rack interfaces (e.g. 10GBE links to at least one research network). '''Evaluation Results:''' [[Color(green,Pass)]] In the [https://wiki.exogeni.net/doku.php?id=public:hardware:start ExoGENI Rack Details] page there are pages that details the hardware: - [https://wiki.exogeni.net/doku.php?id=public:hardware:headnode:start Head Node] - [https://wiki.exogeni.net/doku.php?id=public:hardware:workernode:start Worker Nodes] - [https://wiki.exogeni.net/doku.php?id=public:hardware:iscsi:start iSCSI] - [https://wiki.exogeni.net/doku.php?id=public:hardware:network:start Network] - [https://wiki.exogeni.net/doku.php?id=public:hardware:rack:start Rack] - [https://wiki.exogeni.net/doku.php?id=public:hardware:rack_layout Rack Layout] - [https://wiki.exogeni.net/doku.php?id=public:hardware:power Power] == 2. ExoGENI Rack Part List Documentation == '''The goal of this evaluation:''' To find a summary ExoGENI Rack parts list, including vendor part numbers for "standard" equipment intended for all sites (e.g. a VM server) and per-site equipment options (e.g. transceivers, PDUs etc.), if any. This document should also indicate approximately how much headroom, if any, remains in the standard rack PDUs' power budget to support other equipment that sites may add to the rack. '''Evaluation Results:''' [[Color(green,Pass)]] There is not one document, but a series of parts lists pages were found at the http://www.exogeni.net site: * [https://wiki.exogeni.net/doku.php?id=public:hardware:network:start ExoGENI Network] * [https://wiki.exogeni.net/doku.php?id=public:hardware:rack:start ExoGENI Rack] * [https://wiki.exogeni.net/doku.php?id=public:hardware:workernode:start Worker Node] * [https://wiki.exogeni.net/doku.php?id=public:hardware:headnode:start Head Node] * [https://wiki.exogeni.net/doku.php?id=public:hardware:iscsi:start ExoGENI iSCSI] Power requirements are listed in the [https://wiki.exogeni.net/doku.php?id=public:hardware:rack:start ExoGENI Rack], [https://wiki.exogeni.net/doku.php?id=public:hardware:workernode:start Worker Node], and [https://wiki.exogeni.net/doku.php?id=public:hardware:headnode:start Head Node] pages. PDU requirements are specified in the [https://wiki.exogeni.net/doku.php?id=public:hardware:rack:start ExoGENI Rack] page. Transceivers are defined as site specific. Readers are instructed to confirm their site needs before purchasing any equipment. A [https://svn.exogeni.net/repos/config/diagrams/ExoGENI_Cabling_Diagrams.pdf ExoGENI Connection diagram] is available on the [https://wiki.exogeni.net/doku.php?id=public:hardware:rack:start ExoGENI Rack] that documents node wiring. == 3. ExoGENI Software Components Identification Documentation == '''The goal of this evaluation:''' Find a procedure for identifying the software versions and system file configurations running on a rack, and how to get information about recent changes to the rack software and configuration. '''Evaluation Results:''' [[Color(orange,Partial)]] See exoticket:143 == 4.ExoGENI and OS Upgrade Plans == '''The goal of this evaluation:''' Find documentation that explains how and when ExoGENI software and OS updates can be performed on a rack, including plans for notification and update if important security vulnerabilities in rack software are discovered. '''Evaluation Results:''' [[Color(green,Pass)]] See exoticket:146 == 5. ExoGENI Software Components Functional Description and Access to Source code == '''The goal of this evaluation:''' Find documentation that provides a description of the ExoGENI software running on a standard rack, and explanation of how to get access to the source code of each piece of standard GENI software. '''Evaluation Results:''' [[Color(orange,Partial)]] Source can be found at the https://geni-orca.renci.org, but it is not obvious how to map it to the ExoGENI rack components. There is the components overview in the wiki.exogeni.net location at the [https://wiki.exogeni.net/doku.php?id=public:software:start ExoGENI Software] page which provides a detailed overview of the ORCA components, but there are few pointers to source: - [https://wiki.exogeni.net/doku.php?id=private:configuration:orca:xmpp XMPP] page provides pointer to the [https://wiki.exogeni.net/doku.php?id=private:configuration:orca:xmpp blowhole] software, with instruction for installation and configuration. - [https://wiki.exogeni.net/doku.php?id=private:configuration:openflow:start !OpenFlow] page provides pointers to !FloodLight version 0.82) download and modified ORCA 3.1 [https://geni-orca.renci.org/trac/browser/orca/branches/3.1-extended/handlers/providers/resources/scripts scripts]. - [https://wiki.exogeni.net/doku.php?id=private:monitoring:gmoc GMOC] page provides link to [https://geni-orca.renci.org/trac/browser/orca-external/pubsub-daemon blowhole] code. == 6. ExoGENI Rack Resources Usage and Policy Documentation == '''The goal of this evaluation:''' Find documentation that provides a description of all the GENI experimental resources within the rack, and what policy options exist for each, including: how to configure rack nodes as bare metal vs. VM server, what options exist for configuring automated approval of compute and network resource requests and how to set them, how to configure rack aggregates to trust additional GENI slice authorities, and whether it is possible to trust local users within the rack. '''Evaluation Results:''' [[Color(red,Fail)]] No documentation was found for usage and policy in the known public locations, thus unable to verify the content. == 7. ExoGENI Rack Resources Administration == '''The goal of this evaluation:''' Find documentation that provides a description of the expected state of all the GENI experimental resources in the rack, including how to determine the state of an experimental resource and what state is expected for an unallocated bare metal node. '''Evaluation Results:''' [[Color(orange,Partial)]] The following '''private''' pages provide insight into rack resources administration: - [https://wiki.exogeni.net/doku.php?id=private:configuration:start ExoGENI rack configuration] - [https://wiki.exogeni.net/doku.php?id=private:management:start Rack Management] - [https://wiki.exogeni.net/doku.php?id=private:monitoring:start Monitoring] - [https://wiki.exogeni.net/doku.php?id=private:operation:start Operational Procedures] - [https://wiki.exogeni.net/doku.php?id=private:links:start useful administrative links] Found various smaller documents that start to define the available resources such as: * [https://geni-orca.renci.org/trac/wiki/deploy-sm Deploy SM] page. * [https://wiki.exogeni.net/doku.php?id=public:experimenters:resource_types:start available OS for Experimenter VMs] == 8. ExoGENI Rack Administrative procedure for user accounts == '''The goal of this evaluation:''' Find documentation that provides a procedure for creating new site administrator and operator accounts. '''Evaluation Results:''' [[Color(red,Fail)]] See exoticket:145 == 9. ExoGENI Rack Administrative procedure for rack operations == '''The goal of this evaluation:''' Find documentation that provides a procedure for changing IP addresses for all rack components. '''Evaluation Results:''' [[Color(red,Fail)]] Unable to find any procedure for modifying IP addresses for all rack components. == 10. ExoGENI Rack Administrative procedure for rack scheduled shutdown == '''The goal of this evaluation:''' Find documentation that provides a procedure for cleanly shutting down an entire rack in case of a scheduled site outage. '''Evaluation Results:''' [[Color(red,Fail)]] A procedure is being worked on at the [https://wiki.exogeni.net/doku.php?id=private:operation:emergency_stop Emergency Stop] page. There is an [http://groups.geni.net/geni/wiki/GENIRacksHome/ExogeniRacks/Administration#Emergencydisconnect Emergency disconnect] documented at the GPO [http://groups.geni.net/geni/wiki/GENIRacksHome/ExogeniRacks/Administration ExoGENI Administration] page. ExoGENI team needs to review review and make part of their procedures. == 11. ExoGENI Rack Administrative procedure for Emergency Stop. == '''The goal of this evaluation:''' Find documentation that provides a procedure for performing a shutdown operation on any type of sliver on a rack, in support of an Emergency Stop request. '''Evaluation Results:''' [[Color(#63B8FF,In Progress)]] See exoticket:143 == 12. ExoGENI Rack Administrative procedure for health status == '''The goal of this evaluation:''' Find documentation that provides a procedure for performing comprehensive health checks for a rack (or, if those health checks are being run automatically, how to view the current/recent results). '''Evaluation Results:''' [[Color(orange,Partial)]] No health check procedures found. The private [https://wiki.exogeni.net/doku.php?id=private:monitoring:start monitoring] page provides links for: - ExoGENI [https://wiki.exogeni.net/doku.php?id=private:monitoring:links Nagios installations] - setting up [https://wiki.exogeni.net/doku.php?id=private:monitoring:nagios_ec2 Nagios EC2] - [https://wiki.exogeni.net/doku.php?id=private:monitoring:gmoc Submitting data to GMOC] == 13. ExoGENI Rack Site Operations documentation == '''The goal of this evaluation:''' Find documentation that provides a Technical plan for handing off primary rack operations to site operators at all sites. '''Evaluation Results:''' [[Color(red,Fail)]] No documentation was found that detailed a Rack Operations Hand-off plan. == 14. Site Specific ExoGENI Deployment Data == '''The goal of this evaluation:''' Find documentation that provides Per-site deployment details/documentation. This documentation should be prepared before sites are installed and kept updated after installation to reflect any changes or upgrades after delivery. Text, network diagrams, wiring diagrams and labeled photos are all acceptable for site documents. Per-site documentation should include the following items for each site: 1. Part numbers and quantities of PDUs, with NEMA input power connector types, and an inventory of which equipment connects to which PDU. 2. Physical network interfaces for each control and data plane port that connects to the site's existing network(s), including type, part numbers, maximum speed etc. (eg. 10-GB-SR fiber) 3. Public IP addresses allocated to the rack, including: number of distinct IP ranges and size of each range, hostname to IP mappings which should be placed in site DNS, whether the last-hop routers for public IP ranges subnets sit within the rack or elsewhere on the site, and what firewall configuration is desired for the control network. 4. Data plane network connectivity and procedures for each rack, including core backbone connectivity and documentation, switch configuration options to set for compatibility with the L2 core, and the site and rack procedures for connecting non-rack-controlled VLANs and resources to the rack data plane. A network diagram is highly recommended (See existing !OpenFlow meso-scale network diagrams on the GENI wiki for examples.) '''Evaluation Results:''' [[Color(orange,Partial)]] No site information was found in the public pages, but the private ExoGENI wiki area has a [https://wiki.exogeni.net/doku.php?id=private:sites:start Sites] page captures each rack site. All site sub-pages consistently showed that the following are being tracked: "Site Address", , "Site Contacts", "Public IP Space", and "Private IP Space". Some sites sub-pages inconsistently may have any one of these topics: "Rack Requirements", "Head Node Interfaces", "Public Internet Connectivity", "Management Switch Layout", "Dataplane Switch Layout", "Out-of-band Connectivity", "iSCI Addresses" Suggestion: Creating a consistent template for each sub-page for gathering site specific data may help ensure consistency.