Changes between Initial Version and Version 1 of GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-ADM-1

Timestamp:

05/07/12 19:15:01 (12 years ago)

Author:

chaos@bbn.com

Comment:

--

GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-ADM-1

@@ +1 @@
+[[PageOutline]]
+
+= Detailed test plan for IG-ADM-1: Rack Receipt and Inventory Test =
+
+''This page is GPO's working page for performing IG-ADM-1.  It is public for informational purposes, but it is not an official status report.  See [wiki:GENIRacksHome/InstageniRacks/AcceptanceTestStatus] for the current status of InstaGENI acceptance tests.''
+
+''Last substantive edit of this page: 2012-05-07''
+
+== Page format ==
+
+ * The status chart summarizes the state of this test
+ * The high-level description from test plan contains text copied exactly from the public test plan and acceptance criteria pages.
+ * The steps contain things i will actually do/verify:
+   * Steps may be composed of related substeps where i find this useful for clarity 
+   * Each step is identified as either "(prep)" or "(verify)":
+     * Prep steps are just things we have to do.  They're not tests of the rack, but are prerequisites for subsequent verification steps
+     * Verify steps are steps in which we will actually look at rack output and make sure it is as expected.  They contain a '''Using:''' block, which lists the steps to run the verification, and an '''Expect:''' block which lists what outcome is expected for the test to pass.
+
+== Status of test ==
+
+See [wiki:GENIRacksHome/InstageniRacks/AcceptanceTestStatus] for the meanings of test states.
+
+''Note: all steps of this test are blocked on the arrival of the BBN rack at BBN.  However, we plan to do preliminary testing of some steps using the rack at Utah.  For the time being, we need to differentiate between steps which are blocked until the BBN rack arrives, and steps which may be blocked from testing at Utah by some shorter-term requirement, as follows:''
+ * [[Color(yellow,Blocked-site)]]: A step which will not be tested on the Utah rack, and is blocked until the BBN site rack arrives.
+ * [[Color(orange,Blocked-Utah)]]: A step which will be tested on the Utah rack, and is blocked on a requirement for access or configuration of the Utah rack.
+
+|| '''Step''' || '''State'''                    || '''Date completed''' || '''Tickets''' || '''Comments'''                                                                 ||
+|| 1          || [[Color(yellow,Blocked-site)]] ||                      ||               || blocked on purchase and shipping of BBN rack                                   ||
+|| 2A         || [[Color(yellow,Blocked-site)]] ||                      ||               || blocked on 1                                                                   ||
+|| 2B         || [[Color(yellow,Blocked-site)]] ||                      ||               || blocked on 2A                                                                  ||
+|| 2C         || [[Color(yellow,Blocked-site)]] ||                      ||               || blocked on 2B                                                                  ||
+|| 3A         ||                                ||                      ||               || ready for preliminary testing using the Utah rack                              ||
+|| 3B         || [[Color(orange,Blocked-Utah)]] ||                      ||               || blocked on GPO access to the FOAM VM                                           ||
+|| 3C         || [[Color(orange,Blocked-Utah)]] ||                      ||               || blocked on GPO access to the infrastructure VM server node                     ||
+|| 3D         || [[Color(orange,Blocked-Utah)]] ||                      ||               || blocked on GPO access to the control and dataplane switches                    ||
+|| 3E         || [[Color(orange,Blocked-Utah)]] ||                      ||               || blocked on GPO access to OpenVZ (maybe just on sudo access to boss?)           ||
+|| 3F         || [[Color(orange,Blocked-Utah)]] ||                      ||               || blocked on GPO access to rack iLO                                              ||
+|| 3G         || [[Color(orange,Blocked-Utah)]] ||                      ||               || blocked on GPO sudo access on rack boss                                        ||
+|| 4A         || [[Color(yellow,Blocked-site)]] ||                      ||               || blocked on 1                                                                   ||
+|| 4B         || [[Color(yellow,Blocked-site)]] ||                      ||               || blocked on 1                                                                   ||
+|| 4C         || [[Color(orange,Blocked-Utah)]] ||                      ||               || blocked on GPO access to control and dataplane switches                        ||
+|| 4D         || [[Color(yellow,Blocked-site)]] ||                      ||               || blocked on 1                                                                   ||
+|| 5A         || [[Color(orange,Blocked-Utah)]] ||                      ||               || blocked on public IPs/hostnames for Utah infrastructure devices                ||
+|| 5B         || [[Color(orange,Blocked-Utah)]] ||                      ||               || blocked on a rack dataplane vlan being connected to the GENI mesoscale network ||
+|| 5C         || [[Color(orange,Blocked-Utah)]] ||                      ||               || blocked on information about rack monitoring solution                          ||
+|| 5D         || [[Color(orange,Blocked-Utah)]] ||                      ||               || blocked on information about rack monitoring solution                          ||
+|| 6A         ||                                ||                      ||               ||                                                                                ||
+|| 6B         ||                                ||                      ||               ||                                                                                ||
+|| 6C         ||                                ||                      ||               ||                                                                                ||
+
+== High-level description from test plan ==
+
+This "test" uses BBN as an example site by verifying that we can do all the things we need to do to integrate the rack into our standard local procedures for systems we host.
+
+=== Procedure ===
+
+ * InstaGENI and GPO power and wire the BBN rack
+ * GPO configures the instageni.gpolab.bbn.com DNS namespace and 192.1.242.128/25 IP space, and enters all public IP addresses used by the rack into DNS.
+ * GPO requests and receives administrator accounts on the rack and GPO sysadmins receive read access to all InstaGENI monitoring of the rack.
+ * GPO inventories the physical rack contents, network connections and VLAN configuration, and power connectivity, using our standard operational inventories.
+ * GPO, InstaGENI, and GMOC share information about contact information and change control procedures, and InstaGENI operators subscribe to GENI operations mailing lists and submit their contact information to GMOC.
+ * GPO reviews the documented parts list, power requirements, physical and logical network connectivity requirements, and site administrator community requirements, verifying that these documents should be sufficient for a new site to use when setting up a rack.
+
+=== Criteria to verify as part of this test ===
+
+ * VI.02. A public document contains a parts list for each rack. (F.1)
+ * VI.03. A public document states the detailed power requirements of the rack, including how many PDUs are shipped with the rack, how many of the PDUs are required to power the minimal set of shipped equipment, the part numbers of the PDUs, and the NEMA input connector type needed by each PDU. (F.1)
+ * VI.04. A public document states the physical network connectivity requirements between the rack and the site network, including number, allowable bandwidth range, and allowed type of physical connectors, for each of the control and dataplane networks. (F.1)
+ * VI.05. A public document states the minimal public IP requirements for the rack, including: number of distinct IP ranges and size of each range, hostname to IP mappings which should be placed in site DNS, whether the last-hop routers for public IP ranges subnets sit within the rack or elsewhere on the site, and what firewall configuration is desired for the control network. (F.1)
+ * VI.06. A public document states the dataplane network requirements and procedures for a rack, including necessary core backbone connectivity and documentation, any switch configuration options needed for compatibility with the L2 core, and the procedure for connecting non-rack-controlled VLANs and resources to the rack dataplane. (F.1)
+ * VI.07. A public document explains the requirements that site administrators have to the GENI community, including how to join required mailing lists, how to keep their support contact information up-to-date, how and under what circumstances to work with Legal, Law Enforcement and Regulatory(LLR) Plan, how to best contact the rack vendor with operational problems, what information needs to be provided to GMOC to support emergency stop, and how to interact with GMOC when an Emergency Stop request is received. (F.3, C.3.d)
+ * VI.14. A procedure is documented for creating new site administrator and operator accounts. (C.3.a)
+ * VII.01. Using the provided documentation, GPO is able to successfully power and wire their rack, and to configure all needed IP space within a per-rack subdomain of gpolab.bbn.com. (F.1)
+ * VII.02. Site administrators can understand the physical power, console, and network wiring of components inside their rack and document this in their preferred per-site way. (F.1)
+
+== Step 1 (prep): InstaGENI and GPO power and wire the BBN rack ==
+
+This step covers the physical delivery of the rack to BBN, the transport of the rack inside BBN to the GPO lab, and the cabling, powering, and initial configuration of the rack.
+
+== Step 2: Configure and verify DNS ==
+
+=== Step 2A (verify): Find out what IP-to-hostname mapping to use ===
+
+'''Using:'''
+ * If the rack IP requirements documentation for the rack exists:
+   * Review that documentation and determine what IP to hostname mappings should exist for `192.1.242.128/25`
+ * Otherwise:
+   * Iterate with `instageni-ops` to determine the IP to hostname mappings to use for `192.1.242.128/25`
+
+'''Expect:'''
+ * Reasonable IP-to-hostname mappings for 126 valid IPs allocated for InstaGENI use in `192.1.242.128/25`
+
+=== Step 2B (prep): Insert IP-to-hostname mapping in DNS ===
+
+ * Fully populate `192.1.242.128/25` PTR entries in GPO lab DNS
+ * Fully populate `instageni.gpolab.bbn.com` A entries in GPO lab DNS
+
+=== Step 2C (verify): Test all PTR records ===
+
+'''Using:'''
+ * From a BBN desktop host:
+{{{
+for lastoct in {129..255}; do
+host 192.1.242.$lastoct
+done
+}}}
+
+'''Expect:'''
+ * All results look like:
+{{{
+$lastoct.242.1.192.in-addr.arpa domain name pointer <something reasonable>
+}}}
+ and none look like:
+{{{
+Host $lastoct.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
+}}}
+
+== Step 3: GPO requests and receives administrator accounts ==
+
+=== Step 3A: GPO requests access to boss and ops nodes ===
+
+'''Using:'''
+ * Request accounts for GPO ops staffers on boss.instageni.gpolab.bbn.com and ops.instageni.gpolab.bbn.com
+ * Chaos tries to SSH to chaos@boss.instageni.gpolab.bbn.com
+ * Josh tries to SSH to jbs@boss.instageni.gpolab.bbn.com
+ * Tim tries to SSH to tupty@boss.instageni.gpolab.bbn.com
+ * Chaos tries to SSH to chaos@ops.instageni.gpolab.bbn.com
+ * Josh tries to SSH to jbs@ops.instageni.gpolab.bbn.com
+ * Tim tries to SSH to tupty@ops.instageni.gpolab.bbn.com
+ * Chaos tries to run a minimal command as sudo on boss:
+{{{
+sudo whoami
+}}}
+ * Chaos tries to run a minimal command as sudo on ops:
+{{{
+sudo whoami
+}}}
+
+'''Verify:'''
+ * Logins succeed for Chaos, Josh, and Tim on both nodes
+ * The commands work:
+{{{
+$ sudo whoami
+root
+}}}
+
+=== Step 3B: GPO requests access to FOAM VM ===
+
+ * Request accounts for GPO ops staffers on foam.instageni.gpolab.bbn.com
+ * Chaos tries to SSH to chaos@foam.instageni.gpolab.bbn.com
+ * Josh tries to SSH to jbs@foam.instageni.gpolab.bbn.com
+ * Tim tries to SSH to tupty@foam.instageni.gpolab.bbn.com
+ * Chaos tries to run a minimal command as sudo on foam:
+{{{
+sudo whoami
+}}}
+
+'''Verify:'''
+ * Logins succeed for Chaos, Josh, and Tim on the FOAM VM
+ * The command works:
+{{{
+$ sudo whoami
+root
+}}}
+
+=== Step 3C: GPO requests access to infrastructure server ===
+
+ * Request accounts for GPO ops staffers on the VM server node which runs boss, ops, and foam
+ * Chaos tries to SSH to the VM server node
+ * Josh tries to SSH to the VM server node
+ * Tim tries to SSH to the VM server node
+ * Chaos tries to run a minimal command as sudo on the VM server node:
+{{{
+sudo whoami
+}}}
+
+'''Verify:'''
+ * Logins succeed for Chaos, Josh, and Tim on the host
+ * The command works:
+{{{
+$ sudo whoami
+root
+}}}
+
+=== Step 3D: GPO requests access to network devices ===
+
+'''Using:'''
+ * Request accounts for GPO ops staffers on the InstaGENI rack control and dataplane network devices from instageni-ops
+
+'''Verify:'''
+ * I know what hostname or IP address to login to to reach each of the control and dataplane switches
+ * I know what source IPs are allowed to remotely access the control and dataplane switches via SSH
+ * I can successfully perform those logins at least once
+ * I can successfully run a few test commands to verify enable mode:
+{{{
+show running-config
+show mac-address-table
+}}}
+
+=== Step 3E: GPO requests access to shared OpenVZ nodes ===
+
+'''Using:'''
+ * Determine an experimental host which is currently configured as a shared OpenVZ node
+ * From boss.instageni.gpolab.bbn.com, try to SSH to the node
+ * On the node, try to run a minimal command as sudo:
+{{{
+sudo whoami
+}}}
+
+'''Verify:'''
+ * Login to the OpenVZ host is successful
+ * Access to the node is as root and/or it is possible to run a command via sudo
+
+=== Step 3F: GPO requests access to iLO remote management interfaces for experimental nodes ===
+
+'''Using:'''
+ * GPO requests access to the experimental node iLO management interfaces from instageni-ops
+ * Determine how to use these interfaces to access remote control and remote console interfaces for experimental nodes 
+ * For each experimental node in the BBN rack:
+   * Access the iLO interface and view status information 
+   * View the interface for remotely power-cycling the node
+   * Launch the remote console for the node
+
+'''Verify:'''
+ * GPO is able to determine the procedure for accessing the iLO interfaces
+ * Login to each iLO succeeds
+ * The remote power-cycle interface exists and appears to be usable (don't try power-cycling any nodes during this test)
+ * Launching the remote console at each iLO succeeds
+
+=== Step 3G: GPO gets access to allocated bare metal nodes by default ===
+
+'''Prerequisites:'''
+ * A bare metal node is available for allocation by InstaGENI
+ * Someone has successfully allocated the node for a bare metal experiment
+
+'''Using:'''
+ * From boss, try to SSH into root on the allocated worker node
+
+'''Verify:'''
+ * We find out the IP address/hostname at which to reach the allocated worker node
+ * Login to the node using root's SSH key succeeds
+
+== Step 4: GPO inventories the rack based on our own processes == 
+
+=== Step 4A: Inventory and label physical rack contents ===
+
+'''Using:'''
+ * Enumerate all physical objects in the rack
+ * Use available rack documentation to determine the correct name of each object
+ * If any objects can't be found in public documentation, compare to internal notes, and iterate with InstaGENI
+ * Physically label each device in the rack with its name on front and back 
+ * Inventory all hardware details for rack contents on OpsHardwareInventory
+ * Add an ascii rack diagram to OpsHardwareInventory
+
+'''Verify:'''
+ * Public documentation and/or rack diagrams identify all rack objects
+ * There is a public parts list which matches the parts we received
+ * We succeed in labelling the devices and adding their hardware details and locations to our inventory
+
+=== Step 4B: Inventory rack power requirements ===
+
+'''Using:'''
+ * Add rack circuit information to OpsPowerConnectionInventory
+
+'''Verify:'''
+ * We succeed in locating and documenting information about rack power circuits in use
+
+=== Step 4C: Inventory rack network connections ===
+
+'''Using:'''
+ * Add all rack ethernet and fiber connections and their VLAN configurations to OpsConnectionInventory
+ * Add static rack OpenFlow datapath information to OpsDpidInventory
+
+'''Verify:'''
+ * We are able to identify and determine all rack network connections and VLAN configurations
+ * We are able to determine the OpenFlow configuration of the rack dataplane switch
+
+=== Step 4D: Verify government property accounting for the rack ===
+
+'''Using:'''
+ * Receive a completed DD1149 form from InstaGENI
+ * Receive and inventory a property tag number for the BBN InstaGENI rack
+
+'''Verify:'''
+ * The DD1149 paperwork is complete to BBN government property standards
+ * We receive a single property tag for the rack, as expected
+
+== Step 5: Configure operational alerting for the rack ==
+
+=== Step 5A: GPO installs active control network monitoring ===
+
+'''Using:'''
+ * Add a monitored control network ping from ilian.gpolab.bbn.com to the boss VM
+ * Add a monitored control network ping from ilian.gpolab.bbn.com to the ops VM
+ * Add a monitored control network ping from ilian.gpolab.bbn.com to the foam VM
+ * Add a monitored control network ping from ilian.gpolab.bbn.com to the infrastructure VM host
+ * Add a monitored control network ping from ilian.gpolab.bbn.com to the control switch's management IP
+ * Add a monitored control network ping from ilian.gpolab.bbn.com to the dataplane switch's management IP
+
+'''Verify:'''
+ * Active monitoring of the control network is successful
+ * Each monitored IPs is successfully available at least once
+
+=== Step 5B: GPO installs active shared dataplane monitoring ===
+
+'''Using:'''
+ * Add a monitored dataplane network ping from a lab dataplane test host on vlan 1750 to the rack dataplane
+ * If necessary, add an openflow controller to handle traffic for the monitoring subnet
+
+'''Verify:'''
+ * Active monitoring of the dataplane network is successful
+ * The monitored IP is successfully available at least once
+
+=== Step 5C: GPO gets access to monitoring information about the BBN rack ===
+
+'''Using:'''
+ * GPO determines what monitoring tool InstaGENI will make available for site administrators
+ * GPO successfully accesses and views status data about the BBN rack
+
+'''Verify:'''
+ * I can see general data about all devices in the BBN rack
+ * I can see detailed information about any services checked
+
+=== Step 5D: GPO receives e-mail about BBN rack alerts ===
+
+'''Using:'''
+ * Request e-mail notifications for BBN rack problems to be sent to GPO ops
+ * Collect a number of notifications
+ * Inspect three representative messages
+
+'''Verify:'''
+ * E-mail messages about rack problems are received
+ * For each inspected message, i can determine:
+   * The affected device
+   * The affected service
+   * The type of problem being reported
+   * The duration of the outage
+
+== Step 6: Setup contact info and change control procedures ==
+
+=== Step 6A: InstaGENI operations staff should subscribe to response-team ===
+
+'''Using:'''
+ * Ask InstaGENI operators to subscribe `instageni-ops@flux.utah.edu` (or individual operators) to `response-team@geni.net`
+
+'''Verify:'''
+ * This subscription has happened.  On daulis:
+{{{
+sudo -u mailman /usr/lib/mailman/bin/find_member -l response-team utah.edu
+}}}
+
+=== Step 6B: InstaGENI operations staff should provide contact info to GMOC ===
+
+'''Using:'''
+ * Ask InstaGENI operators to submit primary and secondary e-mail and phone contact information to GMOC
+
+'''Verify:'''
+ * Browse to [https://gmoc-db.grnoc.iu.edu/protected/], login, and look at the "organizations" table.  Make sure either:
+   * The Utah contact information is up-to-date and includes instageni-ops and some reasonable phone numbers
+   * A new InstaGENI contact has been added
+
+=== Step 6C: Negotiate an interim change control notification procedure ===
+
+'''Using:'''
+ * Ask InstaGENI operators to notify either instageni-design@geni.net or gpo-infra@geni.net about planned outages and changes.
+
+'''Verify:'''
+ * InstaGENI agrees to send notifications about planned outages and changes.
+