[[PageOutline]] = Detailed test plan for IG-ADM-4: Emergency Stop Test = ''This page is GPO's working page for performing IG-ADM-4. It is public for informational purposes, but it is not an official status report. See [wiki:GENIRacksHome/InstageniRacks/AcceptanceTestStatus] for the current status of InstaGENI acceptance tests.'' ''Last substantive edit of this page: 2013-02-14'' = Page format = * The status chart summarizes the state of this test * The high-level description from test plan contains text copied exactly from the public test plan and acceptance criteria pages. * The steps contain things I will actually do/verify: * Steps may be composed of related substeps where I find this useful for clarity * Each step is identified as either "(prep)" or "(verify)": * Prep steps are just things we have to do. They're not tests of the rack, but are prerequisites for subsequent verification steps * Verify steps are steps in which we will actually look at rack output and make sure it is as expected. They contain a '''Using:''' block, which lists the steps to run the verification, and an '''Expect:''' block which lists what outcome is expected for the test to pass. = Status of test = See [wiki:GENIRacksHome/InstageniRacks/AcceptanceTestStatus#Legend] for the meanings of test states. || '''Step''' || '''State''' || '''Date completed''' || '''Open Tickets''' || '''Closed Tickets/Comments''' || || 1 || || || || || || 2A || || || || || || 2B || || || || || || 3A || || || || || || 3B || || || || || = High-level description from test plan = In this test, an Emergency Stop drill is performed on a sliver in the rack. == Procedure == * A site administrator reviews the Emergency Stop and sliver shutdown procedures, and verifies that these two documents combined fully document the campus side of the Emergency Stop procedure. * A second administrator (or the GPO) submits an Emergency Stop request to GMOC, referencing activity from a public IP address assigned to a compute sliver in the rack that is part of the test experiment. * GMOC and the first site administrator perform an Emergency Stop drill in which the site administrator successfully shuts down the sliver in coordination with GMOC. * GMOC completes the Emergency Stop workflow, including updating/closing GMOC tickets. == Criteria to verify as part of this test == FIXME: fill this in = Step 1 (prep): Site administrator reviews GMOC and InstaGENI sliver shutdown procedures = The site administrator should review the Emergency Stop procedure provided by the GMOC as well as the InstaGENI sliver shutdown procedure. The site administrator should identify parts of the procedure where they need to take action on the aggregate, and where they might need to interface with another party, such as the GMOC or an experimenter. This parts identified by the site administrator should be verified with the GMOC and with the InstaGENI team. = Step 2: (prep) GPO, GMOC, and InstaGENI team coordinate a time to run an ES test = The GPO will coordinate with parties at the GMOC and on the InstaGENI team to identify when an Emergency Stop test can be run. This test will focus primarily on the interactions with the site administator(s) and performing the procedures documented by the rack team, and a member of the GPO will act as the site administrator for this test. The site administrator must not communicate with the InstaGENI team or GMOC using outside channels, but other members of the GPO, the GMOC, and the InstaGENI team may communicate freely during the process. == Step 3 (verify): GMOC initiates test == '''Using:''' * If the rack IP requirements documentation for the rack exists: * Review that documentation and determine what IP to hostname mappings should exist for `192.1.242.128/25` * Otherwise: * Iterate with `instageni-ops` to determine the IP to hostname mappings to use for `192.1.242.128/25` '''Expect:''' * Reasonable IP-to-hostname mappings for 126 valid IPs allocated for InstaGENI use in `192.1.242.128/25` === Results of testing step 2A: 2012-12-20 === We discussed this via e-mail, and concluded that we should create these DNS entries in gpolab.bbn.com: {{{ ;; 192.1.242.128/25: InstaGENI rack ; Delegate the whole subdomain to boss.instageni.gpolab.bbn.com, with ; ns.emulab.net as a secondary. ns.instageni IN A 192.1.242.132 instageni IN NS ns.instageni instageni IN NS ns.emulab.net. }}} And these in 242.1.192.in-addr.arpa: {{{ ;; 192.1.242.129/25: instageni.gpolab.bbn.com (InstaGENI rack control network) ; Delegate a subdomain to boss.instageni.gpolab.bbn.com, and generate ; CNAMEs pointing to it. 129/25 IN NS ns.instageni.gpolab.bbn.com. 129/25 IN NS ns.emulab.net. $GENERATE 129-255 $ IN CNAME $.129/25.242.1.192.in-addr.arpa. }}} == Step 2B (prep): Insert IP-to-hostname mapping in DNS == * Fully populate `192.1.242.128/25` PTR entries in GPO lab DNS * Fully populate `instageni.gpolab.bbn.com` A entries in GPO lab DNS == Step 2C (verify): Test all PTR records == '''Using:''' * From a BBN desktop host: {{{ for lastoct in {129..255}; do host 192.1.242.$lastoct done }}} '''Expect:''' * All results look like: {{{ $lastoct.242.1.192.in-addr.arpa domain name pointer }}} and none look like: {{{ Host $lastoct.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) }}} === Results of testing step 2C: 2012-12-20 === Many addresses aren't defined: {{{ [13:46:15] jbs@anubis:/home/jbs +$ for lastoct in {129..255} ; do host 192.1.242.$lastoct ; done Host 129.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) 130.242.1.192.in-addr.arpa is an alias for 130.129/25.242.1.192.in-addr.arpa. 130.129/25.242.1.192.in-addr.arpa domain name pointer control.instageni.gpolab.bbn.com. 131.242.1.192.in-addr.arpa is an alias for 131.129/25.242.1.192.in-addr.arpa. 131.129/25.242.1.192.in-addr.arpa domain name pointer control-ilo.instageni.gpolab.bbn.com. 132.242.1.192.in-addr.arpa is an alias for 132.129/25.242.1.192.in-addr.arpa. 132.129/25.242.1.192.in-addr.arpa domain name pointer boss.instageni.gpolab.bbn.com. 133.242.1.192.in-addr.arpa is an alias for 133.129/25.242.1.192.in-addr.arpa. 133.129/25.242.1.192.in-addr.arpa domain name pointer ops.instageni.gpolab.bbn.com. 134.242.1.192.in-addr.arpa is an alias for 134.129/25.242.1.192.in-addr.arpa. 134.129/25.242.1.192.in-addr.arpa domain name pointer foam.instageni.gpolab.bbn.com. 135.242.1.192.in-addr.arpa is an alias for 135.129/25.242.1.192.in-addr.arpa. 135.129/25.242.1.192.in-addr.arpa domain name pointer flowvisor.instageni.gpolab.bbn.com. Host 136.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 137.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 138.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 139.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) 140.242.1.192.in-addr.arpa is an alias for 140.129/25.242.1.192.in-addr.arpa. 140.129/25.242.1.192.in-addr.arpa domain name pointer pc1.instageni.gpolab.bbn.com. 141.242.1.192.in-addr.arpa is an alias for 141.129/25.242.1.192.in-addr.arpa. 141.129/25.242.1.192.in-addr.arpa domain name pointer pc2.instageni.gpolab.bbn.com. 142.242.1.192.in-addr.arpa is an alias for 142.129/25.242.1.192.in-addr.arpa. 142.129/25.242.1.192.in-addr.arpa domain name pointer pc3.instageni.gpolab.bbn.com. 143.242.1.192.in-addr.arpa is an alias for 143.129/25.242.1.192.in-addr.arpa. 143.129/25.242.1.192.in-addr.arpa domain name pointer pc4.instageni.gpolab.bbn.com. 144.242.1.192.in-addr.arpa is an alias for 144.129/25.242.1.192.in-addr.arpa. 144.129/25.242.1.192.in-addr.arpa domain name pointer pc5.instageni.gpolab.bbn.com. Host 145.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 146.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 147.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 148.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 149.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 150.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 151.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 152.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 153.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 154.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 155.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 156.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 157.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 158.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 159.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 160.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 161.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 162.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 163.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 164.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 165.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 166.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 167.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 168.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 169.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 170.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 171.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 172.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 173.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 174.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 175.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 176.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 177.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 178.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 179.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 180.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 181.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 182.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 183.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 184.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 185.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 186.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 187.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 188.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 189.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 190.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 191.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 192.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 193.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 194.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 195.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 196.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 197.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 198.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 199.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 200.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 201.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 202.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 203.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 204.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 205.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 206.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 207.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 208.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 209.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 210.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 211.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 212.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 213.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 214.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 215.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 216.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 217.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 218.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 219.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 220.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 221.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 222.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 223.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 224.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 225.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 226.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 227.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 228.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 229.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 230.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 231.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 232.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 233.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 234.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 235.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 236.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 237.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 238.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 239.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 240.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 241.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 242.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 243.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 244.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 245.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 246.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 247.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 248.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 249.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 250.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 251.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 252.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 253.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 254.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) Host 255.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) }}} We think that's normal: The in-use ones are in DNS, the not-in-use ones aren't. I tried creating a VM with a public IP address, using this rspec: {{{ }}} According to my manifest rspec, I got {{{ }}} That hostname and IP address now resolve: {{{ [15:03:32] jbs@anubis:/home/jbs/rspecs/request +$ host pcvm2-3.instageni.gpolab.bbn.com pcvm2-3.instageni.gpolab.bbn.com has address 192.1.242.150 [15:03:34] jbs@anubis:/home/jbs/rspecs/request +$ host 192.1.242.150 150.242.1.192.in-addr.arpa is an alias for 150.129/25.242.1.192.in-addr.arpa. 150.129/25.242.1.192.in-addr.arpa domain name pointer pcvm2-3.instageni.gpolab.bbn.com. }}} After I delete my sliver: {{{ [15:03:58] jbs@anubis:/home/jbs/rspecs/request +$ omni -a $am deletesliver $slicename [* snip *] Result Summary: Deleted sliver urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs on unspecified_AM_URN at https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am INFO:omni: ============================================================ [15:04:43] jbs@anubis:/home/jbs/rspecs/request +$ host pcvm2-3.instageni.gpolab.bbn.com Host pcvm2-3.instageni.gpolab.bbn.com not found: 3(NXDOMAIN) [15:05:57] jbs@anubis:/home/jbs/rspecs/request +$ host 192.1.242.150 150.242.1.192.in-addr.arpa is an alias for 150.129/25.242.1.192.in-addr.arpa. 150.129/25.242.1.192.in-addr.arpa domain name pointer pcvm2-3.instageni.gpolab.bbn.com. }}} That second one still works because it's cached on my local nameserver; if I ask the source, it's gone: {{{ [15:32:13] jbs@ops.instageni.gpolab.bbn.com:/users/jbs +$ host 192.1.242.150 Host 150.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN) }}} So, I think this is fine: Records exist when they're in use, and not when they're not, and that's fine. = Step 3: GPO requests and receives administrator accounts = == Step 3A: GPO requests access to boss and ops nodes == '''Using:''' * Request accounts for GPO ops staffers on boss.instageni.gpolab.bbn.com and ops.instageni.gpolab.bbn.com * Chaos tries to SSH to chaos@boss.instageni.gpolab.bbn.com * Josh tries to SSH to jbs@boss.instageni.gpolab.bbn.com * Tim tries to SSH to tupty@boss.instageni.gpolab.bbn.com * Chaos tries to SSH to chaos@ops.instageni.gpolab.bbn.com * Josh tries to SSH to jbs@ops.instageni.gpolab.bbn.com * Tim tries to SSH to tupty@ops.instageni.gpolab.bbn.com * Chaos tries to run a minimal command as sudo on boss: {{{ sudo whoami }}} * Chaos tries to run a minimal command as sudo on ops: {{{ sudo whoami }}} '''Verify:''' * Logins succeed for Chaos, Josh, and Tim on both nodes * The commands work: {{{ $ sudo whoami root }}} === Results of testing step 3A: 2012-12-20 === I followed the procedure at https://users.emulab.net/trac/protogeni/wiki/RackAdminAccounts#AdminAccountsinEmulab to join the emulab-ops project, and once the Utah folks approved that and made an admin, I was able to log in to boss and ops, and use sudo: {{{ [15:50:40] jbs@anubis:/home/jbs +$ ssh ops.instageni.gpolab.bbn.com sudo whoami root [15:50:50] jbs@anubis:/home/jbs +$ ssh boss.instageni.gpolab.bbn.com sudo whoami root }}} I asked Chaos and Tim to follow the procedure at that URL as well, and they did, and I approved their accounts by following the procedure at https://users.emulab.net/trac/protogeni/wiki/RackAdminAccounts#AddingmoreadminaccountstoEmulab, and they confirmed that they could log in to boss and ops. === Results of testing step 3A: 2012-05-15 === ''Note: This test was run on the Utah rack, where only Chaos has an account. So Tim and Josh will not be testing, and the hosts to test are `boss.utah.geniracks.net` and `ops.utah.geniracks.net`.'' * Chaos successfully used public-key login and sudo from a BBN subnet (128.89.68.0/23) to boss: {{{ capybara,[~],11:39(0)$ ssh chaos@boss.utah.geniracks.net Last login: Tue May 15 07:29:07 2012 from capybara.bbn.co Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 8.3-RC1 (XEN) #0: Tue Mar 13 16:27:12 MDT 2012 Welcome to FreeBSD! Need to see the calendar for this month? Simply type "cal". To see the whole year, type "cal -y". -- Dru > bash boss,[~],09:39(0)$ sudo whoami root }}} * Chaos successfully used public-key login and sudo from a BBN subnet (128.89.68.0/23) to ops: {{{ capybara,[~],11:40(0)$ ssh chaos@ops.utah.geniracks.net Last login: Sat May 12 15:41:57 2012 from capybara.bbn.co Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 8.3-RC1 (XEN) #0: Tue Mar 13 16:27:12 MDT 2012 Welcome to FreeBSD! ops,[~],09:40(0)$ sudo whoami root }}} == Step 3B: GPO requests access to FOAM VM == * Request accounts for GPO ops staffers on foam.instageni.gpolab.bbn.com * Chaos tries to SSH to chaos@foam.instageni.gpolab.bbn.com * Josh tries to SSH to jbs@foam.instageni.gpolab.bbn.com * Tim tries to SSH to tupty@foam.instageni.gpolab.bbn.com * Chaos tries to run a minimal command as sudo on foam: {{{ sudo whoami }}} '''Verify:''' * Logins succeed for Chaos, Josh, and Tim on the FOAM VM * The command works: {{{ $ sudo whoami root }}} === Results of testing step 3B: 2012-12-20 === I was named as the site admin in the site survey, and was given an account on the FOAM VM. I was able to log in and use sudo: {{{ [15:57:46] jbs@anubis:/home/jbs +$ ssh foam.instageni.gpolab.bbn.com sudo whoami root }}} I then created accounts for Chaos and Tim, following the procedure at https://users.emulab.net/trac/protogeni/wiki/RackAdminAccounts#AdminAccountsonInstaGeniRacks. I got their public keys from their Emulab accounts, and put them into chaos.keys and tupty.keys in my homedir, and then: {{{ sudo /usr/local/bin/mkadmin.pl chaos chaos.keys sudo /usr/local/bin/mkadmin.pl tupty tupty.keys }}} They then confirmed that they could log in, and run 'sudo whoami'. === Results of testing step 3B: 2012-07-04 === ''Note: This test was run on the Utah rack." * Chaos can SSH to foam.utah.geniracks.net: {{{ $ ssh foam.utah.geniracks.net Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-24-generic x86_64) * Documentation: https://help.ubuntu.com/ Last login: Tue Jul 3 12:57:20 2012 from capybara.bbn.com foam,[~],09:33(0)$ }}} * Chaos can sudo on foam: {{{ foam,[~],09:33(0)$ sudo whoami root }}}