Changes between Version 5 and Version 6 of GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-ADM-4


Ignore:
Timestamp:
02/15/13 11:04:40 (11 years ago)
Author:
tupty@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-ADM-4

    v5 v6  
    2323|| '''Step''' || '''State'''                    || '''Date completed''' || '''Open Tickets''' || '''Closed Tickets/Comments''' ||
    2424|| 1          ||           ||            ||                    ||   ||
    25 || 2A         ||           ||            ||                    ||   ||
    26 || 2B         ||           ||            ||                    ||   ||
    27 || 3A         ||           ||            ||                    ||   ||
    28 || 3B         ||           ||            ||                    ||   ||
     25|| 2          ||           ||            ||                    ||   ||
     26|| 3          ||           ||            ||                    ||   ||
     27|| 4          ||           ||            ||                    ||   ||
     28|| 5A         ||           ||            ||                    ||   ||
     29|| 5B         ||           ||            ||                    ||   ||
     30|| 5C         ||           ||            ||                    ||   ||
     31|| 6          ||           ||            ||                    ||   ||
    2932
    3033= High-level description from test plan =
     
    3437== Procedure ==
    3538
    36  * A site administrator reviews the Emergency Stop and sliver shutdown procedures, and verifies that these two documents combined fully document the campus side of the Emergency Stop procedure.
     39 * A site administrator reviews the Emergency Stop and sliver shut down procedures, and verifies that these two documents combined fully document the campus side of the Emergency Stop procedure.
    3740 * A second administrator (or the GPO) submits an Emergency Stop request to GMOC, referencing activity from a public IP address assigned to a compute sliver in the rack that is part of the test experiment.
    3841 * GMOC and the first site administrator perform an Emergency Stop drill in which the site administrator successfully shuts down the sliver in coordination with GMOC.
     
    4144== Criteria to verify as part of this test ==
    4245
    43 FIXME: fill this in
     46'''FIXME: fill this in'''
    4447
    45 = Step 1 (prep): Site administrator reviews GMOC and InstaGENI sliver shutdown procedures =
     48= Step 1 (prep): Site administrator reviews GMOC and InstaGENI sliver shut down procedures =
    4649
    47 The site administrator should review the Emergency Stop procedure provided by the GMOC as well as the InstaGENI sliver shutdown procedure.  The site administrator should identify parts of the procedure where they need to take action on the aggregate, and where they might need to interface with another party, such as the GMOC or an experimenter.  This parts identified by the site administrator should be verified with the GMOC and with the InstaGENI team.
     50The site administrator should review the Emergency Stop procedure provided by the GMOC as well as the InstaGENI sliver shut down procedure.  The site administrator should identify parts of the procedure where they need to take action on the aggregate, and where they might need to interface with another party, such as the GMOC or an experimenter.  This parts identified by the site administrator should be verified with the GMOC and with the InstaGENI team.
    4851
    4952= Step 2 (prep): GPO, GMOC, and InstaGENI team coordinate a time to run an ES test =
     
    5154The GPO will coordinate with parties at the GMOC and on the InstaGENI team to identify when an Emergency Stop test can be run.  This test will focus primarily on the interactions with the site administator(s) and performing the procedures documented by the rack team.  The following roles will need to be defined for this test:
    5255 * '''GMOC Coordinator''': person from the GMOC who coordinates the Emergency Stop activity on the GMOC's side
    53  * '''InstaGENI Contact''': person from the InstaGENI team who can be around if there are questions about the document or sliver shutdown procedure
     56 * '''InstaGENI Contact''': person from the InstaGENI team who can be around if there are questions about the document or sliver shut down procedure
    5457 * '''Emergency Stop Initiator''': GPO person who initiates an Emergency Stop request
    5558 * '''Experimenter''': GPO person who has created a sliver
     
    5861= Step 3 (prep): Experimenter sets up a slice =
    5962
    60 The experimenter will set up a slice that includes a sliver on the GPO InstaGENI rack.  The sliver should be a VM that is attached to the shared mesoscale VLAN, and it should be sending measurable traffic.
     63The experimenter will set up a slice that includes a sliver on the GPO InstaGENI rack.  The sliver should be a VM that is attached to the shared mesoscale VLAN, and it should be sending traffic that is visible through monitoring.
    6164
    6265= Step 4 (prep): Emergency Stop initiated =
     
    6467The Emergency Stop Initiator contacts the GMOC Coordinator to initiate an emergency stop request describing the slice URN.  The GMOC walks quickly walks through their procedure, skipping more formal steps as needed, in order to contact the aggregate operator primary contact.
    6568
    66 == Step 5 (verify): Site Administrator receives Emergency Stop request ==
     69== Step 5: Site Administrator receives Emergency Stop request ==
    6770
    68 
    69 
    70 
     71=== Step 5A (verify): Data passed from GMOC to Site Administrator is in expected format ===
    7172
    7273'''Using:'''
    73  * If the rack IP requirements documentation for the rack exists:
    74    * Review that documentation and determine what IP to hostname mappings should exist for `192.1.242.128/25`
    75  * Otherwise:
    76    * Iterate with `instageni-ops` to determine the IP to hostname mappings to use for `192.1.242.128/25`
     74 * Documented InstaGENI sliver shut down procedure
    7775
    78 '''Expect:'''
    79  * Reasonable IP-to-hostname mappings for 126 valid IPs allocated for InstaGENI use in `192.1.242.128/25`
     76'''Verify:'''
     77 * The GMOC sends a request with slice-specific or sliver-specific data in a format that can be fed into the shut down procedure
     78 * There is a step for the Site Administrator to acknowledge that the GMOC's request is being processed
    8079
    81 === Results of testing step 2A: 2012-12-20 ===
    82 
    83 We discussed this via e-mail, and concluded that we should create these DNS entries in gpolab.bbn.com:
    84 
    85 {{{
    86 ;; 192.1.242.128/25: InstaGENI rack
    87 
    88 ; Delegate the whole subdomain to boss.instageni.gpolab.bbn.com, with
    89 ; ns.emulab.net as a secondary.
    90 ns.instageni            IN      A       192.1.242.132
    91 instageni               IN      NS      ns.instageni
    92 instageni               IN      NS      ns.emulab.net.
    93 }}}
    94 
    95 And these in 242.1.192.in-addr.arpa:
    96 
    97 {{{
    98 ;; 192.1.242.129/25: instageni.gpolab.bbn.com (InstaGENI rack control network)
    99 
    100 ; Delegate a subdomain to boss.instageni.gpolab.bbn.com, and generate
    101 ; CNAMEs pointing to it.
    102 129/25               IN   NS      ns.instageni.gpolab.bbn.com.
    103 129/25               IN   NS      ns.emulab.net.
    104 $GENERATE 129-255 $  IN   CNAME   $.129/25.242.1.192.in-addr.arpa.
    105 }}}
    106 
    107 == Step 2B (prep): Insert IP-to-hostname mapping in DNS ==
    108 
    109  * Fully populate `192.1.242.128/25` PTR entries in GPO lab DNS
    110  * Fully populate `instageni.gpolab.bbn.com` A entries in GPO lab DNS
    111 
    112 == Step 2C (verify): Test all PTR records ==
     80=== Step 5B (verify): Shut down procedure can be followed to successfully shut down a sliver ===
    11381
    11482'''Using:'''
    115  * From a BBN desktop host:
    116 {{{
    117 for lastoct in {129..255}; do
    118 host 192.1.242.$lastoct
    119 done
    120 }}}
     83 * Documented InstaGENI sliver shut down procedure
     84 * Administrative tools to shut down a sliver
     85 * Available monitoring tools
    12186
    122 '''Expect:'''
    123  * All results look like:
    124 {{{
    125 $lastoct.242.1.192.in-addr.arpa domain name pointer <something reasonable>
    126 }}}
    127  and none look like:
    128 {{{
    129 Host $lastoct.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    130 }}}
     87'''Verify:'''
     88 * The shutdown procedure includes the complete set of steps to
     89 * The shutdown procedure results in a sliver being deactivated on a rack
     90 * Experimental traffic from the sliver is no longer being sent
    13191
    132 === Results of testing step 2C: 2012-12-20 ===
    133 
    134 Many addresses aren't defined:
    135 
    136 {{{
    137 [13:46:15] jbs@anubis:/home/jbs
    138 +$ for lastoct in {129..255} ; do host 192.1.242.$lastoct ; done
    139 Host 129.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    140 130.242.1.192.in-addr.arpa is an alias for 130.129/25.242.1.192.in-addr.arpa.
    141 130.129/25.242.1.192.in-addr.arpa domain name pointer control.instageni.gpolab.bbn.com.
    142 131.242.1.192.in-addr.arpa is an alias for 131.129/25.242.1.192.in-addr.arpa.
    143 131.129/25.242.1.192.in-addr.arpa domain name pointer control-ilo.instageni.gpolab.bbn.com.
    144 132.242.1.192.in-addr.arpa is an alias for 132.129/25.242.1.192.in-addr.arpa.
    145 132.129/25.242.1.192.in-addr.arpa domain name pointer boss.instageni.gpolab.bbn.com.
    146 133.242.1.192.in-addr.arpa is an alias for 133.129/25.242.1.192.in-addr.arpa.
    147 133.129/25.242.1.192.in-addr.arpa domain name pointer ops.instageni.gpolab.bbn.com.
    148 134.242.1.192.in-addr.arpa is an alias for 134.129/25.242.1.192.in-addr.arpa.
    149 134.129/25.242.1.192.in-addr.arpa domain name pointer foam.instageni.gpolab.bbn.com.
    150 135.242.1.192.in-addr.arpa is an alias for 135.129/25.242.1.192.in-addr.arpa.
    151 135.129/25.242.1.192.in-addr.arpa domain name pointer flowvisor.instageni.gpolab.bbn.com.
    152 Host 136.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    153 Host 137.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    154 Host 138.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    155 Host 139.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    156 140.242.1.192.in-addr.arpa is an alias for 140.129/25.242.1.192.in-addr.arpa.
    157 140.129/25.242.1.192.in-addr.arpa domain name pointer pc1.instageni.gpolab.bbn.com.
    158 141.242.1.192.in-addr.arpa is an alias for 141.129/25.242.1.192.in-addr.arpa.
    159 141.129/25.242.1.192.in-addr.arpa domain name pointer pc2.instageni.gpolab.bbn.com.
    160 142.242.1.192.in-addr.arpa is an alias for 142.129/25.242.1.192.in-addr.arpa.
    161 142.129/25.242.1.192.in-addr.arpa domain name pointer pc3.instageni.gpolab.bbn.com.
    162 143.242.1.192.in-addr.arpa is an alias for 143.129/25.242.1.192.in-addr.arpa.
    163 143.129/25.242.1.192.in-addr.arpa domain name pointer pc4.instageni.gpolab.bbn.com.
    164 144.242.1.192.in-addr.arpa is an alias for 144.129/25.242.1.192.in-addr.arpa.
    165 144.129/25.242.1.192.in-addr.arpa domain name pointer pc5.instageni.gpolab.bbn.com.
    166 Host 145.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    167 Host 146.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    168 Host 147.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    169 Host 148.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    170 Host 149.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    171 Host 150.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    172 Host 151.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    173 Host 152.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    174 Host 153.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    175 Host 154.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    176 Host 155.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    177 Host 156.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    178 Host 157.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    179 Host 158.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    180 Host 159.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    181 Host 160.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    182 Host 161.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    183 Host 162.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    184 Host 163.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    185 Host 164.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    186 Host 165.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    187 Host 166.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    188 Host 167.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    189 Host 168.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    190 Host 169.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    191 Host 170.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    192 Host 171.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    193 Host 172.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    194 Host 173.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    195 Host 174.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    196 Host 175.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    197 Host 176.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    198 Host 177.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    199 Host 178.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    200 Host 179.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    201 Host 180.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    202 Host 181.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    203 Host 182.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    204 Host 183.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    205 Host 184.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    206 Host 185.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    207 Host 186.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    208 Host 187.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    209 Host 188.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    210 Host 189.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    211 Host 190.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    212 Host 191.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    213 Host 192.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    214 Host 193.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    215 Host 194.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    216 Host 195.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    217 Host 196.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    218 Host 197.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    219 Host 198.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    220 Host 199.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    221 Host 200.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    222 Host 201.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    223 Host 202.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    224 Host 203.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    225 Host 204.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    226 Host 205.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    227 Host 206.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    228 Host 207.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    229 Host 208.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    230 Host 209.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    231 Host 210.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    232 Host 211.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    233 Host 212.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    234 Host 213.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    235 Host 214.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    236 Host 215.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    237 Host 216.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    238 Host 217.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    239 Host 218.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    240 Host 219.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    241 Host 220.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    242 Host 221.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    243 Host 222.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    244 Host 223.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    245 Host 224.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    246 Host 225.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    247 Host 226.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    248 Host 227.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    249 Host 228.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    250 Host 229.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    251 Host 230.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    252 Host 231.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    253 Host 232.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    254 Host 233.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    255 Host 234.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    256 Host 235.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    257 Host 236.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    258 Host 237.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    259 Host 238.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    260 Host 239.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    261 Host 240.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    262 Host 241.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    263 Host 242.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    264 Host 243.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    265 Host 244.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    266 Host 245.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    267 Host 246.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    268 Host 247.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    269 Host 248.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    270 Host 249.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    271 Host 250.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    272 Host 251.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    273 Host 252.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    274 Host 253.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    275 Host 254.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    276 Host 255.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    277 }}}
    278 
    279 We think that's normal: The in-use ones are in DNS, the not-in-use ones aren't.
    280 
    281 I tried creating a VM with a public IP address, using this rspec:
    282 
    283 {{{
    284 <?xml version="1.0" encoding="UTF-8"?>
    285 <rspec xmlns="http://www.geni.net/resources/rspec/3"
    286        xmlns:xs="http://www.w3.org/2001/XMLSchema-instance"
    287        xmlns:emulab="http://www.protogeni.net/resources/rspec/ext/emulab/1"
    288        xs:schemaLocation="http://www.geni.net/resources/rspec/3
    289            http://www.geni.net/resources/rspec/3/request.xsd"
    290        type="request">
    291   <node client_id="carlin" exclusive="false">
    292     <sliver_type name="emulab-openvz" />
    293     <emulab:routable_control_ip />
    294   </node>
    295 </rspec>
    296 }}}
    297 
    298 According to my manifest rspec, I got
    299 
    300 {{{
    301 <emulab:vnode name="pcvm2-3"/>    <host name="carlin.jbs.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com"/>    <services>      <login authentication="ssh-keys" hostname="pcvm2-3.instageni.gpolab.bbn.com" port="22" username="jbs"/>    </services>  </node>
    302 }}}
    303 
    304 That hostname and IP address now resolve:
    305 
    306 {{{
    307 [15:03:32] jbs@anubis:/home/jbs/rspecs/request
    308 +$ host pcvm2-3.instageni.gpolab.bbn.com
    309 pcvm2-3.instageni.gpolab.bbn.com has address 192.1.242.150
    310 
    311 [15:03:34] jbs@anubis:/home/jbs/rspecs/request
    312 +$ host 192.1.242.150
    313 150.242.1.192.in-addr.arpa is an alias for 150.129/25.242.1.192.in-addr.arpa.
    314 150.129/25.242.1.192.in-addr.arpa domain name pointer pcvm2-3.instageni.gpolab.bbn.com.
    315 }}}
    316 
    317 After I delete my sliver:
    318 
    319 {{{
    320 [15:03:58] jbs@anubis:/home/jbs/rspecs/request
    321 +$ omni -a $am deletesliver $slicename
    322 [* snip *]
    323   Result Summary: Deleted sliver urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs on unspecified_AM_URN at https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am
    324 INFO:omni: ============================================================
    325 
    326 [15:04:43] jbs@anubis:/home/jbs/rspecs/request
    327 +$ host pcvm2-3.instageni.gpolab.bbn.com
    328 Host pcvm2-3.instageni.gpolab.bbn.com not found: 3(NXDOMAIN)
    329 
    330 [15:05:57] jbs@anubis:/home/jbs/rspecs/request
    331 +$ host 192.1.242.150
    332 150.242.1.192.in-addr.arpa is an alias for 150.129/25.242.1.192.in-addr.arpa.
    333 150.129/25.242.1.192.in-addr.arpa domain name pointer pcvm2-3.instageni.gpolab.bbn.com.
    334 }}}
    335 
    336 That second one still works because it's cached on my local nameserver; if I ask the source, it's gone:
    337 
    338 {{{
    339 [15:32:13] jbs@ops.instageni.gpolab.bbn.com:/users/jbs
    340 +$ host 192.1.242.150
    341 Host 150.242.1.192.in-addr.arpa. not found: 3(NXDOMAIN)
    342 }}}
    343 
    344 So, I think this is fine: Records exist when they're in use, and not when they're not, and that's fine.
    345 
    346 = Step 3: GPO requests and receives administrator accounts =
    347 
    348 == Step 3A: GPO requests access to boss and ops nodes ==
     92=== Step 5C (verify): Documented procedure includes a step to follow up with GMOC ===
    34993
    35094'''Using:'''
    351  * Request accounts for GPO ops staffers on boss.instageni.gpolab.bbn.com and ops.instageni.gpolab.bbn.com
    352  * Chaos tries to SSH to chaos@boss.instageni.gpolab.bbn.com
    353  * Josh tries to SSH to jbs@boss.instageni.gpolab.bbn.com
    354  * Tim tries to SSH to tupty@boss.instageni.gpolab.bbn.com
    355  * Chaos tries to SSH to chaos@ops.instageni.gpolab.bbn.com
    356  * Josh tries to SSH to jbs@ops.instageni.gpolab.bbn.com
    357  * Tim tries to SSH to tupty@ops.instageni.gpolab.bbn.com
    358  * Chaos tries to run a minimal command as sudo on boss:
    359 {{{
    360 sudo whoami
    361 }}}
    362  * Chaos tries to run a minimal command as sudo on ops:
    363 {{{
    364 sudo whoami
    365 }}}
     95 * Documented InstaGENI sliver shut down procedure
    36696
    36797'''Verify:'''
    368  * Logins succeed for Chaos, Josh, and Tim on both nodes
    369  * The commands work:
    370 {{{
    371 $ sudo whoami
    372 root
    373 }}}
     98 * There is a step for the site administrator to follow up with the GMOC that a sliver has been shut down
    37499
    375 === Results of testing step 3A: 2012-12-20 ===
     100== Step 6 (verify): Documented procedure includes a clean-up step ==
    376101
    377 I followed the procedure at https://users.emulab.net/trac/protogeni/wiki/RackAdminAccounts#AdminAccountsinEmulab to join the emulab-ops project, and once the Utah folks approved that and made an admin, I was able to log in to boss and ops, and use sudo:
    378 
    379 {{{
    380 [15:50:40] jbs@anubis:/home/jbs
    381 +$ ssh ops.instageni.gpolab.bbn.com sudo whoami
    382 root
    383 
    384 [15:50:50] jbs@anubis:/home/jbs
    385 +$ ssh boss.instageni.gpolab.bbn.com sudo whoami
    386 root
    387 }}}
    388 
    389 I asked Chaos and Tim to follow the procedure at that URL as well, and they did, and I approved their accounts by following the procedure at https://users.emulab.net/trac/protogeni/wiki/RackAdminAccounts#AddingmoreadminaccountstoEmulab, and they confirmed that they could log in to boss and ops.
    390 
    391 === Results of testing step 3A: 2012-05-15 ===
    392 
    393 ''Note: This test was run on the Utah rack, where only Chaos has an account.  So Tim and Josh will not be testing, and the hosts to test are `boss.utah.geniracks.net` and `ops.utah.geniracks.net`.''
    394 
    395  * Chaos successfully used public-key login and sudo from a BBN subnet (128.89.68.0/23) to boss:
    396 {{{
    397 capybara,[~],11:39(0)$ ssh chaos@boss.utah.geniracks.net
    398 Last login: Tue May 15 07:29:07 2012 from capybara.bbn.co
    399 Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
    400         The Regents of the University of California.  All rights reserved.
    401 
    402 FreeBSD 8.3-RC1 (XEN) #0: Tue Mar 13 16:27:12 MDT 2012
    403 
    404 Welcome to FreeBSD!
    405 
    406 Need to see the calendar for this month? Simply type "cal".  To see the
    407 whole year, type "cal -y".
    408                 -- Dru <genesis@istar.ca>
    409 > bash
    410 boss,[~],09:39(0)$ sudo whoami
    411 root
    412 }}}
    413  * Chaos successfully used public-key login and sudo from a BBN subnet (128.89.68.0/23) to ops:
    414 {{{
    415 capybara,[~],11:40(0)$ ssh chaos@ops.utah.geniracks.net
    416 Last login: Sat May 12 15:41:57 2012 from capybara.bbn.co
    417 Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
    418         The Regents of the University of California.  All rights reserved.
    419 
    420 FreeBSD 8.3-RC1 (XEN) #0: Tue Mar 13 16:27:12 MDT 2012
    421 
    422 Welcome to FreeBSD!
    423 
    424 ops,[~],09:40(0)$ sudo whoami
    425 root
    426 }}}
    427 
    428 == Step 3B: GPO requests access to FOAM VM ==
    429 
    430  * Request accounts for GPO ops staffers on foam.instageni.gpolab.bbn.com
    431  * Chaos tries to SSH to chaos@foam.instageni.gpolab.bbn.com
    432  * Josh tries to SSH to jbs@foam.instageni.gpolab.bbn.com
    433  * Tim tries to SSH to tupty@foam.instageni.gpolab.bbn.com
    434  * Chaos tries to run a minimal command as sudo on foam:
    435 {{{
    436 sudo whoami
    437 }}}
     102'''Using:'''
     103 * Documented InstaGENI sliver shut down procedure
    438104
    439105'''Verify:'''
    440  * Logins succeed for Chaos, Josh, and Tim on the FOAM VM
    441  * The command works:
    442 {{{
    443 $ sudo whoami
    444 root
    445 }}}
    446 
    447 === Results of testing step 3B: 2012-12-20 ===
    448 
    449 I was named as the site admin in the site survey, and was given an account on the FOAM VM. I was able to log in and use sudo:
    450 
    451 {{{
    452 [15:57:46] jbs@anubis:/home/jbs
    453 +$ ssh foam.instageni.gpolab.bbn.com sudo whoami
    454 root
    455 }}}
    456 
    457 I then created accounts for Chaos and Tim, following the procedure at https://users.emulab.net/trac/protogeni/wiki/RackAdminAccounts#AdminAccountsonInstaGeniRacks. I got their public keys from their Emulab accounts, and put them into chaos.keys and tupty.keys in my homedir, and then:
    458 
    459 {{{
    460 sudo /usr/local/bin/mkadmin.pl chaos chaos.keys
    461 sudo /usr/local/bin/mkadmin.pl tupty tupty.keys
    462 }}}
    463 
    464 They then confirmed that they could log in, and run 'sudo whoami'.
    465 
    466 === Results of testing step 3B: 2012-07-04 ===
    467 
    468 ''Note: This test was run on the Utah rack."
    469 
    470  * Chaos can SSH to foam.utah.geniracks.net:
    471 {{{
    472 $ ssh foam.utah.geniracks.net
    473 Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-24-generic x86_64)
    474 
    475  * Documentation:  https://help.ubuntu.com/
    476 Last login: Tue Jul  3 12:57:20 2012 from capybara.bbn.com
    477 foam,[~],09:33(0)$
    478 }}}
    479  * Chaos can sudo on foam:
    480 {{{
    481 foam,[~],09:33(0)$ sudo whoami
    482 root
    483 }}}
     106 * If the shutdown procedure affects other experimenters, include a clean up step