| Version 5 (modified by , 12 years ago) (diff) |
|---|
- Detailed test plan for EG-ADM-6: Control Network Disconnection Test
Detailed test plan for EG-ADM-6: Control Network Disconnection Test
This page is GPO's working page for performing EG-ADM-6. It is public for informational purposes, but it is not an official status report. See GENIRacksHome/ExogeniRacks/AcceptanceTestStatus for the current status of ExoGENI acceptance tests.
Last substantive edit of this page: 2012-08-29
Page format
- The status chart summarizes the state of this test
- The high-level description from test plan contains text copied exactly from the public test plan and acceptance criteria pages.
- The steps contain things i will actually do/verify:
- Steps may be composed of related substeps where i find this useful for clarity
- Each step is either a preparatory step (identified by "(prep)") or a verification step (the default):
- Preparatory steps are just things we have to do. They're not tests of the rack, but are prerequisites for subsequent verification steps
- Verification steps are steps in which we will actually look at rack output and make sure it is as expected. They contain a Using: block, which lists the steps to run the verification, and an Expect: block which lists what outcome is expected for the test to pass.
Status of test
| Step | State | Date completed | Open Tickets | Closed Tickets/Comments |
| 1A | Color(green,Pass)? | 2012-08-29 | ||
| 1B | Color(green,Pass)? | 2012-08-29 | ||
| 1C | Color(green,Pass)? | 2012-08-29 |
High-level description from test plan
In this test, we disconnect parts of the rack control network or its dependencies to test partial rack functionality in an outage situation.
Procedure
- Simulate an outage of geni.renci.org by inserting a firewall rule on the GPO router blocking the rack from reaching it. Verify that an administrator can still access the rack, that rack monitoring to GMOC continues through the outage, and that some experimenter operations still succeed.
- Simulate an outage of each of the rack head node and management switch by disabling their respective interfaces on the GPO's control network switch. Verify that GPO, ExoGENI, and GMOC monitoring all see the outage.
Criteria to verify as part of this test
- V.09. When the rack control network is partially down or the rack vendor's home site is inaccessible from the rack, it is still possible to access the primary control network device and server for recovery. All devices/networks which must be operational in order for the control network switch and primary server to be reachable, are documented. (C.3.b)
- VII.14. A site administrator can locate information about the network reachability of all rack infrastructure which should live on the control network, and can get alerts when any rack infrastructure control IP becomes unavailable from the rack server host, or when the rack server host cannot reach the commodity internet. (D.6.c)
Step 1: prevent 152.54.0.0/16 from sending traffic to the ExoGENI rack
Step 1A: baseline before simulating control network outage
Overview of Step 1A
Run all checks before performing the disconnection test, so that if something is already down, we won't be confused:
- Attempt to SSH to bbn-hn.exogeni.gpolab.bbn.com:
ssh bbn-hn.exogeni.gpolab.bbn.com
- If SSH is successful, attempt to sudo on bbn-hn:
sudo -v
- Browse to https://bbn-hn.exogeni.net/rack_bbn/ and attempt to login to nagios
- If successful, enumerate any errors currently outstanding in rack nagios
- Browse to http://monitor.gpolab.bbn.com/nagios/cgi-bin/status.cgi?hostgroup=all&style=detail&sorttype=2&sortoption=3 and ensure that GPO nagios is available
- If successful, enumerate any exogeni-relevant errors currently outstanding in GPO nagios
- Run omni getversion and listresources against the BBN rack ORCA AM:
omni -a https://bbn-hn.exogeni.gpolab.bbn.com:11443/orca/xmlrpc getversion omni -a https://bbn-hn.exogeni.gpolab.bbn.com:11443/orca/xmlrpc listresources
- Run omni getversion and listresources against the BBN rack FOAM AM:
omni -a https://bbn-hn.exogeni.gpolab.bbn.com:3626/foam/gapi/1 getversion omni -a https://bbn-hn.exogeni.gpolab.bbn.com:3626/foam/gapi/1 listresources
- Verify that http://monitor.gpolab.bbn.com/connectivity/campus.html currently shows a successful connection from argos to exogeni-vlan1750.bbn.dataplane.geni.net
Results of testing step 1A on 2012-08-29
- Login to bbn-hn was successful
- I noticed that
sudo -vdoesn't ask for my password... I guess i was usingsudo whoamifor the previous tests, but anywaysudo -ldid ask for a password. I want to ask about that configuration, but that's not part of this test. - Login to nagios was successful. Outstanding errors:
- critical:
- Updates needed on bbn-w1, bbn-w2, bbn-w3
- 8052.bbn.xo reports that interfaces: 31-36,45-46 are down
- warning:
- Updates needed on bbn-hn
- 8052.bbn.xo reports that interfaces: 10-15,41-44 are the "wrong speed" (running at 1Gbps, expected 100Mbps)
- critical:
- GPO nagios is available. Current errors are:
TANGO-openflow4.stanford.edu/gmoc_monitoring ganel.gpolab.bbn.com/cpu_idle gardil.gpolab.bbn.com/cpu_idle ilian.gpolab.bbn.com/ping_192.1.248.250
none of which pertain to the ExoGENI rack. - ORCA getversion output:
jericho,[~],16:23(0)$ omni -a https://bbn-hn.exogeni.gpolab.bbn.com:11443/orca/xmlrpc getversion INFO:omni:Loading config file /home/chaos/omni/omni_pgeni INFO:omni:Using control framework pg INFO:omni:AM URN: unspecified_AM_URN (url: https://bbn-hn.exogeni.gpolab.bbn.com:11443/orca/xmlrpc) has version: INFO:omni:{ 'code': { 'geni_code': 0}, 'geni_api': 2, 'value': { 'geni_ad_rspec_versions': [ { 'extensions': [ 'http://hpn.east.isi.edu/rspec/ext/stitch/0.1/stitch-schema.xsd', 'http://www.protogeni.net/resources/rspec/ext/emulab/1/ptop_extension.xsd'], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd', 'type': 'GENI', 'version': '3'}], 'geni_api': 2, 'geni_api_versions': { '1': 'https://geni.renci.org:11443/orca/xmlrpc/geniV1', '2': 'https://geni.renci.org:11443/orca/xmlrpc/geni'}, 'geni_request_rspec_versions': [ { 'extensions': [ 'http://www.protogeni.net/resources/rspec/ext/shared-vlan/1', 'http://www.geni.net/resources/rspec/ext/postBootScript/1'], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/request.xsd', 'type': 'GENI', 'version': '3'}], 'orca_version': 'ORCA Camano: v.3.1-extended.build-4724'}} INFO:omni: ------------------------------------------------------------ INFO:omni: Completed getversion: Options as run: aggregate: https://bbn-hn.exogeni.gpolab.bbn.com:11443/orca/xmlrpc configfile: /home/chaos/omni/omni_pgeni framework: pg native: True Args: getversion Result Summary: Got version for 1 out of 1 aggregates INFO:omni: ============================================================ - ORCA listresources succeeded, and i saved the output to:
rspec-bbn-hn-exogeni-gpolab-bbn-com-11443-orca.20120829.080807.xml
- FOAM getversion:
jericho,[~],08:08(0)$ omni -a https://bbn-hn.exogeni.gpolab.bbn.com:3626/foam/gapi/1 getversion INFO:omni:Loading config file /home/chaos/omni/omni_pgeni INFO:omni:Using control framework pg INFO:omni:AM URN: unspecified_AM_URN (url: https://bbn-hn.exogeni.gpolab.bbn.com:3626/foam/gapi/1) has version: INFO:omni:{ 'ad_rspec_versions': [ { 'extensions': [ 'http://www.geni.net/resources/rspec/ext/openflow/3'], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd', 'type': 'GENI', 'version': '3'}], 'foam_version': '0.8.2', 'geni_api': 1, 'request_rspec_versions': [ { 'extensions': [ 'http://www.geni.net/resources/rspec/ext/openflow/3', 'http://www.geni.net/resources/rspec/ext/openflow/4', 'http://www.geni.net/resources/rspec/ext/flowvisor/1'], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/request.xsd', 'type': 'GENI', 'version': '3'}], 'site_info': { }} INFO:omni: ------------------------------------------------------------ INFO:omni: Completed getversion: Options as run: aggregate: https://bbn-hn.exogeni.gpolab.bbn.com:3626/foam/gapi/1 configfile: /home/chaos/omni/omni_pgeni framework: pg native: True Args: getversion Result Summary: Got version for 1 out of 1 aggregates INFO:omni: ============================================================ - FOAM listresources succeeded, and i saved the output to:
rspec-bbn-hn-exogeni-gpolab-bbn-com-3626-foam-gapi.20120829.080927.xml
- The GPO lab to ExoGENI test dataplane connection is currently up
Step 1B: simulate a network problem
Overview of Step 1B
Using:
- On maple (lab subnet router), do:
conf t ip access-list standard gst-4083-eg-adm-6 deny 152.54.0.0 0.0.255.255 permit any exit int gi 0/1.829 ip access-group gst-4083-eg-adm-6 out exit exit - Wait 5 minutes
- Attempt to SSH to bbn-hn.exogeni.gpolab.bbn.com:
ssh bbn-hn.exogeni.gpolab.bbn.com
- If SSH is successful, attempt to sudo on bbn-hn:
sudo -v
- Browse to https://bbn-hn.exogeni.net/rack_bbn/ and attempt to login to nagios
- If successful, enumerate any errors currently outstanding in rack nagios
- Browse to http://monitor.gpolab.bbn.com/nagios/cgi-bin/status.cgi?hostgroup=all&style=detail&sorttype=2&sortoption=3 and ensure that GPO nagios is available
- If successful, enumerate any exogeni-relevant errors currently outstanding in GPO nagios
- Run omni getversion and listresources against the BBN rack ORCA AM:
omni -a https://bbn-hn.exogeni.gpolab.bbn.com:11443/orca/xmlrpc getversion omni -a https://bbn-hn.exogeni.gpolab.bbn.com:11443/orca/xmlrpc listresources
- Run omni getversion and listresources against the BBN rack FOAM AM:
omni -a https://bbn-hn.exogeni.gpolab.bbn.com:3626/foam/gapi/1 getversion omni -a https://bbn-hn.exogeni.gpolab.bbn.com:3626/foam/gapi/1 listresources
- Look at http://monitor.gpolab.bbn.com/connectivity/campus.html to see the status of the connection from argos to exogeni-vlan1750.bbn.dataplane.geni.net
Verify:
- Site admins should be able to login to bbn-hn and perform sudo operations
- Rack nagios should be available (though it may show some errors based on the disconnection, obviously)
- Rack ORCA and FOAM aggregates should respond to getversion and listresources
- Existing experiments should continue to run, and should be able to respond to dataplane traffic
Results of testing step 1B on 2012-08-29
- SSH is still successful
- I can still run
sudo whoami - I can still login to rack nagios, and the set of problems is exactly as before
- GPO nagios is still available, and the set of down services is exactly as before
- ORCA omni getversion succeeds:
jericho,[~],08:09(0)$ omni -a https://bbn-hn.exogeni.gpolab.bbn.com:11443/orca/xmlrpc getversion INFO:omni:Loading config file /home/chaos/omni/omni_pgeni INFO:omni:Using control framework pg INFO:omni:AM URN: unspecified_AM_URN (url: https://bbn-hn.exogeni.gpolab.bbn.com:11443/orca/xmlrpc) has version: INFO:omni:{ 'code': { 'geni_code': 0}, 'geni_api': 2, 'value': { 'geni_ad_rspec_versions': [ { 'extensions': [ 'http://hpn.east.isi.edu/rspec/ext/stitch/0.1/stitch-schema.xsd', 'http://www.protogeni.net/resources/rspec/ext/emulab/1/ptop_extension.xsd'], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd', 'type': 'GENI', 'version': '3'}], 'geni_api': 2, 'geni_api_versions': { '1': 'https://geni.renci.org:11443/orca/xmlrpc/geniV1', '2': 'https://geni.renci.org:11443/orca/xmlrpc/geni'}, 'geni_request_rspec_versions': [ { 'extensions': [ 'http://www.protogeni.net/resources/rspec/ext/shared-vlan/1', 'http://www.geni.net/resources/rspec/ext/postBootScript/1'], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/request.xsd', 'type': 'GENI', 'version': '3'}], 'orca_version': 'ORCA Camano: v.3.1-extended.build-4724'}} INFO:omni: ------------------------------------------------------------ INFO:omni: Completed getversion: Options as run: aggregate: https://bbn-hn.exogeni.gpolab.bbn.com:11443/orca/xmlrpc configfile: /home/chaos/omni/omni_pgeni framework: pg native: True Args: getversion Result Summary: Got version for 1 out of 1 aggregates INFO:omni: ============================================================ - ORCA omni listresources succeeds, and i've saved that output as:
rspec-bbn-hn-exogeni-gpolab-bbn-com-11443-orca.20120829.082044.xml
- FOAM omni getversion succeeds:
jericho,[~],08:20(0)$ omni -a https://bbn-hn.exogeni.gpolab.bbn.com:3626/foam/gapi/1 getversion INFO:omni:Loading config file /home/chaos/omni/omni_pgeni INFO:omni:Using control framework pg INFO:omni:AM URN: unspecified_AM_URN (url: https://bbn-hn.exogeni.gpolab.bbn.com:3626/foam/gapi/1) has version: INFO:omni:{ 'ad_rspec_versions': [ { 'extensions': [ 'http://www.geni.net/resources/rspec/ext/openflow/3'], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd', 'type': 'GENI', 'version': '3'}], 'foam_version': '0.8.2', 'geni_api': 1, 'request_rspec_versions': [ { 'extensions': [ 'http://www.geni.net/resources/rspec/ext/openflow/3', 'http://www.geni.net/resources/rspec/ext/openflow/4', 'http://www.geni.net/resources/rspec/ext/flowvisor/1'], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/request.xsd', 'type': 'GENI', 'version': '3'}], 'site_info': { }} INFO:omni: ------------------------------------------------------------ INFO:omni: Completed getversion: Options as run: aggregate: https://bbn-hn.exogeni.gpolab.bbn.com:3626/foam/gapi/1 configfile: /home/chaos/omni/omni_pgeni framework: pg native: True Args: getversion Result Summary: Got version for 1 out of 1 aggregates INFO:omni: ============================================================ - FOAM omni listresources succeeds, and i've saved that output as:
rspec-bbn-hn-exogeni-gpolab-bbn-com-3626-foam-gapi.20120829.082137.xml
- The dataplane connection between argos and Tim's test VM is still working
- On bbn-hn, i did verify that i couldn't ping geni.renci.org or control.exogeni.net:
bbn-hn,[~],12:16(0)$ ping geni.renci.org PING geni.renci.org (152.54.3.34) 56(84) bytes of data. ^C --- geni.renci.org ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 2170ms bbn-hn,[~],12:19(1)$ ping control.exogeni.net PING control.exogeni.net (152.54.1.65) 56(84) bytes of data. ^C --- control.exogeni.net ping statistics --- 2 packets transmitted, 0 received, 100% packet loss, time 1198ms
Step 1C: undo the changes
Overview of Step 1C
Using:
- On maple (lab subnet router), do:
conf t int gi 0/1.829 no ip access-group gst-4083-eg-adm-6 out exit no ip access-list standard gst-4083-eg-adm-6 exit - Wait 5 minutes
- Attempt to SSH to bbn-hn.exogeni.gpolab.bbn.com:
ssh bbn-hn.exogeni.gpolab.bbn.com
- If SSH is successful, attempt to sudo on bbn-hn:
sudo -v
- Browse to https://bbn-hn.exogeni.net/rack_bbn/ and attempt to login to nagios
- Run omni getversion and listresources against the BBN rack ORCA AM:
omni -a https://bbn-hn.exogeni.gpolab.bbn.com:11443/orca/xmlrpc getversion omni -a https://bbn-hn.exogeni.gpolab.bbn.com:11443/orca/xmlrpc listresources
- Run omni getversion and listresources against the BBN rack FOAM AM:
omni -a https://bbn-hn.exogeni.gpolab.bbn.com:3626/foam/gapi/1 getversion omni -a https://bbn-hn.exogeni.gpolab.bbn.com:3626/foam/gapi/1 listresources
- Look at http://monitor.gpolab.bbn.com/connectivity/campus.html to see the status of the connection from argos to exogeni-vlan1750.bbn.dataplane.geni.net
Verify:
- Site admins should be able to login to bbn-hn and perform sudo operations
- Rack nagios should be available (though it may show some errors based on the disconnection, obviously)
- Rack ORCA and FOAM aggregates should respond to getversion and listresources
- Existing experiments should continue to run, and should be able to respond to dataplane traffic
Results of testing step 1C on 2012-08-29
- I can login to bbn-hn and sudo (no surprises there)
- I verified that i can now ping geni.renci.org and control.exogeni.net:
bbn-hn,[~],12:19(1)$ ping geni.renci.org PING geni.renci.org (152.54.3.34) 56(84) bytes of data. 64 bytes from geni.renci.org (152.54.3.34): icmp_seq=1 ttl=49 time=23.7 ms 64 bytes from geni.renci.org (152.54.3.34): icmp_seq=2 ttl=49 time=22.7 ms ^C --- geni.renci.org ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1369ms rtt min/avg/max/mdev = 22.794/23.277/23.760/0.483 ms bbn-hn,[~],12:24(0)$ ping control.exogeni.net PING control.exogeni.net (152.54.1.65) 56(84) bytes of data. 64 bytes from control.exogeni.net (152.54.1.65): icmp_seq=1 ttl=49 time=22.8 ms 64 bytes from control.exogeni.net (152.54.1.65): icmp_seq=2 ttl=49 time=23.1 ms ^C --- control.exogeni.net ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1390ms rtt min/avg/max/mdev = 22.896/23.005/23.114/0.109 ms
- Everything else seems to be unchanged.
I tried one additional thing:
- I ran omni listresources against geni.renci.org's ORCA
- I redid the changes on maple to block 152.54.0.0/16 access
- I ran omni listresources against geni.renci.org's ORCA again
- Those two listresources outputs also did not differ
