[[PageOutline]] = Detailed test plan for IG-ADM-6: Control Network Disconnection Test = ''This page is GPO's working page for performing IG-ADM-6. It is public for informational purposes, but it is not an official status report. See [wiki:GENIRacksHome/InstageniRacks/AcceptanceTestStatus] for the current status of InstaGENI acceptance tests.'' ''Last substantive edit of this page: 2013-03-08'' == Page format == * The status chart summarizes the state of this test * The high-level description from test plan contains text copied exactly from the public test plan and acceptance criteria pages. * The steps contain things i will actually do/verify: * Steps may be composed of related substeps where i find this useful for clarity * Each step is either a preparatory step (identified by "(prep)") or a verification step (the default): * Preparatory steps are just things we have to do. They're not tests of the rack, but are prerequisites for subsequent verification steps * Verification steps are steps in which we will actually look at rack output and make sure it is as expected. They contain a '''Using:''' block, which lists the steps to run the verification, and an '''Expect:''' block which lists what outcome is expected for the test to pass. == Status of test == || '''Step''' || '''State''' || '''Date completed''' || '''Open Tickets''' || '''Closed Tickets/Comments''' || || 1A || || || || || || 1B || || || || || || 1C || || || || || == High-level description from test plan == In this test, we disconnect parts of the rack control network or its dependencies to test partial rack functionality in an outage situation. === Procedure === * Simulate an outage of emulab.net by inserting a firewall rule on the GPO router blocking the rack from reaching it. Verify that an administrator can still access the rack, that rack monitoring to GMOC continues through the outage, and that some experimenter operations still succeed. === Criteria to verify as part of this test === * V.09. When the rack control network is partially down or the rack vendor's home site is inaccessible from the rack, it is still possible to access the primary control network device and server for recovery. All devices/networks which must be operational in order for the control network switch and primary server to be reachable, are documented. (C.3.b) * VII.14. A site administrator can locate information about the network reachability of all rack infrastructure which should live on the control network, and can get alerts when any rack infrastructure control IP becomes unavailable from the rack server host, or when the rack server host cannot reach the commodity internet. (D.6.c) == Step 1: prevent 155.98.33.0/24 from sending traffic to the InstaGENI rack == === Step 1A: baseline before simulating control network outage === ==== Overview of Step 1A ==== Run all checks before performing the disconnection test, so that if something is already down, we won't be confused: * Attempt to SSH to boss.instageni.gpolab.bbn.com: {{{ ssh boss.instageni.gpolab.bbn.com }}} * If SSH is successful, attempt to sudo on boss: {{{ sudo whoami }}} * If SSH is successful, ping ops.emulab.net to verify connectivity: {{{ ping ops.emulab.net }}} * Attempt to SSH to gpolab.control-nodes.geniracks.net: {{{ ssh gpolab.control-nodes.geniracks.net }}} * If SSH is successful, attempt to sudo on control: {{{ sudo whoami }}} * Attempt to SSH to foam.instageni.gpolab.bbn.com: {{{ ssh foam.instageni.gpolab.bbn.com }}} * If SSH is successful, attempt to sudo on foam: {{{ sudo whoami }}} * Browse to [https://instageni.gpolab.bbn.com/] and attempt to login as chaos * If successful, enter red dot mode * Browse to [http://monitor.gpolab.bbn.com/nagios/cgi-bin/status.cgi?hostgroup=all&style=detail&sorttype=2&sortoption=3] and ensure that GPO nagios is available * If successful, enumerate any instageni-relevant errors currently outstanding in GPO nagios * Run omni getversion and listresources against the BBN rack ProtoGENI AM: {{{ omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am getversion omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am listresources -o }}} * Run omni getversion and listresources against the BBN rack FOAM AM: {{{ omni -a https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 getversion omni -a https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 listresources -o }}} * Verify that [http://monitor.gpolab.bbn.com/connectivity/campus.html] currently shows a successful connection from argos to instageni-vlan1750.bbn.dataplane.geni.net ==== Results of testing Step 1A on 2013-03-08 ==== * SSH and sudo on boss look fine: {{{ capybara,[~],11:07(0)$ ssh boss.instageni.gpolab.bbn.com Last login: Fri Mar 8 11:06:19 2013 from capybara.bbn.com FreeBSD 9.0-STABLE (XEN) #4: Mon Nov 26 10:02:53 MST 2012 Welcome to ProtoGeni! Remember to put /usr/testbed/bin and /usr/testbed/sbin in your path! Need to print a manpage? Use man name_of_manpage | col -bx | lpr -- Dru > sudo whoami root > }}} * If SSH is successful, ping ops.emulab.net to verify connectivity: fine: {{{ > ping ops.emulab.net PING ops.emulab.net (155.98.33.74): 56 data bytes 64 bytes from 155.98.33.74: icmp_seq=0 ttl=47 time=61.098 ms 64 bytes from 155.98.33.74: icmp_seq=1 ttl=47 time=66.921 ms 64 bytes from 155.98.33.74: icmp_seq=2 ttl=47 time=67.713 ms ^C --- ops.emulab.net ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 61.098/65.244/67.713/2.949 ms > }}} * Attempt to SSH to gpolab.control-nodes.geniracks.net: fine: {{{ capybara,[~],11:08(0)$ ssh gpolab.control-nodes.geniracks.net Welcome to Ubuntu precise (development branch) (GNU/Linux 3.2.0-23-generic x86_64) * Documentation: https://help.ubuntu.com/ System information as of Fri Mar 8 09:08:02 MST 2013 System load: 0.0 Users logged in: 0 Usage of /: 24.7% of 5.85GB IP address for xenbr0: 192.1.242.130 Memory usage: 42% IP address for xenbr1: 10.1.1.254 Swap usage: 0% IP address for xenbr2: 10.2.1.254 Processes: 143 IP address for xenbr3: 10.3.1.254 Graph this data and manage this system at https://landscape.canonical.com/ Last login: Thu Feb 28 07:26:30 2013 from capybara.bbn.com gpolab,[~],09:08(0)$ sudo whoami root gpolab,[~],09:08(0)$ }}} * Attempt to SSH to foam.instageni.gpolab.bbn.com: fine: {{{ capybara,[~],11:09(0)$ ssh foam.instageni.gpolab.bbn.com Welcome to Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-34-generic x86_64) * Documentation: https://help.ubuntu.com/ Last login: Wed Jan 23 10:01:30 2013 from capybara.bbn.com foam,[~],11:09(0)$ sudo whoami root foam,[~],11:09(0)$ }}} * Browse to [https://instageni.gpolab.bbn.com/] and attempt to login as chaos: fine, though i got a security certificate warning from firefox --- i guess i usually go to `https://boss.instageni.gpolab.bbn.com`. Anyway, login worked fine. * If successful, enter red dot mode: yep * Browse to [http://monitor.gpolab.bbn.com/nagios/cgi-bin/status.cgi?hostgroup=all&style=detail&sorttype=2&sortoption=3] and ensure that GPO nagios is available: fine * If successful, enumerate any instageni-relevant errors currently outstanding in GPO nagios: no service errors related to anything ending in `instageni.gpolab.bbn.com` * Run omni getversion and listresources against the BBN rack ProtoGENI AM: {{{ jericho,[~],11:13(0)$ omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am getversion INFO:omni:Loading config file /home/chaos/omni/omni_pgeni INFO:omni:Using control framework pg INFO:omni:AM URN: unspecified_AM_URN (url: https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am) has version: INFO:omni:{ 'code': { 'am_code': 0, 'am_type': 'protogeni', 'geni_code': 0, 'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+ff955d693a123012969aa7f3a55aa222', 'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=ff955d693a123012969aa7f3a55aa222'}, 'geni_api': 2, 'output': '', 'value': { 'code_tag': 'b8883d9be04ae81a2180349b95501e30078abb1f', 'geni_ad_rspec_versions': [ { 'extensions': [], 'namespace': 'http://www.protogeni.net/resources/rspec/0.1', 'schema': 'http://www.protogeni.net/resources/rspec/0.1/ad.xsd', 'type': 'ProtoGENI', 'version': '0.1'}, { 'extensions': [], 'namespace': 'http://www.protogeni.net/resources/rspec/0.2', 'schema': 'http://www.protogeni.net/resources/rspec/0.2/ad.xsd', 'type': 'ProtoGENI', 'version': '0.2'}, { 'extensions': [ 'http://www.protogeni.net/resources/rspec/ext/emulab/1'], 'namespace': 'http://www.protogeni.net/resources/rspec/2', 'schema': 'http://www.protogeni.net/resources/rspec/2/ad.xsd', 'type': 'ProtoGENI', 'version': '2'}, { 'extensions': [ 'http://www.protogeni.net/resources/rspec/ext/emulab/1'], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd', 'type': 'GENI', 'version': '3'}], 'geni_api': 2, 'geni_api_versions': { '1': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/1.0', '2': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0', '3': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/3.0'}, 'geni_request_rspec_versions': [ { 'extensions': [], 'namespace': 'http://www.protogeni.net/resources/rspec/0.1', 'schema': 'http://www.protogeni.net/resources/rspec/0.1/request.xsd', 'type': 'ProtoGENI', 'version': '0.1'}, { 'extensions': [], 'namespace': 'http://www.protogeni.net/resources/rspec/0.2', 'schema': 'http://www.protogeni.net/resources/rspec/0.2/request.xsd', 'type': 'ProtoGENI', 'version': '0.2'}, { 'extensions': [ 'http://www.protogeni.net/resources/rspec/ext/emulab/1'], 'namespace': 'http://www.protogeni.net/resources/rspec/2', 'schema': 'http://www.protogeni.net/resources/rspec/2/request.xsd', 'type': 'ProtoGENI', 'version': '2'}, { 'extensions': [ 'http://www.protogeni.net/resources/rspec/ext/emulab/1'], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/request.xsd', 'type': 'GENI', 'version': '3'}], 'hostname': 'boss.instageni.gpolab.bbn.com', 'hrn': 'bbninstageni.cm', 'interface': 'aggregate', 'url': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am', 'urn': 'urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm'}} INFO:omni: ------------------------------------------------------------ INFO:omni: Completed getversion: Options as run: aggregate: https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am configfile: /home/chaos/omni/omni_pgeni framework: pg native: True Args: getversion Result Summary: Got version for 1 out of 1 aggregates INFO:omni: ============================================================ jericho,[~],11:13(0)$ jericho,[~],11:16(0)$ omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am -V2 -t GENI 3 listresources -o INFO:omni:Loading config file /home/chaos/omni/omni_pgeni INFO:omni:Using control framework pg INFO:omni:Saving output to a file. INFO:omni:Listed resources on 1 out of 1 possible aggregates. INFO:omni:Writing to 'rspec-instageni-gpolab-bbn-com-12369-protogeni.xml' INFO:omni: ------------------------------------------------------------ INFO:omni: Completed listresources: Options as run: aggregate: https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am api_version: 2 configfile: /home/chaos/omni/omni_pgeni framework: pg native: True output: True rspectype: ('GENI', '3') Args: listresources Result Summary: Retrieved resources from 1 aggregates. Wrote rspecs from 1 aggregates to 1 files Saved listresources RSpec at 'unspecified_AM_URN' to file rspec-instageni-gpolab-bbn-com-12369-protogeni.xml; . INFO:omni: ============================================================ jericho,[~],11:18(0)$ mv rspec-instageni-gpolab-bbn-com-12369-protogeni.xml rspec-instageni-gpolab-bbn-com-12369-protogeni-IG-ADM-6-01.xml jericho,[~],11:18(0)$ }}} * Run omni getversion and listresources against the BBN rack FOAM AM: {{{ jericho,[~],11:19(0)$ omni -a https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 getversion INFO:omni:Loading config file /home/chaos/omni/omni_pgeni INFO:omni:Using control framework pg INFO:omni:AM URN: unspecified_AM_URN (url: https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1) has version: INFO:omni:{ 'ad_rspec_versions': [ { 'extensions': [ 'http://www.geni.net/resources/rspec/ext/openflow/3'], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd', 'type': 'GENI', 'version': '3'}], 'foam_version': '0.8.2', 'geni_api': 1, 'request_rspec_versions': [ { 'extensions': [ 'http://www.geni.net/resources/rspec/ext/openflow/3', 'http://www.geni.net/resources/rspec/ext/openflow/4', 'http://www.geni.net/resources/rspec/ext/flowvisor/1'], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/request.xsd', 'type': 'GENI', 'version': '3'}], 'site_info': { 'admin-email': 'instageni-foam-admin@gpolab.bbn.com'}} INFO:omni: ------------------------------------------------------------ INFO:omni: Completed getversion: Options as run: aggregate: https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 configfile: /home/chaos/omni/omni_pgeni framework: pg native: True Args: getversion Result Summary: Got version for 1 out of 1 aggregates INFO:omni: ============================================================ jericho,[~],11:19(0)$ omni -a https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 listresources -o INFO:omni:Loading config file /home/chaos/omni/omni_pgeni INFO:omni:Using control framework pg INFO:omni:Saving output to a file. INFO:omni:Listed resources on 1 out of 1 possible aggregates. INFO:omni:Writing to 'rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1.xml' INFO:omni: ------------------------------------------------------------ INFO:omni: Completed listresources: Options as run: aggregate: https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 configfile: /home/chaos/omni/omni_pgeni framework: pg native: True output: True Args: listresources Result Summary: Retrieved resources from 1 aggregates. Wrote rspecs from 1 aggregates to 1 files Saved listresources RSpec at 'unspecified_AM_URN' to file rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1.xml; . INFO:omni: ============================================================ jericho,[~],11:19(0)$ mv rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1.xml rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1-IG-ADM-6-01.xml jericho,[~],11:20(0)$ }}} * Verify that [http://monitor.gpolab.bbn.com/connectivity/campus.html] currently shows a successful connection from argos to instageni-vlan1750.bbn.dataplane.geni.net: yep === Step 1B: simulate a network problem === ==== Overview of Step 1B ==== '''Using:''' * On maple (lab subnet router), do: {{{ conf t ip access-list standard gst-4544-ig-adm-6 deny 155.98.33.0 0.0.0.255 permit any exit int gi 0/1.830 ip access-group gst-4544-ig-adm-6 out exit exit }}} * Wait 5 minutes * Attempt to SSH to boss.instageni.gpolab.bbn.com: {{{ ssh boss.instageni.gpolab.bbn.com }}} * If SSH is successful, attempt to sudo on boss: {{{ sudo whoami }}} * If SSH is successful, ping ops.emulab.net to verify connectivity: {{{ ping ops.emulab.net }}} * Attempt to SSH to gpolab.control-nodes.geniracks.net: {{{ ssh gpolab.control-nodes.geniracks.net }}} * If SSH is successful, attempt to sudo on control: {{{ sudo whoami }}} * Attempt to SSH to foam.instageni.gpolab.bbn.com: {{{ ssh foam.instageni.gpolab.bbn.com }}} * If SSH is successful, attempt to sudo on foam: {{{ sudo whoami }}} * Browse to [https://instageni.gpolab.bbn.com/] and attempt to login as chaos * If successful, enter red dot mode * Browse to [http://monitor.gpolab.bbn.com/nagios/cgi-bin/status.cgi?hostgroup=all&style=detail&sorttype=2&sortoption=3] and ensure that GPO nagios is available * If successful, enumerate any instageni-relevant errors currently outstanding in GPO nagios * Run omni getversion and listresources against the BBN rack ProtoGENI AM: {{{ omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am getversion omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am -V2 -t GENI 3 listresources -o }}} * Run omni getversion and listresources against the BBN rack FOAM AM: {{{ omni -a https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 getversion omni -a https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 listresources -o }}} * Verify that [http://monitor.gpolab.bbn.com/connectivity/campus.html] currently shows a successful connection from argos to instageni-vlan1750.bbn.dataplane.geni.net '''Verify:''' * Site admins should be able to login to boss, control, and foam, and perform sudo operations * The Emulab web UI should be available, and red dot mode should work * Monitoring of the rack should be available (though it may show some errors due to the disconnection) * Rack ProtoGENI and FOAM aggregates should respond to getversion and listresources * Existing experiments should continue to run, and should be able to respond to dataplane traffic ==== Results of testing Step 1B on 2013-03-08 ==== * On maple (lab subnet router), do: {{{ maple#conf t Enter configuration commands, one per line. End with CNTL/Z. maple(config)# ip access-list standard gst-4544-ig-adm-6 maple(config-std-nacl)# deny 155.98.33.0 0.0.0.255 maple(config-std-nacl)# permit any maple(config-std-nacl)# exit maple(config)# int gi 0/1.830 maple(config-subif)# ip access-group gst-4544-ig-adm-6 out maple(config-subif)# exit maple(config)#exit maple# }}} * Wait 5 minutes: wait until 11:26 * Attempt to SSH to boss.instageni.gpolab.bbn.com: works: {{{ capybara,[~],11:25(0)$ ssh boss.instageni.gpolab.bbn.com Last login: Fri Mar 8 11:08:54 2013 from capybara.bbn.com FreeBSD 9.0-STABLE (XEN) #4: Mon Nov 26 10:02:53 MST 2012 Welcome to ProtoGeni! Remember to put /usr/testbed/bin and /usr/testbed/sbin in your path! In order to search for a string in some files, use 'grep' like this: grep "string" filename1 [filename2 filename3 ...] This will print out the lines in the files that contain the string. grep can also do a lot more advanced searches - type 'man grep' for details. > sudo whoami root > }}} * As expected, emulab is now unreachable, but the general internet is still reachable: {{{ > ping ops.emulab.net PING ops.emulab.net (155.98.33.74): 56 data bytes ^C --- ops.emulab.net ping statistics --- 6 packets transmitted, 0 packets received, 100.0% packet loss > ping www.google.com PING www.google.com (173.194.76.106): 56 data bytes 64 bytes from 173.194.76.106: icmp_seq=0 ttl=46 time=23.398 ms 64 bytes from 173.194.76.106: icmp_seq=1 ttl=46 time=24.046 ms 64 bytes from 173.194.76.106: icmp_seq=2 ttl=46 time=23.147 ms ^C --- www.google.com ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 23.147/23.530/24.046/0.379 ms > }}} * Attempt to SSH to gpolab.control-nodes.geniracks.net: {{{ capybara,[~],11:27(0)$ ssh gpolab.control-nodes.geniracks.net Welcome to Ubuntu precise (development branch) (GNU/Linux 3.2.0-23-generic x86_64) * Documentation: https://help.ubuntu.com/ System information as of Fri Mar 8 09:27:22 MST 2013 System load: 0.08 Users logged in: 0 Usage of /: 24.7% of 5.85GB IP address for xenbr0: 192.1.242.130 Memory usage: 42% IP address for xenbr1: 10.1.1.254 Swap usage: 0% IP address for xenbr2: 10.2.1.254 Processes: 144 IP address for xenbr3: 10.3.1.254 Graph this data and manage this system at https://landscape.canonical.com/ Last login: Fri Mar 8 09:27:09 2013 from capybara.bbn.com gpolab,[~],09:27(0)$ sudo whoami root gpolab,[~],09:27(0)$ }}} * Attempt to SSH to foam.instageni.gpolab.bbn.com: {{{ capybara,[~],11:27(0)$ ssh foam.instageni.gpolab.bbn.com Welcome to Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-34-generic x86_64) * Documentation: https://help.ubuntu.com/ Last login: Fri Mar 8 11:09:34 2013 from capybara.bbn.com foam,[~],11:27(0)$ sudo whoami root foam,[~],11:27(0)$ }}} * Browse to [https://instageni.gpolab.bbn.com/] and attempt to login as chaos: huh, something looks funny about the login screen, maybe an icon is missing? I screencapped it to compare later. * Ah, when i went to login, it suggested i was already logged in. So that's probably the weirdness --- going directly to instageni.gpolab.bbn.com/ (instead of boss.), while logged in. I'll try that out. * If successful, enter red dot mode: fine * Browse to [http://monitor.gpolab.bbn.com/nagios/cgi-bin/status.cgi?hostgroup=all&style=detail&sorttype=2&sortoption=3] and ensure that GPO nagios is available * If successful, enumerate any instageni-relevant errors currently outstanding in GPO nagios: no new instageni-related errors * Run omni getversion and listresources against the BBN rack ProtoGENI AM: {{{ jericho,[~],11:20(0)$ omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am getversion INFO:omni:Loading config file /home/chaos/omni/omni_pgeni INFO:omni:Using control framework pg INFO:omni:AM URN: unspecified_AM_URN (url: https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am) has version: INFO:omni:{ 'code': { 'am_code': 0, 'am_type': 'protogeni', 'geni_code': 0, 'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+1e6e7e527427c03b90295b6559f59491', 'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=1e6e7e527427c03b90295b6559f59491'}, 'geni_api': 2, 'output': '', 'value': { 'code_tag': 'b8883d9be04ae81a2180349b95501e30078abb1f', 'geni_ad_rspec_versions': [ { 'extensions': [], 'namespace': 'http://www.protogeni.net/resources/rspec/0.1', 'schema': 'http://www.protogeni.net/resources/rspec/0.1/ad.xsd', 'type': 'ProtoGENI', 'version': '0.1'}, { 'extensions': [], 'namespace': 'http://www.protogeni.net/resources/rspec/0.2', 'schema': 'http://www.protogeni.net/resources/rspec/0.2/ad.xsd', 'type': 'ProtoGENI', 'version': '0.2'}, { 'extensions': [ 'http://www.protogeni.net/resources/rspec/ext/emulab/1'], 'namespace': 'http://www.protogeni.net/resources/rspec/2', 'schema': 'http://www.protogeni.net/resources/rspec/2/ad.xsd', 'type': 'ProtoGENI', 'version': '2'}, { 'extensions': [ 'http://www.protogeni.net/resources/rspec/ext/emulab/1'], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd', 'type': 'GENI', 'version': '3'}], 'geni_api': 2, 'geni_api_versions': { '1': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/1.0', '2': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0', '3': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/3.0'}, 'geni_request_rspec_versions': [ { 'extensions': [], 'namespace': 'http://www.protogeni.net/resources/rspec/0.1', 'schema': 'http://www.protogeni.net/resources/rspec/0.1/request.xsd', 'type': 'ProtoGENI', 'version': '0.1'}, { 'extensions': [], 'namespace': 'http://www.protogeni.net/resources/rspec/0.2', 'schema': 'http://www.protogeni.net/resources/rspec/0.2/request.xsd', 'type': 'ProtoGENI', 'version': '0.2'}, { 'extensions': [ 'http://www.protogeni.net/resources/rspec/ext/emulab/1'], 'namespace': 'http://www.protogeni.net/resources/rspec/2', 'schema': 'http://www.protogeni.net/resources/rspec/2/request.xsd', 'type': 'ProtoGENI', 'version': '2'}, { 'extensions': [ 'http://www.protogeni.net/resources/rspec/ext/emulab/1'], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/request.xsd', 'type': 'GENI', 'version': '3'}], 'hostname': 'boss.instageni.gpolab.bbn.com', 'hrn': 'bbninstageni.cm', 'interface': 'aggregate', 'url': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am', 'urn': 'urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm'}} INFO:omni: ------------------------------------------------------------ INFO:omni: Completed getversion: Options as run: aggregate: https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am configfile: /home/chaos/omni/omni_pgeni framework: pg native: True Args: getversion Result Summary: Got version for 1 out of 1 aggregates INFO:omni: ============================================================ jericho,[~],11:34(0)$ omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am -V2 -t GENI 3 listresources -o INFO:omni:Loading config file /home/chaos/omni/omni_pgeni INFO:omni:Using control framework pg INFO:omni:Saving output to a file. INFO:omni:Listed resources on 1 out of 1 possible aggregates. INFO:omni:Writing to 'rspec-instageni-gpolab-bbn-com-12369-protogeni.xml' INFO:omni: ------------------------------------------------------------ INFO:omni: Completed listresources: Options as run: aggregate: https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am api_version: 2 configfile: /home/chaos/omni/omni_pgeni framework: pg native: True output: True rspectype: ('GENI', '3') Args: listresources Result Summary: Retrieved resources from 1 aggregates. Wrote rspecs from 1 aggregates to 1 files Saved listresources RSpec at 'unspecified_AM_URN' to file rspec-instageni-gpolab-bbn-com-12369-protogeni.xml; . INFO:omni: ============================================================ jericho,[~],11:35(0)$ mv rspec-instageni-gpolab-bbn-com-12369-protogeni.xml rspec-instageni-gpolab-bbn-com-12369-protogeni-IG-ADM-6-02.xml jericho,[~],11:35(0)$ }}} * Run omni getversion and listresources against the BBN rack FOAM AM: {{{ jericho,[~],11:35(0)$ omni -a https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 getversion INFO:omni:Loading config file /home/chaos/omni/omni_pgeni INFO:omni:Using control framework pg INFO:omni:AM URN: unspecified_AM_URN (url: https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1) has version: INFO:omni:{ 'ad_rspec_versions': [ { 'extensions': [ 'http://www.geni.net/resources/rspec/ext/openflow/3'], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd', 'type': 'GENI', 'version': '3'}], 'foam_version': '0.8.2', 'geni_api': 1, 'request_rspec_versions': [ { 'extensions': [ 'http://www.geni.net/resources/rspec/ext/openflow/3', 'http://www.geni.net/resources/rspec/ext/openflow/4', 'http://www.geni.net/resources/rspec/ext/flowvisor/1'], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/request.xsd', 'type': 'GENI', 'version': '3'}], 'site_info': { 'admin-email': 'instageni-foam-admin@gpolab.bbn.com'}} INFO:omni: ------------------------------------------------------------ INFO:omni: Completed getversion: Options as run: aggregate: https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 configfile: /home/chaos/omni/omni_pgeni framework: pg native: True Args: getversion Result Summary: Got version for 1 out of 1 aggregates INFO:omni: ============================================================ jericho,[~],11:36(0)$ omni -a https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 listresources -o INFO:omni:Loading config file /home/chaos/omni/omni_pgeni INFO:omni:Using control framework pg INFO:omni:Saving output to a file. INFO:omni:Listed resources on 1 out of 1 possible aggregates. INFO:omni:Writing to 'rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1.xml' INFO:omni: ------------------------------------------------------------ INFO:omni: Completed listresources: Options as run: aggregate: https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 configfile: /home/chaos/omni/omni_pgeni framework: pg native: True output: True Args: listresources Result Summary: Retrieved resources from 1 aggregates. Wrote rspecs from 1 aggregates to 1 files Saved listresources RSpec at 'unspecified_AM_URN' to file rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1.xml; . INFO:omni: ============================================================ jericho,[~],11:36(0)$ mv rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1.xml rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1-02.xml jericho,[~],11:36(0)$ }}} * Verify that [http://monitor.gpolab.bbn.com/connectivity/campus.html] currently shows a successful connection from argos to instageni-vlan1750.bbn.dataplane.geni.net: yep, looks fine '''Verify:''' * Site admins should be able to login to boss, control, and foam, and perform sudo operations * The Emulab web UI should be available, and red dot mode should work * Monitoring of the rack should be available (though it may show some errors due to the disconnection) * Rack ProtoGENI and FOAM aggregates should respond to getversion and listresources * Existing experiments should continue to run, and should be able to respond to dataplane traffic === Step 1C: undo the changes === ==== Overview of Step 1C ==== '''Using:''' * On maple (lab subnet router), do: {{{ conf t int gi 0/1.830 no ip access-group gst-4544-ig-adm-6 out exit no ip access-list standard gst-4544-ig-adm-6 exit }}} * Wait 5 minutes * Attempt to SSH to boss.instageni.gpolab.bbn.com: {{{ ssh boss.instageni.gpolab.bbn.com }}} * If SSH is successful, attempt to sudo on boss: {{{ sudo whoami }}} * If SSH is successful, ping ops.emulab.net to verify connectivity: {{{ ping ops.emulab.net }}} * Attempt to SSH to gpolab.control-nodes.geniracks.net: {{{ ssh gpolab.control-nodes.geniracks.net }}} * If SSH is successful, attempt to sudo on control: {{{ sudo whoami }}} * Attempt to SSH to foam.instageni.gpolab.bbn.com: {{{ ssh foam.instageni.gpolab.bbn.com }}} * If SSH is successful, attempt to sudo on foam: {{{ sudo whoami }}} * Browse to [https://instageni.gpolab.bbn.com/] and attempt to login as chaos * If successful, enter red dot mode * Browse to [http://monitor.gpolab.bbn.com/nagios/cgi-bin/status.cgi?hostgroup=all&style=detail&sorttype=2&sortoption=3] and ensure that GPO nagios is available * If successful, enumerate any instageni-relevant errors currently outstanding in GPO nagios * Run omni getversion and listresources against the BBN rack ProtoGENI AM: {{{ omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am getversion omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am -V2 -t GENI 3 listresources -o }}} * Run omni getversion and listresources against the BBN rack FOAM AM: {{{ omni -a https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 getversion omni -a https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 listresources -o }}} * Verify that [http://monitor.gpolab.bbn.com/connectivity/campus.html] currently shows a successful connection from argos to instageni-vlan1750.bbn.dataplane.geni.net '''Verify:''' * Site admins should be able to login to boss, control, and foam, and perform sudo operations * The Emulab web UI should be available, and red dot mode should work * Monitoring of the rack should be available (though it may show some errors due to the disconnection) * Rack ProtoGENI and FOAM aggregates should respond to getversion and listresources * Existing experiments should continue to run, and should be able to respond to dataplane traffic ==== Results of testing Step 1C on 2013-03-08 ==== '''Using:''' * On maple (lab subnet router), do: {{{ conf t int gi 0/1.830 no ip access-group gst-4544-ig-adm-6 out exit no ip access-list standard gst-4544-ig-adm-6 exit }}} * Wait 5 minutes: waiting until 11:44 * Attempt to SSH to boss.instageni.gpolab.bbn.com: {{{ capybara,[~],11:43(0)$ ssh boss.instageni.gpolab.bbn.com Last login: Fri Mar 8 11:25:58 2013 from capybara.bbn.com FreeBSD 9.0-STABLE (XEN) #4: Mon Nov 26 10:02:53 MST 2012 Welcome to ProtoGeni! Remember to put /usr/testbed/bin and /usr/testbed/sbin in your path! You can automatically download and install binary packages by doing pkg_add -r where you replace with the URL to the package. This will also automatically install the packages the package you download is dependent on (ie, the packages it needs in order to work.) > sudo whoami root > }}} * If SSH is successful, ping ops.emulab.net to verify connectivity: {{{ > ping ops.emulab.net PING ops.emulab.net (155.98.33.74): 56 data bytes 64 bytes from 155.98.33.74: icmp_seq=0 ttl=47 time=67.399 ms 64 bytes from 155.98.33.74: icmp_seq=1 ttl=47 time=59.152 ms 64 bytes from 155.98.33.74: icmp_seq=2 ttl=47 time=59.708 ms ^C --- ops.emulab.net ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 59.152/62.086/67.399/3.763 ms > }}} * Attempt to SSH to gpolab.control-nodes.geniracks.net: {{{ capybara,[~],11:44(0)$ ssh gpolab.control-nodes.geniracks.net Welcome to Ubuntu precise (development branch) (GNU/Linux 3.2.0-23-generic x86_64) * Documentation: https://help.ubuntu.com/ System information as of Fri Mar 8 09:44:38 MST 2013 System load: 0.02 Users logged in: 0 Usage of /: 24.7% of 5.85GB IP address for xenbr0: 192.1.242.130 Memory usage: 42% IP address for xenbr1: 10.1.1.254 Swap usage: 0% IP address for xenbr2: 10.2.1.254 Processes: 144 IP address for xenbr3: 10.3.1.254 Graph this data and manage this system at https://landscape.canonical.com/ Last login: Fri Mar 8 09:44:30 2013 from capybara.bbn.com gpolab,[~],09:44(0)$ sudo whoami root gpolab,[~],09:44(0)$ }}} * Attempt to SSH to foam.instageni.gpolab.bbn.com: {{{ capybara,[~],11:44(0)$ ssh foam.instageni.gpolab.bbn.com Welcome to Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-34-generic x86_64) * Documentation: https://help.ubuntu.com/ Last login: Fri Mar 8 11:27:40 2013 from capybara.bbn.com foam,[~],11:44(0)$ sudo whoami root foam,[~],11:45(0)$ }}} * Browse to [https://instageni.gpolab.bbn.com/] and attempt to login as chaos * If successful, enter red dot mode: yep * While playing with this, i verified that the weird visual effect has to do with a particular pattern of browsing, and is reproducible while emulab.net is available. I'll report it as a bug, but it's not this bug. * Browse to [http://monitor.gpolab.bbn.com/nagios/cgi-bin/status.cgi?hostgroup=all&style=detail&sorttype=2&sortoption=3] and ensure that GPO nagios is available * If successful, enumerate any instageni-relevant errors currently outstanding in GPO nagios: `TANGO-foam.instageni.gpolab.bbn.com/gmoc_monitoring` is having problems submitting data, due to an unrelated GMOC maintenance which ran over. * Run omni getversion and listresources against the BBN rack ProtoGENI AM: {{{ jericho,[~],11:36(0)$ omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am getversion INFO:omni:Loading config file /home/chaos/omni/omni_pgeni INFO:omni:Using control framework pg INFO:omni:AM URN: unspecified_AM_URN (url: https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am) has version: INFO:omni:{ 'code': { 'am_code': 0, 'am_type': 'protogeni', 'geni_code': 0, 'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+c91268c3fed57bf7807b20802ba3c8b2', 'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=c91268c3fed57bf7807b20802ba3c8b2'}, 'geni_api': 2, 'output': '', 'value': { 'code_tag': 'b8883d9be04ae81a2180349b95501e30078abb1f', 'geni_ad_rspec_versions': [ { 'extensions': [], 'namespace': 'http://www.protogeni.net/resources/rspec/0.1', 'schema': 'http://www.protogeni.net/resources/rspec/0.1/ad.xsd', 'type': 'ProtoGENI', 'version': '0.1'}, { 'extensions': [], 'namespace': 'http://www.protogeni.net/resources/rspec/0.2', 'schema': 'http://www.protogeni.net/resources/rspec/0.2/ad.xsd', 'type': 'ProtoGENI', 'version': '0.2'}, { 'extensions': [ 'http://www.protogeni.net/resources/rspec/ext/emulab/1'], 'namespace': 'http://www.protogeni.net/resources/rspec/2', 'schema': 'http://www.protogeni.net/resources/rspec/2/ad.xsd', 'type': 'ProtoGENI', 'version': '2'}, { 'extensions': [ 'http://www.protogeni.net/resources/rspec/ext/emulab/1'], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd', 'type': 'GENI', 'version': '3'}], 'geni_api': 2, 'geni_api_versions': { '1': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/1.0', '2': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0', '3': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/3.0'}, 'geni_request_rspec_versions': [ { 'extensions': [], 'namespace': 'http://www.protogeni.net/resources/rspec/0.1', 'schema': 'http://www.protogeni.net/resources/rspec/0.1/request.xsd', 'type': 'ProtoGENI', 'version': '0.1'}, { 'extensions': [], 'namespace': 'http://www.protogeni.net/resources/rspec/0.2', 'schema': 'http://www.protogeni.net/resources/rspec/0.2/request.xsd', 'type': 'ProtoGENI', 'version': '0.2'}, { 'extensions': [ 'http://www.protogeni.net/resources/rspec/ext/emulab/1'], 'namespace': 'http://www.protogeni.net/resources/rspec/2', 'schema': 'http://www.protogeni.net/resources/rspec/2/request.xsd', 'type': 'ProtoGENI', 'version': '2'}, { 'extensions': [ 'http://www.protogeni.net/resources/rspec/ext/emulab/1'], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/request.xsd', 'type': 'GENI', 'version': '3'}], 'hostname': 'boss.instageni.gpolab.bbn.com', 'hrn': 'bbninstageni.cm', 'interface': 'aggregate', 'url': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am', 'urn': 'urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm'}} INFO:omni: ------------------------------------------------------------ INFO:omni: Completed getversion: Options as run: aggregate: https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am configfile: /home/chaos/omni/omni_pgeni framework: pg native: True Args: getversion Result Summary: Got version for 1 out of 1 aggregates INFO:omni: ============================================================ jericho,[~],11:49(0)$ omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am -V2 -t GENI 3 listresources -o INFO:omni:Loading config file /home/chaos/omni/omni_pgeni INFO:omni:Using control framework pg INFO:omni:Saving output to a file. INFO:omni:Listed resources on 1 out of 1 possible aggregates. INFO:omni:Writing to 'rspec-instageni-gpolab-bbn-com-12369-protogeni.xml' INFO:omni: ------------------------------------------------------------ INFO:omni: Completed listresources: Options as run: aggregate: https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am api_version: 2 configfile: /home/chaos/omni/omni_pgeni framework: pg native: True output: True rspectype: ('GENI', '3') Args: listresources Result Summary: Retrieved resources from 1 aggregates. Wrote rspecs from 1 aggregates to 1 files Saved listresources RSpec at 'unspecified_AM_URN' to file rspec-instageni-gpolab-bbn-com-12369-protogeni.xml; . INFO:omni: ============================================================ jericho,[~],11:49(0)$ mv rspec-instageni-gpolab-bbn-com-12369-protogeni.xml rspec-instageni-gpolab-bbn-com-12369-protogeni-IG-ADM-6-03.xml jericho,[~],11:50(0)$ }}} * Run omni getversion and listresources against the BBN rack FOAM AM: {{{ jericho,[~],11:50(0)$ omni -a https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 getversion INFO:omni:Loading config file /home/chaos/omni/omni_pgeni INFO:omni:Using control framework pg INFO:omni:AM URN: unspecified_AM_URN (url: https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1) has version: INFO:omni:{ 'ad_rspec_versions': [ { 'extensions': [ 'http://www.geni.net/resources/rspec/ext/openflow/3'], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd', 'type': 'GENI', 'version': '3'}], 'foam_version': '0.8.2', 'geni_api': 1, 'request_rspec_versions': [ { 'extensions': [ 'http://www.geni.net/resources/rspec/ext/openflow/3', 'http://www.geni.net/resources/rspec/ext/openflow/4', 'http://www.geni.net/resources/rspec/ext/flowvisor/1'], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/request.xsd', 'type': 'GENI', 'version': '3'}], 'site_info': { 'admin-email': 'instageni-foam-admin@gpolab.bbn.com'}} INFO:omni: ------------------------------------------------------------ INFO:omni: Completed getversion: Options as run: aggregate: https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 configfile: /home/chaos/omni/omni_pgeni framework: pg native: True Args: getversion Result Summary: Got version for 1 out of 1 aggregates INFO:omni: ============================================================ jericho,[~],11:50(0)$ omni -a https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 listresources -o INFO:omni:Loading config file /home/chaos/omni/omni_pgeni INFO:omni:Using control framework pg INFO:omni:Saving output to a file. INFO:omni:Listed resources on 1 out of 1 possible aggregates. INFO:omni:Writing to 'rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1.xml' INFO:omni: ------------------------------------------------------------ INFO:omni: Completed listresources: Options as run: aggregate: https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 configfile: /home/chaos/omni/omni_pgeni framework: pg native: True output: True Args: listresources Result Summary: Retrieved resources from 1 aggregates. Wrote rspecs from 1 aggregates to 1 files Saved listresources RSpec at 'unspecified_AM_URN' to file rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1.xml; . INFO:omni: ============================================================ jericho,[~],11:50(0)$ mv rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1.xml rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1-IG-ADM-6-03.xml jericho,[~],11:50(0)$ }}} * Verify that [http://monitor.gpolab.bbn.com/connectivity/campus.html] currently shows a successful connection from argos to instageni-vlan1750.bbn.dataplane.geni.net: yep '''Verify:''' * Site admins should be able to login to boss, control, and foam, and perform sudo operations * The Emulab web UI should be available, and red dot mode should work * Monitoring of the rack should be available (though it may show some errors due to the disconnection) * Rack ProtoGENI and FOAM aggregates should respond to getversion and listresources * Existing experiments should continue to run, and should be able to respond to dataplane traffic I compared the rspecs too --- the only differences in the compute ones were the timestamps: {{{ jericho,[~],11:53(0)$ diff rspec-instageni-gpolab-bbn-com-12369-protogeni-IG-ADM-6-01.xml rspec-instageni-gpolab-bbn-com-12369-protogeni-IG-ADM-6-02.xml 6c6 < --- > 736c736 < --- > 743,744c743,744 < 2013-03-08T16:18:30Z < 2013-03-08T16:18:30Z --- > 2013-03-08T16:35:02Z > 2013-03-08T16:35:02Z }}} and there were no differences in the FOAM ones.