Changes between Version 2 and Version 3 of GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-ADM-6


Ignore:
Timestamp:
03/08/13 11:51:50 (11 years ago)
Author:
chaos@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-ADM-6

    v2 v3  
    55''This page is GPO's working page for performing IG-ADM-6.  It is public for informational purposes, but it is not an official status report.  See [wiki:GENIRacksHome/InstageniRacks/AcceptanceTestStatus] for the current status of InstaGENI acceptance tests.''
    66
    7 ''Last substantive edit of this page: 2013-02-28''
     7''Last substantive edit of this page: 2013-03-08''
    88
    99== Page format ==
    … …  
    5050   * If SSH is successful, attempt to sudo on boss:
    5151{{{
    52 sudo -v
     52sudo whoami
    5353}}}
    5454   * If SSH is successful, ping ops.emulab.net to verify connectivity:
    … …  
    6262 * If SSH is successful, attempt to sudo on control:
    6363{{{
    64 sudo -v
     64sudo whoami
    6565}}}
    6666 * Attempt to SSH to foam.instageni.gpolab.bbn.com:
    … …  
    7070 * If SSH is successful, attempt to sudo on foam:
    7171{{{
    72 sudo -v
     72sudo whoami
    7373}}}
    7474 * Browse to [https://instageni.gpolab.bbn.com/] and attempt to login as chaos
    … …  
    7878 * Run omni getversion and listresources against the BBN rack ProtoGENI AM:
    7979{{{
    80 omni -a https://utah.geniracks.net:12369/protogeni/xmlrpc/am getversion
    81 omni -a https://utah.geniracks.net:12369/protogeni/xmlrpc/am listresources -o
     80omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am getversion
     81omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am listresources -o
    8282}}}
    8383 * Run omni getversion and listresources against the BBN rack FOAM AM:
    … …  
    8787}}}
    8888 * Verify that [http://monitor.gpolab.bbn.com/connectivity/campus.html] currently shows a successful connection from argos to instageni-vlan1750.bbn.dataplane.geni.net
     89
     90==== Results of testing Step 1A on 2013-03-08 ====
     91
     92 * SSH and sudo on boss look fine:
     93{{{
     94capybara,[~],11:07(0)$ ssh boss.instageni.gpolab.bbn.com
     95Last login: Fri Mar  8 11:06:19 2013 from capybara.bbn.com
     96FreeBSD 9.0-STABLE (XEN) #4: Mon Nov 26 10:02:53 MST 2012
     97
     98Welcome to ProtoGeni!
     99
     100Remember to put /usr/testbed/bin and /usr/testbed/sbin in your path!
     101
     102
     103Need to print a manpage? Use
     104
     105        man name_of_manpage | col -bx | lpr
     106                -- Dru <genesis@istar.ca>
     107> sudo whoami
     108root
     109>
     110}}}
     111   * If SSH is successful, ping ops.emulab.net to verify connectivity: fine:
     112{{{
     113> ping ops.emulab.net
     114PING ops.emulab.net (155.98.33.74): 56 data bytes
     11564 bytes from 155.98.33.74: icmp_seq=0 ttl=47 time=61.098 ms
     11664 bytes from 155.98.33.74: icmp_seq=1 ttl=47 time=66.921 ms
     11764 bytes from 155.98.33.74: icmp_seq=2 ttl=47 time=67.713 ms
     118^C
     119--- ops.emulab.net ping statistics ---
     1203 packets transmitted, 3 packets received, 0.0% packet loss
     121round-trip min/avg/max/stddev = 61.098/65.244/67.713/2.949 ms
     122>
     123}}}
     124 * Attempt to SSH to gpolab.control-nodes.geniracks.net: fine:
     125{{{
     126capybara,[~],11:08(0)$ ssh gpolab.control-nodes.geniracks.net
     127Welcome to Ubuntu precise (development branch) (GNU/Linux 3.2.0-23-generic x86_64)
     128
     129 * Documentation:  https://help.ubuntu.com/
     130
     131  System information as of Fri Mar  8 09:08:02 MST 2013
     132
     133  System load:  0.0               Users logged in:       0
     134  Usage of /:   24.7% of 5.85GB   IP address for xenbr0: 192.1.242.130
     135  Memory usage: 42%               IP address for xenbr1: 10.1.1.254
     136  Swap usage:   0%                IP address for xenbr2: 10.2.1.254
     137  Processes:    143               IP address for xenbr3: 10.3.1.254
     138
     139  Graph this data and manage this system at https://landscape.canonical.com/
     140Last login: Thu Feb 28 07:26:30 2013 from capybara.bbn.com
     141gpolab,[~],09:08(0)$ sudo whoami
     142root
     143gpolab,[~],09:08(0)$
     144}}}
     145 * Attempt to SSH to foam.instageni.gpolab.bbn.com: fine:
     146{{{
     147capybara,[~],11:09(0)$ ssh foam.instageni.gpolab.bbn.com
     148Welcome to Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-34-generic x86_64)
     149
     150 * Documentation:  https://help.ubuntu.com/
     151Last login: Wed Jan 23 10:01:30 2013 from capybara.bbn.com
     152foam,[~],11:09(0)$ sudo whoami
     153root
     154foam,[~],11:09(0)$
     155}}}
     156 * Browse to [https://instageni.gpolab.bbn.com/] and attempt to login as chaos: fine, though i got a security certificate warning from firefox --- i guess i usually go to `https://boss.instageni.gpolab.bbn.com`.  Anyway, login worked fine.
     157   * If successful, enter red dot mode: yep
     158 * Browse to [http://monitor.gpolab.bbn.com/nagios/cgi-bin/status.cgi?hostgroup=all&style=detail&sorttype=2&sortoption=3] and ensure that GPO nagios is available: fine
     159   * If successful, enumerate any instageni-relevant errors currently outstanding in GPO nagios: no service errors related to anything ending in `instageni.gpolab.bbn.com`
     160 * Run omni getversion and listresources against the BBN rack ProtoGENI AM:
     161{{{
     162jericho,[~],11:13(0)$ omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am getversion
     163INFO:omni:Loading config file /home/chaos/omni/omni_pgeni
     164INFO:omni:Using control framework pg
     165INFO:omni:AM URN: unspecified_AM_URN (url: https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am) has version:
     166INFO:omni:{   'code': {   'am_code': 0,
     167                'am_type': 'protogeni',
     168                'geni_code': 0,
     169                'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+ff955d693a123012969aa7f3a55aa222',
     170                'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=ff955d693a123012969aa7f3a55aa222'},
     171    'geni_api': 2,
     172    'output': '',
     173    'value': {   'code_tag': 'b8883d9be04ae81a2180349b95501e30078abb1f',
     174                 'geni_ad_rspec_versions': [   {   'extensions': [],
     175                                                   'namespace': 'http://www.protogeni.net/resources/rspec/0.1',
     176                                                   'schema': 'http://www.protogeni.net/resources/rspec/0.1/ad.xsd',
     177                                                   'type': 'ProtoGENI',
     178                                                   'version': '0.1'},
     179                                               {   'extensions': [],
     180                                                   'namespace': 'http://www.protogeni.net/resources/rspec/0.2',
     181                                                   'schema': 'http://www.protogeni.net/resources/rspec/0.2/ad.xsd',
     182                                                   'type': 'ProtoGENI',
     183                                                   'version': '0.2'},
     184                                               {   'extensions': [   'http://www.protogeni.net/resources/rspec/ext/emulab/1'],
     185                                                   'namespace': 'http://www.protogeni.net/resources/rspec/2',
     186                                                   'schema': 'http://www.protogeni.net/resources/rspec/2/ad.xsd',
     187                                                   'type': 'ProtoGENI',
     188                                                   'version': '2'},
     189                                               {   'extensions': [   'http://www.protogeni.net/resources/rspec/ext/emulab/1'],
     190                                                   'namespace': 'http://www.geni.net/resources/rspec/3',
     191                                                   'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd',
     192                                                   'type': 'GENI',
     193                                                   'version': '3'}],
     194                 'geni_api': 2,
     195                 'geni_api_versions': {   '1': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/1.0',
     196                                          '2': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0',
     197                                          '3': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/3.0'},
     198                 'geni_request_rspec_versions': [   {   'extensions': [],
     199                                                        'namespace': 'http://www.protogeni.net/resources/rspec/0.1',
     200                                                        'schema': 'http://www.protogeni.net/resources/rspec/0.1/request.xsd',
     201                                                        'type': 'ProtoGENI',
     202                                                        'version': '0.1'},
     203                                                    {   'extensions': [],
     204                                                        'namespace': 'http://www.protogeni.net/resources/rspec/0.2',
     205                                                        'schema': 'http://www.protogeni.net/resources/rspec/0.2/request.xsd',
     206                                                        'type': 'ProtoGENI',
     207                                                        'version': '0.2'},
     208                                                    {   'extensions': [   'http://www.protogeni.net/resources/rspec/ext/emulab/1'],
     209                                                        'namespace': 'http://www.protogeni.net/resources/rspec/2',
     210                                                        'schema': 'http://www.protogeni.net/resources/rspec/2/request.xsd',
     211                                                        'type': 'ProtoGENI',
     212                                                        'version': '2'},
     213                                                    {   'extensions': [   'http://www.protogeni.net/resources/rspec/ext/emulab/1'],
     214                                                        'namespace': 'http://www.geni.net/resources/rspec/3',
     215                                                        'schema': 'http://www.geni.net/resources/rspec/3/request.xsd',
     216                                                        'type': 'GENI',
     217                                                        'version': '3'}],
     218                 'hostname': 'boss.instageni.gpolab.bbn.com',
     219                 'hrn': 'bbninstageni.cm',
     220                 'interface': 'aggregate',
     221                 'url': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am',
     222                 'urn': 'urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm'}}
     223INFO:omni: ------------------------------------------------------------
     224INFO:omni: Completed getversion:
     225
     226  Options as run:
     227                aggregate: https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am
     228                configfile: /home/chaos/omni/omni_pgeni
     229                framework: pg
     230                native: True
     231
     232  Args: getversion
     233
     234  Result Summary:
     235Got version for 1 out of 1 aggregates
     236 
     237INFO:omni: ============================================================
     238jericho,[~],11:13(0)$
     239
     240jericho,[~],11:16(0)$ omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am -V2 -t GENI 3 listresources -o
     241INFO:omni:Loading config file /home/chaos/omni/omni_pgeni
     242INFO:omni:Using control framework pg
     243INFO:omni:Saving output to a file.
     244INFO:omni:Listed resources on 1 out of 1 possible aggregates.
     245INFO:omni:Writing to 'rspec-instageni-gpolab-bbn-com-12369-protogeni.xml'
     246INFO:omni: ------------------------------------------------------------
     247INFO:omni: Completed listresources:
     248
     249  Options as run:
     250                aggregate: https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am
     251                api_version: 2
     252                configfile: /home/chaos/omni/omni_pgeni
     253                framework: pg
     254                native: True
     255                output: True
     256                rspectype: ('GENI', '3')
     257
     258  Args: listresources
     259
     260  Result Summary: Retrieved resources from 1 aggregates.
     261Wrote rspecs from 1 aggregates to 1 files
     262Saved listresources RSpec at 'unspecified_AM_URN' to file rspec-instageni-gpolab-bbn-com-12369-protogeni.xml; .
     263INFO:omni: ============================================================
     264jericho,[~],11:18(0)$ mv rspec-instageni-gpolab-bbn-com-12369-protogeni.xml rspec-instageni-gpolab-bbn-com-12369-protogeni-IG-ADM-6-01.xml
     265jericho,[~],11:18(0)$
     266}}}
     267 * Run omni getversion and listresources against the BBN rack FOAM AM:
     268{{{
     269jericho,[~],11:19(0)$ omni -a https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 getversion
     270INFO:omni:Loading config file /home/chaos/omni/omni_pgeni
     271INFO:omni:Using control framework pg
     272INFO:omni:AM URN: unspecified_AM_URN (url: https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1) has version:
     273INFO:omni:{   'ad_rspec_versions': [   {   'extensions': [   'http://www.geni.net/resources/rspec/ext/openflow/3'],
     274                                 'namespace': 'http://www.geni.net/resources/rspec/3',
     275                                 'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd',
     276                                 'type': 'GENI',
     277                                 'version': '3'}],
     278    'foam_version': '0.8.2',
     279    'geni_api': 1,
     280    'request_rspec_versions': [   {   'extensions': [   'http://www.geni.net/resources/rspec/ext/openflow/3',
     281                                                        'http://www.geni.net/resources/rspec/ext/openflow/4',
     282                                                        'http://www.geni.net/resources/rspec/ext/flowvisor/1'],
     283                                      'namespace': 'http://www.geni.net/resources/rspec/3',
     284                                      'schema': 'http://www.geni.net/resources/rspec/3/request.xsd',
     285                                      'type': 'GENI',
     286                                      'version': '3'}],
     287    'site_info': {   'admin-email': 'instageni-foam-admin@gpolab.bbn.com'}}
     288INFO:omni: ------------------------------------------------------------
     289INFO:omni: Completed getversion:
     290
     291  Options as run:
     292                aggregate: https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1
     293                configfile: /home/chaos/omni/omni_pgeni
     294                framework: pg
     295                native: True
     296
     297  Args: getversion
     298
     299  Result Summary:
     300Got version for 1 out of 1 aggregates
     301 
     302INFO:omni: ============================================================
     303jericho,[~],11:19(0)$ omni -a https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 listresources -o
     304INFO:omni:Loading config file /home/chaos/omni/omni_pgeni
     305INFO:omni:Using control framework pg
     306INFO:omni:Saving output to a file.
     307INFO:omni:Listed resources on 1 out of 1 possible aggregates.
     308INFO:omni:Writing to 'rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1.xml'
     309INFO:omni: ------------------------------------------------------------
     310INFO:omni: Completed listresources:
     311
     312  Options as run:
     313                aggregate: https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1
     314                configfile: /home/chaos/omni/omni_pgeni
     315                framework: pg
     316                native: True
     317                output: True
     318
     319  Args: listresources
     320
     321  Result Summary: Retrieved resources from 1 aggregates.
     322Wrote rspecs from 1 aggregates to 1 files
     323Saved listresources RSpec at 'unspecified_AM_URN' to file rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1.xml; .
     324INFO:omni: ============================================================
     325jericho,[~],11:19(0)$ mv rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1.xml rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1-IG-ADM-6-01.xml
     326jericho,[~],11:20(0)$
     327}}}
     328 * Verify that [http://monitor.gpolab.bbn.com/connectivity/campus.html] currently shows a successful connection from argos to instageni-vlan1750.bbn.dataplane.geni.net: yep
    89329
    90330=== Step 1B: simulate a network problem ===
    … …  
    112352   * If SSH is successful, attempt to sudo on boss:
    113353{{{
    114 sudo -v
     354sudo whoami
    115355}}}
    116356   * If SSH is successful, ping ops.emulab.net to verify connectivity:
    … …  
    124364 * If SSH is successful, attempt to sudo on control:
    125365{{{
    126 sudo -v
     366sudo whoami
    127367}}}
    128368 * Attempt to SSH to foam.instageni.gpolab.bbn.com:
    … …  
    132372 * If SSH is successful, attempt to sudo on foam:
    133373{{{
    134 sudo -v
     374sudo whoami
    135375}}}
    136376 * Browse to [https://instageni.gpolab.bbn.com/] and attempt to login as chaos
    … …  
    140380 * Run omni getversion and listresources against the BBN rack ProtoGENI AM:
    141381{{{
    142 omni -a https://utah.geniracks.net:12369/protogeni/xmlrpc/am getversion
    143 omni -a https://utah.geniracks.net:12369/protogeni/xmlrpc/am listresources -o
     382omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am getversion
     383omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am -V2 -t GENI 3 listresources -o
    144384}}}
    145385 * Run omni getversion and listresources against the BBN rack FOAM AM:
    … …  
    158398 * Existing experiments should continue to run, and should be able to respond to dataplane traffic
    159399
     400==== Results of testing Step 1B on 2013-03-08 ====
     401
     402 * On maple (lab subnet router), do:
     403{{{
     404maple#conf t
     405Enter configuration commands, one per line.  End with CNTL/Z.
     406maple(config)#  ip access-list standard gst-4544-ig-adm-6
     407maple(config-std-nacl)#    deny 155.98.33.0 0.0.0.255
     408maple(config-std-nacl)#    permit any
     409maple(config-std-nacl)#  exit
     410maple(config)#  int gi 0/1.830
     411maple(config-subif)#    ip access-group gst-4544-ig-adm-6 out
     412maple(config-subif)#  exit
     413maple(config)#exit
     414maple#
     415}}}
     416 * Wait 5 minutes: wait until 11:26
     417 * Attempt to SSH to boss.instageni.gpolab.bbn.com: works:
     418{{{
     419capybara,[~],11:25(0)$ ssh boss.instageni.gpolab.bbn.com
     420Last login: Fri Mar  8 11:08:54 2013 from capybara.bbn.com
     421FreeBSD 9.0-STABLE (XEN) #4: Mon Nov 26 10:02:53 MST 2012
     422
     423Welcome to ProtoGeni!
     424
     425Remember to put /usr/testbed/bin and /usr/testbed/sbin in your path!
     426
     427
     428In order to search for a string in some files, use 'grep' like this:
     429
     430         grep "string" filename1 [filename2 filename3 ...]
     431
     432This will print out the lines in the files that contain the string.  grep can
     433also do a lot more advanced searches - type 'man grep' for details.
     434> sudo whoami
     435root
     436>
     437}}}
     438   * As expected, emulab is now unreachable, but the general internet is still reachable:
     439{{{
     440> ping ops.emulab.net
     441PING ops.emulab.net (155.98.33.74): 56 data bytes
     442^C
     443--- ops.emulab.net ping statistics ---
     4446 packets transmitted, 0 packets received, 100.0% packet loss
     445> ping www.google.com
     446PING www.google.com (173.194.76.106): 56 data bytes
     44764 bytes from 173.194.76.106: icmp_seq=0 ttl=46 time=23.398 ms
     44864 bytes from 173.194.76.106: icmp_seq=1 ttl=46 time=24.046 ms
     44964 bytes from 173.194.76.106: icmp_seq=2 ttl=46 time=23.147 ms
     450^C
     451--- www.google.com ping statistics ---
     4523 packets transmitted, 3 packets received, 0.0% packet loss
     453round-trip min/avg/max/stddev = 23.147/23.530/24.046/0.379 ms
     454>
     455}}}
     456 * Attempt to SSH to gpolab.control-nodes.geniracks.net:
     457{{{
     458capybara,[~],11:27(0)$ ssh gpolab.control-nodes.geniracks.net
     459Welcome to Ubuntu precise (development branch) (GNU/Linux 3.2.0-23-generic x86_64)
     460
     461 * Documentation:  https://help.ubuntu.com/
     462
     463  System information as of Fri Mar  8 09:27:22 MST 2013
     464
     465  System load:  0.08              Users logged in:       0
     466  Usage of /:   24.7% of 5.85GB   IP address for xenbr0: 192.1.242.130
     467  Memory usage: 42%               IP address for xenbr1: 10.1.1.254
     468  Swap usage:   0%                IP address for xenbr2: 10.2.1.254
     469  Processes:    144               IP address for xenbr3: 10.3.1.254
     470
     471  Graph this data and manage this system at https://landscape.canonical.com/
     472Last login: Fri Mar  8 09:27:09 2013 from capybara.bbn.com
     473gpolab,[~],09:27(0)$ sudo whoami
     474root
     475gpolab,[~],09:27(0)$
     476}}}
     477 * Attempt to SSH to foam.instageni.gpolab.bbn.com:
     478{{{
     479capybara,[~],11:27(0)$ ssh foam.instageni.gpolab.bbn.com
     480Welcome to Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-34-generic x86_64)
     481
     482 * Documentation:  https://help.ubuntu.com/
     483Last login: Fri Mar  8 11:09:34 2013 from capybara.bbn.com
     484foam,[~],11:27(0)$ sudo whoami
     485root
     486foam,[~],11:27(0)$
     487}}}
     488 * Browse to [https://instageni.gpolab.bbn.com/] and attempt to login as chaos: huh, something looks funny about the login screen, maybe an icon is missing?  I screencapped it to compare later.
     489   * Ah, when i went to login, it suggested i was already logged in.  So that's probably the weirdness --- going directly to instageni.gpolab.bbn.com/ (instead of boss.), while logged in.  I'll try that out.
     490   * If successful, enter red dot mode: fine
     491 * Browse to [http://monitor.gpolab.bbn.com/nagios/cgi-bin/status.cgi?hostgroup=all&style=detail&sorttype=2&sortoption=3] and ensure that GPO nagios is available
     492   * If successful, enumerate any instageni-relevant errors currently outstanding in GPO nagios: no new instageni-related errors
     493 * Run omni getversion and listresources against the BBN rack ProtoGENI AM:
     494{{{
     495jericho,[~],11:20(0)$ omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am getversion
     496INFO:omni:Loading config file /home/chaos/omni/omni_pgeni
     497INFO:omni:Using control framework pg
     498INFO:omni:AM URN: unspecified_AM_URN (url: https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am) has version:
     499INFO:omni:{   'code': {   'am_code': 0,
     500                'am_type': 'protogeni',
     501                'geni_code': 0,
     502                'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+1e6e7e527427c03b90295b6559f59491',
     503                'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=1e6e7e527427c03b90295b6559f59491'},
     504    'geni_api': 2,
     505    'output': '',
     506    'value': {   'code_tag': 'b8883d9be04ae81a2180349b95501e30078abb1f',
     507                 'geni_ad_rspec_versions': [   {   'extensions': [],
     508                                                   'namespace': 'http://www.protogeni.net/resources/rspec/0.1',
     509                                                   'schema': 'http://www.protogeni.net/resources/rspec/0.1/ad.xsd',
     510                                                   'type': 'ProtoGENI',
     511                                                   'version': '0.1'},
     512                                               {   'extensions': [],
     513                                                   'namespace': 'http://www.protogeni.net/resources/rspec/0.2',
     514                                                   'schema': 'http://www.protogeni.net/resources/rspec/0.2/ad.xsd',
     515                                                   'type': 'ProtoGENI',
     516                                                   'version': '0.2'},
     517                                               {   'extensions': [   'http://www.protogeni.net/resources/rspec/ext/emulab/1'],
     518                                                   'namespace': 'http://www.protogeni.net/resources/rspec/2',
     519                                                   'schema': 'http://www.protogeni.net/resources/rspec/2/ad.xsd',
     520                                                   'type': 'ProtoGENI',
     521                                                   'version': '2'},
     522                                               {   'extensions': [   'http://www.protogeni.net/resources/rspec/ext/emulab/1'],
     523                                                   'namespace': 'http://www.geni.net/resources/rspec/3',
     524                                                   'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd',
     525                                                   'type': 'GENI',
     526                                                   'version': '3'}],
     527                 'geni_api': 2,
     528                 'geni_api_versions': {   '1': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/1.0',
     529                                          '2': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0',
     530                                          '3': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/3.0'},
     531                 'geni_request_rspec_versions': [   {   'extensions': [],
     532                                                        'namespace': 'http://www.protogeni.net/resources/rspec/0.1',
     533                                                        'schema': 'http://www.protogeni.net/resources/rspec/0.1/request.xsd',
     534                                                        'type': 'ProtoGENI',
     535                                                        'version': '0.1'},
     536                                                    {   'extensions': [],
     537                                                        'namespace': 'http://www.protogeni.net/resources/rspec/0.2',
     538                                                        'schema': 'http://www.protogeni.net/resources/rspec/0.2/request.xsd',
     539                                                        'type': 'ProtoGENI',
     540                                                        'version': '0.2'},
     541                                                    {   'extensions': [   'http://www.protogeni.net/resources/rspec/ext/emulab/1'],
     542                                                        'namespace': 'http://www.protogeni.net/resources/rspec/2',
     543                                                        'schema': 'http://www.protogeni.net/resources/rspec/2/request.xsd',
     544                                                        'type': 'ProtoGENI',
     545                                                        'version': '2'},
     546                                                    {   'extensions': [   'http://www.protogeni.net/resources/rspec/ext/emulab/1'],
     547                                                        'namespace': 'http://www.geni.net/resources/rspec/3',
     548                                                        'schema': 'http://www.geni.net/resources/rspec/3/request.xsd',
     549                                                        'type': 'GENI',
     550                                                        'version': '3'}],
     551                 'hostname': 'boss.instageni.gpolab.bbn.com',
     552                 'hrn': 'bbninstageni.cm',
     553                 'interface': 'aggregate',
     554                 'url': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am',
     555                 'urn': 'urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm'}}
     556INFO:omni: ------------------------------------------------------------
     557INFO:omni: Completed getversion:
     558
     559  Options as run:
     560                aggregate: https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am
     561                configfile: /home/chaos/omni/omni_pgeni
     562                framework: pg
     563                native: True
     564
     565  Args: getversion
     566
     567  Result Summary:
     568Got version for 1 out of 1 aggregates
     569 
     570INFO:omni: ============================================================
     571jericho,[~],11:34(0)$ omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am -V2 -t GENI 3 listresources -o
     572INFO:omni:Loading config file /home/chaos/omni/omni_pgeni
     573INFO:omni:Using control framework pg
     574INFO:omni:Saving output to a file.
     575INFO:omni:Listed resources on 1 out of 1 possible aggregates.
     576INFO:omni:Writing to 'rspec-instageni-gpolab-bbn-com-12369-protogeni.xml'
     577INFO:omni: ------------------------------------------------------------
     578INFO:omni: Completed listresources:
     579
     580  Options as run:
     581                aggregate: https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am
     582                api_version: 2
     583                configfile: /home/chaos/omni/omni_pgeni
     584                framework: pg
     585                native: True
     586                output: True
     587                rspectype: ('GENI', '3')
     588
     589  Args: listresources
     590
     591  Result Summary: Retrieved resources from 1 aggregates.
     592Wrote rspecs from 1 aggregates to 1 files
     593Saved listresources RSpec at 'unspecified_AM_URN' to file rspec-instageni-gpolab-bbn-com-12369-protogeni.xml; .
     594INFO:omni: ============================================================
     595jericho,[~],11:35(0)$ mv rspec-instageni-gpolab-bbn-com-12369-protogeni.xml rspec-instageni-gpolab-bbn-com-12369-protogeni-IG-ADM-6-02.xml
     596jericho,[~],11:35(0)$
     597}}}
     598 * Run omni getversion and listresources against the BBN rack FOAM AM:
     599{{{
     600jericho,[~],11:35(0)$ omni -a https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 getversion
     601INFO:omni:Loading config file /home/chaos/omni/omni_pgeni
     602INFO:omni:Using control framework pg
     603INFO:omni:AM URN: unspecified_AM_URN (url: https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1) has version:
     604INFO:omni:{   'ad_rspec_versions': [   {   'extensions': [   'http://www.geni.net/resources/rspec/ext/openflow/3'],
     605                                 'namespace': 'http://www.geni.net/resources/rspec/3',
     606                                 'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd',
     607                                 'type': 'GENI',
     608                                 'version': '3'}],
     609    'foam_version': '0.8.2',
     610    'geni_api': 1,
     611    'request_rspec_versions': [   {   'extensions': [   'http://www.geni.net/resources/rspec/ext/openflow/3',
     612                                                        'http://www.geni.net/resources/rspec/ext/openflow/4',
     613                                                        'http://www.geni.net/resources/rspec/ext/flowvisor/1'],
     614                                      'namespace': 'http://www.geni.net/resources/rspec/3',
     615                                      'schema': 'http://www.geni.net/resources/rspec/3/request.xsd',
     616                                      'type': 'GENI',
     617                                      'version': '3'}],
     618    'site_info': {   'admin-email': 'instageni-foam-admin@gpolab.bbn.com'}}
     619INFO:omni: ------------------------------------------------------------
     620INFO:omni: Completed getversion:
     621
     622  Options as run:
     623                aggregate: https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1
     624                configfile: /home/chaos/omni/omni_pgeni
     625                framework: pg
     626                native: True
     627
     628  Args: getversion
     629
     630  Result Summary:
     631Got version for 1 out of 1 aggregates
     632 
     633INFO:omni: ============================================================
     634jericho,[~],11:36(0)$ omni -a https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 listresources -o
     635INFO:omni:Loading config file /home/chaos/omni/omni_pgeni
     636INFO:omni:Using control framework pg
     637INFO:omni:Saving output to a file.
     638INFO:omni:Listed resources on 1 out of 1 possible aggregates.
     639INFO:omni:Writing to 'rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1.xml'
     640INFO:omni: ------------------------------------------------------------
     641INFO:omni: Completed listresources:
     642
     643  Options as run:
     644                aggregate: https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1
     645                configfile: /home/chaos/omni/omni_pgeni
     646                framework: pg
     647                native: True
     648                output: True
     649
     650  Args: listresources
     651
     652  Result Summary: Retrieved resources from 1 aggregates.
     653Wrote rspecs from 1 aggregates to 1 files
     654Saved listresources RSpec at 'unspecified_AM_URN' to file rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1.xml; .
     655INFO:omni: ============================================================
     656jericho,[~],11:36(0)$ mv rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1.xml rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1-02.xml
     657jericho,[~],11:36(0)$
     658}}}
     659 * Verify that [http://monitor.gpolab.bbn.com/connectivity/campus.html] currently shows a successful connection from argos to instageni-vlan1750.bbn.dataplane.geni.net: yep, looks fine
     660
     661
     662'''Verify:'''
     663 * Site admins should be able to login to boss, control, and foam, and perform sudo operations
     664 * The Emulab web UI should be available, and red dot mode should work
     665 * Monitoring of the rack should be available (though it may show some errors due to the disconnection)
     666 * Rack ProtoGENI and FOAM aggregates should respond to getversion and listresources
     667 * Existing experiments should continue to run, and should be able to respond to dataplane traffic
     668
    160669=== Step 1C: undo the changes ===
    161670
    … …  
    179688   * If SSH is successful, attempt to sudo on boss:
    180689{{{
    181 sudo -v
     690sudo whoami
    182691}}}
    183692   * If SSH is successful, ping ops.emulab.net to verify connectivity:
    … …  
    191700 * If SSH is successful, attempt to sudo on control:
    192701{{{
    193 sudo -v
     702sudo whoami
    194703}}}
    195704 * Attempt to SSH to foam.instageni.gpolab.bbn.com:
    … …  
    199708 * If SSH is successful, attempt to sudo on foam:
    200709{{{
    201 sudo -v
     710sudo whoami
    202711}}}
    203712 * Browse to [https://instageni.gpolab.bbn.com/] and attempt to login as chaos
    … …  
    207716 * Run omni getversion and listresources against the BBN rack ProtoGENI AM:
    208717{{{
    209 omni -a https://utah.geniracks.net:12369/protogeni/xmlrpc/am getversion
    210 omni -a https://utah.geniracks.net:12369/protogeni/xmlrpc/am listresources -o
     718omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am getversion
     719omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am -V2 -t GENI 3 listresources -o
    211720}}}
    212721 * Run omni getversion and listresources against the BBN rack FOAM AM:
    … …  
    225734 * Existing experiments should continue to run, and should be able to respond to dataplane traffic
    226735
     736==== Results of testing Step 1C on 2013-03-08 ====
     737
     738'''Using:'''
     739 * On maple (lab subnet router), do:
     740{{{
     741conf t
     742  int gi 0/1.830
     743    no ip access-group gst-4544-ig-adm-6 out
     744  exit
     745  no ip access-list standard gst-4544-ig-adm-6
     746exit
     747}}}
     748 * Wait 5 minutes: waiting until 11:44
     749 * Attempt to SSH to boss.instageni.gpolab.bbn.com:
     750{{{
     751capybara,[~],11:43(0)$ ssh boss.instageni.gpolab.bbn.com
     752Last login: Fri Mar  8 11:25:58 2013 from capybara.bbn.com
     753FreeBSD 9.0-STABLE (XEN) #4: Mon Nov 26 10:02:53 MST 2012
     754
     755Welcome to ProtoGeni!
     756
     757Remember to put /usr/testbed/bin and /usr/testbed/sbin in your path!
     758
     759
     760You can automatically download and install binary packages by doing
     761
     762        pkg_add -r <URL>
     763
     764where you replace <URL> with the URL to the package.  This will also
     765automatically install the packages the package you download is dependent on
     766(ie, the packages it needs in order to work.)
     767> sudo whoami
     768root
     769>
     770}}}
     771   * If SSH is successful, ping ops.emulab.net to verify connectivity:
     772{{{
     773> ping ops.emulab.net
     774PING ops.emulab.net (155.98.33.74): 56 data bytes
     77564 bytes from 155.98.33.74: icmp_seq=0 ttl=47 time=67.399 ms
     77664 bytes from 155.98.33.74: icmp_seq=1 ttl=47 time=59.152 ms
     77764 bytes from 155.98.33.74: icmp_seq=2 ttl=47 time=59.708 ms
     778^C
     779--- ops.emulab.net ping statistics ---
     7803 packets transmitted, 3 packets received, 0.0% packet loss
     781round-trip min/avg/max/stddev = 59.152/62.086/67.399/3.763 ms
     782>
     783}}}
     784 * Attempt to SSH to gpolab.control-nodes.geniracks.net:
     785{{{
     786capybara,[~],11:44(0)$ ssh gpolab.control-nodes.geniracks.net
     787Welcome to Ubuntu precise (development branch) (GNU/Linux 3.2.0-23-generic x86_64)
     788
     789 * Documentation:  https://help.ubuntu.com/
     790
     791  System information as of Fri Mar  8 09:44:38 MST 2013
     792
     793  System load:  0.02              Users logged in:       0
     794  Usage of /:   24.7% of 5.85GB   IP address for xenbr0: 192.1.242.130
     795  Memory usage: 42%               IP address for xenbr1: 10.1.1.254
     796  Swap usage:   0%                IP address for xenbr2: 10.2.1.254
     797  Processes:    144               IP address for xenbr3: 10.3.1.254
     798
     799  Graph this data and manage this system at https://landscape.canonical.com/
     800Last login: Fri Mar  8 09:44:30 2013 from capybara.bbn.com
     801gpolab,[~],09:44(0)$ sudo whoami
     802root
     803gpolab,[~],09:44(0)$
     804}}}
     805 * Attempt to SSH to foam.instageni.gpolab.bbn.com:
     806{{{
     807capybara,[~],11:44(0)$ ssh foam.instageni.gpolab.bbn.com
     808Welcome to Ubuntu 12.04.1 LTS (GNU/Linux 3.2.0-34-generic x86_64)
     809
     810 * Documentation:  https://help.ubuntu.com/
     811Last login: Fri Mar  8 11:27:40 2013 from capybara.bbn.com
     812foam,[~],11:44(0)$ sudo whoami
     813root
     814foam,[~],11:45(0)$
     815}}}
     816 * Browse to [https://instageni.gpolab.bbn.com/] and attempt to login as chaos
     817   * If successful, enter red dot mode: yep
     818 * Browse to [http://monitor.gpolab.bbn.com/nagios/cgi-bin/status.cgi?hostgroup=all&style=detail&sorttype=2&sortoption=3] and ensure that GPO nagios is available
     819   * If successful, enumerate any instageni-relevant errors currently outstanding in GPO nagios: `TANGO-foam.instageni.gpolab.bbn.com/gmoc_monitoring` is having problems submitting data, due to an unrelated GMOC maintenance which ran over.
     820 * Run omni getversion and listresources against the BBN rack ProtoGENI AM:
     821{{{
     822jericho,[~],11:36(0)$ omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am getversion
     823INFO:omni:Loading config file /home/chaos/omni/omni_pgeni
     824INFO:omni:Using control framework pg
     825INFO:omni:AM URN: unspecified_AM_URN (url: https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am) has version:
     826INFO:omni:{   'code': {   'am_code': 0,
     827                'am_type': 'protogeni',
     828                'geni_code': 0,
     829                'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+c91268c3fed57bf7807b20802ba3c8b2',
     830                'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=c91268c3fed57bf7807b20802ba3c8b2'},
     831    'geni_api': 2,
     832    'output': '',
     833    'value': {   'code_tag': 'b8883d9be04ae81a2180349b95501e30078abb1f',
     834                 'geni_ad_rspec_versions': [   {   'extensions': [],
     835                                                   'namespace': 'http://www.protogeni.net/resources/rspec/0.1',
     836                                                   'schema': 'http://www.protogeni.net/resources/rspec/0.1/ad.xsd',
     837                                                   'type': 'ProtoGENI',
     838                                                   'version': '0.1'},
     839                                               {   'extensions': [],
     840                                                   'namespace': 'http://www.protogeni.net/resources/rspec/0.2',
     841                                                   'schema': 'http://www.protogeni.net/resources/rspec/0.2/ad.xsd',
     842                                                   'type': 'ProtoGENI',
     843                                                   'version': '0.2'},
     844                                               {   'extensions': [   'http://www.protogeni.net/resources/rspec/ext/emulab/1'],
     845                                                   'namespace': 'http://www.protogeni.net/resources/rspec/2',
     846                                                   'schema': 'http://www.protogeni.net/resources/rspec/2/ad.xsd',
     847                                                   'type': 'ProtoGENI',
     848                                                   'version': '2'},
     849                                               {   'extensions': [   'http://www.protogeni.net/resources/rspec/ext/emulab/1'],
     850                                                   'namespace': 'http://www.geni.net/resources/rspec/3',
     851                                                   'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd',
     852                                                   'type': 'GENI',
     853                                                   'version': '3'}],
     854                 'geni_api': 2,
     855                 'geni_api_versions': {   '1': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/1.0',
     856                                          '2': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0',
     857                                          '3': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/3.0'},
     858                 'geni_request_rspec_versions': [   {   'extensions': [],
     859                                                        'namespace': 'http://www.protogeni.net/resources/rspec/0.1',
     860                                                        'schema': 'http://www.protogeni.net/resources/rspec/0.1/request.xsd',
     861                                                        'type': 'ProtoGENI',
     862                                                        'version': '0.1'},
     863                                                    {   'extensions': [],
     864                                                        'namespace': 'http://www.protogeni.net/resources/rspec/0.2',
     865                                                        'schema': 'http://www.protogeni.net/resources/rspec/0.2/request.xsd',
     866                                                        'type': 'ProtoGENI',
     867                                                        'version': '0.2'},
     868                                                    {   'extensions': [   'http://www.protogeni.net/resources/rspec/ext/emulab/1'],
     869                                                        'namespace': 'http://www.protogeni.net/resources/rspec/2',
     870                                                        'schema': 'http://www.protogeni.net/resources/rspec/2/request.xsd',
     871                                                        'type': 'ProtoGENI',
     872                                                        'version': '2'},
     873                                                    {   'extensions': [   'http://www.protogeni.net/resources/rspec/ext/emulab/1'],
     874                                                        'namespace': 'http://www.geni.net/resources/rspec/3',
     875                                                        'schema': 'http://www.geni.net/resources/rspec/3/request.xsd',
     876                                                        'type': 'GENI',
     877                                                        'version': '3'}],
     878                 'hostname': 'boss.instageni.gpolab.bbn.com',
     879                 'hrn': 'bbninstageni.cm',
     880                 'interface': 'aggregate',
     881                 'url': 'https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am',
     882                 'urn': 'urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm'}}
     883INFO:omni: ------------------------------------------------------------
     884INFO:omni: Completed getversion:
     885
     886  Options as run:
     887                aggregate: https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am
     888                configfile: /home/chaos/omni/omni_pgeni
     889                framework: pg
     890                native: True
     891
     892  Args: getversion
     893
     894  Result Summary:
     895Got version for 1 out of 1 aggregates
     896 
     897INFO:omni: ============================================================
     898jericho,[~],11:49(0)$ omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am -V2 -t GENI 3 listresources -o
     899INFO:omni:Loading config file /home/chaos/omni/omni_pgeni
     900INFO:omni:Using control framework pg
     901INFO:omni:Saving output to a file.
     902INFO:omni:Listed resources on 1 out of 1 possible aggregates.
     903INFO:omni:Writing to 'rspec-instageni-gpolab-bbn-com-12369-protogeni.xml'
     904INFO:omni: ------------------------------------------------------------
     905INFO:omni: Completed listresources:
     906
     907  Options as run:
     908                aggregate: https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am
     909                api_version: 2
     910                configfile: /home/chaos/omni/omni_pgeni
     911                framework: pg
     912                native: True
     913                output: True
     914                rspectype: ('GENI', '3')
     915
     916  Args: listresources
     917
     918  Result Summary: Retrieved resources from 1 aggregates.
     919Wrote rspecs from 1 aggregates to 1 files
     920Saved listresources RSpec at 'unspecified_AM_URN' to file rspec-instageni-gpolab-bbn-com-12369-protogeni.xml; .
     921INFO:omni: ============================================================
     922jericho,[~],11:49(0)$ mv rspec-instageni-gpolab-bbn-com-12369-protogeni.xml rspec-instageni-gpolab-bbn-com-12369-protogeni-IG-ADM-6-03.xml
     923jericho,[~],11:50(0)$
     924}}}
     925 * Run omni getversion and listresources against the BBN rack FOAM AM:
     926{{{
     927jericho,[~],11:50(0)$ omni -a https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 getversion
     928INFO:omni:Loading config file /home/chaos/omni/omni_pgeni
     929INFO:omni:Using control framework pg
     930INFO:omni:AM URN: unspecified_AM_URN (url: https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1) has version:
     931INFO:omni:{   'ad_rspec_versions': [   {   'extensions': [   'http://www.geni.net/resources/rspec/ext/openflow/3'],
     932                                 'namespace': 'http://www.geni.net/resources/rspec/3',
     933                                 'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd',
     934                                 'type': 'GENI',
     935                                 'version': '3'}],
     936    'foam_version': '0.8.2',
     937    'geni_api': 1,
     938    'request_rspec_versions': [   {   'extensions': [   'http://www.geni.net/resources/rspec/ext/openflow/3',
     939                                                        'http://www.geni.net/resources/rspec/ext/openflow/4',
     940                                                        'http://www.geni.net/resources/rspec/ext/flowvisor/1'],
     941                                      'namespace': 'http://www.geni.net/resources/rspec/3',
     942                                      'schema': 'http://www.geni.net/resources/rspec/3/request.xsd',
     943                                      'type': 'GENI',
     944                                      'version': '3'}],
     945    'site_info': {   'admin-email': 'instageni-foam-admin@gpolab.bbn.com'}}
     946INFO:omni: ------------------------------------------------------------
     947INFO:omni: Completed getversion:
     948
     949  Options as run:
     950                aggregate: https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1
     951                configfile: /home/chaos/omni/omni_pgeni
     952                framework: pg
     953                native: True
     954
     955  Args: getversion
     956
     957  Result Summary:
     958Got version for 1 out of 1 aggregates
     959 
     960INFO:omni: ============================================================
     961jericho,[~],11:50(0)$ omni -a https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 listresources -o
     962INFO:omni:Loading config file /home/chaos/omni/omni_pgeni
     963INFO:omni:Using control framework pg
     964INFO:omni:Saving output to a file.
     965INFO:omni:Listed resources on 1 out of 1 possible aggregates.
     966INFO:omni:Writing to 'rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1.xml'
     967INFO:omni: ------------------------------------------------------------
     968INFO:omni: Completed listresources:
     969
     970  Options as run:
     971                aggregate: https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1
     972                configfile: /home/chaos/omni/omni_pgeni
     973                framework: pg
     974                native: True
     975                output: True
     976
     977  Args: listresources
     978
     979  Result Summary: Retrieved resources from 1 aggregates.
     980Wrote rspecs from 1 aggregates to 1 files
     981Saved listresources RSpec at 'unspecified_AM_URN' to file rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1.xml; .
     982INFO:omni: ============================================================
     983jericho,[~],11:50(0)$ mv rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1.xml rspec-foam-instageni-gpolab-bbn-com-3626-foam-gapi-1-IG-ADM-6-03.xml
     984jericho,[~],11:50(0)$
     985}}}
     986 * Verify that [http://monitor.gpolab.bbn.com/connectivity/campus.html] currently shows a successful connection from argos to instageni-vlan1750.bbn.dataplane.geni.net: yep
     987
     988'''Verify:'''
     989 * Site admins should be able to login to boss, control, and foam, and perform sudo operations
     990 * The Emulab web UI should be available, and red dot mode should work
     991 * Monitoring of the rack should be available (though it may show some errors due to the disconnection)
     992 * Rack ProtoGENI and FOAM aggregates should respond to getversion and listresources
     993 * Existing experiments should continue to run, and should be able to respond to dataplane traffic
     994