Changes between Version 61 and Version 62 of GENIRacksHome/ExogeniRacks/AcceptanceTestStatus/EG-ADM-1


Ignore:
Timestamp:
02/19/13 11:37:22 (11 years ago)
Author:
Josh Smift
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • GENIRacksHome/ExogeniRacks/AcceptanceTestStatus/EG-ADM-1

    v61 v62  
    55''This page is GPO's working page for performing EG-ADM-1.  It is public for informational purposes, but it is not an official status report.  See [wiki:GENIRacksHome/ExogeniRacks/AcceptanceTestStatus] for the current status of ExoGENI acceptance tests.''
    66
    7 ''Last substantive edit of this page: 2012-10-10''
     7''Last substantive edit of this page: 2013-02-19''
    88
    99== Page format ==
    … …  
    1212 * The high-level description from test plan contains text copied exactly from the public test plan and acceptance criteria pages.
    1313 * The steps contain things i will actually do/verify:
    14    * Steps may be composed of related substeps where i find this useful for clarity 
     14   * Steps may be composed of related substeps where i find this useful for clarity
    1515   * Each step is identified as either "(prep)" or "(verify)":
    1616     * Prep steps are just things we have to do.  They're not tests of the rack, but are prerequisites for subsequent verification steps
    … …  
    1919== Status of test ==
    2020
    21 || '''Step''' || '''State'''                || '''Date completed''' || '''Open Tickets''' || '''Closed Tickets/Comments'''                     ||
    22 || 1          || [[Color(green,Pass)]]      || 2012-02-24           ||                    ||                                                   ||
    23 || 2A         || [[Color(green,Pass)]]      || 2012-10-10           ||                    || ([exoticket:11])                                  ||
    24 || 2B         || [[Color(green,Pass)]]      || 2012-10-10           ||                    ||                                                   ||
    25 || 2C         || [[Color(green,Pass)]]      || 2012-10-10           ||                    ||                                                   ||
    26 || 3A         || [[Color(green,Pass)]]      || 2012-05-10           ||                    ||                                                   ||
     21|| '''Step''' || '''State'''                || '''Date completed''' || '''Open Tickets''' || '''Closed Tickets/Comments''' ||
     22|| 1          || [[Color(green,Pass)]]      || 2012-02-24           ||                    || ||
     23|| 2A         || [[Color(green,Pass)]]      || 2012-10-10           ||                    || ([exoticket:11]) ||
     24|| 2B         || [[Color(green,Pass)]]      || 2012-10-10           ||                    || ||
     25|| 2C         || [[Color(green,Pass)]]      || 2012-10-10           ||                    || ||
     26|| 3A         || [[Color(green,Pass)]]      || 2012-05-10           ||                    || ||
    2727|| 3B         || [[Color(green,Pass)]]      || 2012-05-10           ||                    || ([exoticket:10], [exoticket:20], [exoticket:32])  ||
    28 || 3C         || [[Color(green,Pass)]]      || 2012-05-10           ||                    ||                                                                                                                    ||
    29 || 3D         || [[Color(green,Pass)]]      || 2012-05-11           ||                    ||                                                                                                                    ||
    30 || 3E         || [[Color(green,Pass)]]      || 2012-07-05           ||                    ||                                                                                                                    ||
    31 || 4A         || [[Color(green,Pass)]]      || 2012-10-10           ||                    || ([exoticket:22], [exoticket:33]) clarify some outstanding DNS questions                                            ||
    32 || 4B         || [[Color(green,Pass)]]      || 2012-07-05           ||                    || ([exoticket:23])                                                                                                   ||
    33 || 4C         || [[Color(green,Pass)]]      || 2012-10-10           ||                    || ([exoticket:71])                                                                                                   ||
    34 || 4D         || [[Color(green,Pass)]]      || 2012-06-21           ||                    || ([exoticket:12])                                                                                                   ||
    35 || 5A         || [[Color(green,Pass)]]      || 2012-05-23           ||                    ||                                                                                                                    ||
     28|| 3C         || [[Color(green,Pass)]]      || 2012-05-10           ||                    || ||
     29|| 3D         || [[Color(green,Pass)]]      || 2012-05-11           ||                    || ||
     30|| 3E         || [[Color(green,Pass)]]      || 2012-07-05           ||                    || ||
     31|| 4A         || [[Color(green,Pass)]]      || 2012-10-10           ||                    || ([exoticket:22], [exoticket:33]) clarify some outstanding DNS questions ||
     32|| 4B         || [[Color(green,Pass)]]      || 2012-07-05           ||                    || ([exoticket:23]) ||
     33|| 4C         || [[Color(green,Pass)]]      || 2012-10-10           ||                    || ([exoticket:71]) ||
     34|| 4D         || [[Color(green,Pass)]]      || 2012-06-21           ||                    || ([exoticket:12]) ||
     35|| 5A         || [[Color(green,Pass)]]      || 2012-05-23           ||                    || ||
    3636|| 5B         || [[Color(yellow,Complete)]] ||                      ||                    || ([exoticket:28]) This test was completed once, but will re-run once hybrid mode is available on the dataplane switch ||
    37 || 5C         || [[Color(green,Pass)]]      || 2012-07-23           ||                    || ([exoticket:29])                                                                                                   ||
    38 || 5D         || [[Color(green,Pass)]]      || 2012-07-23           ||                    || ([exoticket:30])                                                                                                   ||
    39 || 6A         || [[Color(green,Pass)]]      || 2012-05-23           ||                    ||                                                                                                                    ||
    40 || 6B         || [[Color(green,Pass)]]      || 2012-05-23           ||                    ||                                                                                                                    ||
    41 || 6C         || [[Color(yellow,Complete)]] ||                      ||                    || Interim notifications during the test period have been agreed on; revisit longer-term plan for notifications later ||
     37|| 5C         || [[Color(green,Pass)]]      || 2012-07-23           ||                    || ([exoticket:29]) ||
     38|| 5D         || [[Color(green,Pass)]]      || 2012-07-23           ||                    || ([exoticket:30]) ||
     39|| 6A         || [[Color(green,Pass)]]      || 2012-05-23           ||                    || ||
     40|| 6B         || [[Color(green,Pass)]]      || 2012-05-23           ||                    || ||
     41|| 6C         || [[Color(green,Pass)]]      || 2013-02-14           ||                    || ||
    4242
    4343== High-level description from test plan ==
    … …  
    141141...
    142142bbn-hn,[~],14:06(0)$ sudo whoami
    143 [sudo] password for chaos: 
     143[sudo] password for chaos:
    144144root
    145 bbn-hn,[~],14:07(0)$ 
     145bbn-hn,[~],14:07(0)$
    146146}}}
    147147 * Josh reported successful public-key login and sudo from a BBN subnet (128.89.91.0/24)
    … …  
    171171{{{
    172172bbn-hn,[~],14:23(1)$ ssh 192.168.103.2
    173 Enter radius password: 
     173Enter radius password:
    174174
    175175IBM Networking Operating System RackSwitch G8052.
    … …  
    185185switch-type "IBM Networking Operating System RackSwitch G8052"
    186186...
    187 8052.bbn.xo#show mac-address-table 
    188 Mac address Aging Time: 300 
     1878052.bbn.xo#show mac-address-table
     188Mac address Aging Time: 300
    189189...
    1901908052.bbn.xo#exitReceived disconnect from 192.168.103.2: 11: Logged out.
    … …  
    193193{{{
    194194bbn-hn,[~],14:28(1)$ ssh 192.168.103.4
    195 Enter radius password: 
     195Enter radius password:
    196196
    197197IBM Networking Operating System RackSwitch G8264.
    … …  
    208208...
    2092098264.bbn.xo#show mac-address-table
    210 Mac address Aging Time: 300 
     210Mac address Aging Time: 300
    211211
    212212FDB is empty.
    … …  
    218218{{{
    219219capybara,[~/src/cvs/geni-inf/GENI-CVS.BBN.COM/puppet],10:14(0)$ ssh 192.1.242.4
    220 Enter radius password: 
     220Enter radius password:
    221221
    222222IBM Networking Operating System RackSwitch G8052.
    … …  
    231231{{{
    232232[tupty@bbn-hn ~]$ ssh 192.168.103.2
    233 Enter radius password: 
     233Enter radius password:
    234234Received disconnect from 192.168.103.2: 11: Logged out.
    235235}}}
    236236
    237 In summary, all of the access works for me because i am in `xoadmins`, but Tim is not able to login because `bbnadmins` does not have access. 
     237In summary, all of the access works for me because i am in `xoadmins`, but Tim is not able to login because `bbnadmins` does not have access.
    238238
    239239==== Results of testing step 3B: 2012-05-26 ====
    240240
    241 Testing assertion that exoticket:20 has been resolved, so my site admin account, `cgolubit`, should be able to run this test. 
     241Testing assertion that exoticket:20 has been resolved, so my site admin account, `cgolubit`, should be able to run this test.
    242242
    243243 * Per e-mail from Chris, the 8052 is 192.168.103.2, and the 8264 is 192.168.103.4.  The 8052 also has the public IP address 192.1.242.4.
    … …  
    251251Are you sure you want to continue connecting (yes/no)? yes
    252252Warning: Permanently added '192.168.103.2' (DSA) to the list of known hosts.
    253 Enter radius password: 
     253Enter radius password:
    254254
    255255IBM Networking Operating System RackSwitch G8052.
    … …  
    275275{{{
    2762768052.bbn.xo>show mac-address-table
    277 Mac address Aging Time: 300 
     277Mac address Aging Time: 300
    278278
    279279Total number of FDB entries : 26
    … …  
    295295Are you sure you want to continue connecting (yes/no)? yes
    296296Warning: Permanently added '192.168.103.4' (DSA) to the list of known hosts.
    297 Enter radius password: 
     297Enter radius password:
    298298
    299299IBM Networking Operating System RackSwitch G8264.
    … …  
    317317 * Mac address table (which is empty here) can be viewed:
    318318{{{
    319 8264.bbn.xo>show mac-address-table 
    320 Mac address Aging Time: 300 
     3198264.bbn.xo>show mac-address-table
     320Mac address Aging Time: 300
    321321
    322322FDB is empty.
    … …  
    324324 * Openflow informaton can be viewed, including DPID and controllers for an active instance:
    325325{{{
    326 8264.bbn.xo>show openflow 1         
     3268264.bbn.xo>show openflow 1
    327327Open Flow Instance ID: 1
    328328        DataPath ID: 0x640817f4b52a00
    329329...
    330 Configured Controllers: 
     330Configured Controllers:
    331331        IP Address: 192.168.103.10
    332332                State: Active
    … …  
    340340{{{
    341341capybara,[~],09:35(255)$ ssh cgolubit@192.1.242.4
    342 Enter radius password: 
     342Enter radius password:
    343343
    344344IBM Networking Operating System RackSwitch G8052.
    … …  
    411411...
    412412bbn-w1,[~],16:19(0)$ sudo whoami
    413 [sudo] password for chaos: 
     413[sudo] password for chaos:
    414414root
    415415}}}
    … …  
    421421...
    422422bbn-w2,[~],16:25(0)$ sudo whoami
    423 [sudo] password for chaos: 
     423[sudo] password for chaos:
    424424root
    425425}}}
    … …  
    430430...
    431431bbn-w3,[~],16:26(0)$ sudo whoami
    432 [sudo] password for chaos: 
     432[sudo] password for chaos:
    433433root
    434434}}}
    … …  
    485485Name: 192.168.100.2
    486486Address: 192.168.100.2#53
    487 Aliases: 
     487Aliases:
    488488
    489489bbn-w1.bbn.xo has address 192.168.103.101
    … …  
    521521 * Logout
    522522 * Now browse to `http://bbn-w2.bbn.xo`:
    523    * Login as before 
     523   * Login as before
    524524   * Tasks -> Remote Control -> Start Remote Control in Multi-User Mode
    525525   * IMM Control -> Configuration File -> view the current configuration summary, and make a copy
    526526 * Now browse to `http://bbn-w3.bbn.xo`:
    527    * Login as before 
     527   * Login as before
    528528   * Tasks -> Remote Control -> Start Remote Control in Multi-User Mode
    529529   * IMM Control -> Configuration File -> view the current configuration summary, and make a copy
    530530 * Now browse to `http://bbn-w4.bbn.xo`:
    531    * Login as before 
     531   * Login as before
    532532   * Tasks -> Remote Control -> Start Remote Control in Multi-User Mode
    533533     * the console here shows that bbn-w4 is at a PXE boot prompt
    … …  
    564564}}}
    565565 * Now browse to `http://bbn-w4.bbn.xo`:
    566    * Login as before 
     566   * Login as before
    567567   * IMM Control -> Configuration File -> view the current configuration summary, and make a copy
    568568     * This time, the configuration eventually loaded with no trouble
    569569 * Now browse to `http://bbn-hn.bbn.xo`:
    570    * Login as before 
     570   * Login as before
    571571   * Tasks -> Remote Control -> Start Remote Control in Multi-User Mode
    572572   * IMM Control -> Configuration File -> view the current configuration summary, and make a copy
    … …  
    607607{{{
    608608bbn-hn,[~],16:55(0)$ ssh bbn-w4
    609 chaos@bbn-w4's password: 
     609chaos@bbn-w4's password:
    610610Permission denied, please try again.
    611611
    612612bbn-hn,[~],16:55(130)$ ssh cgolubit@bbn-w4
    613 cgolubit@bbn-w4's password: 
     613cgolubit@bbn-w4's password:
    614614Permission denied, please try again.
    615615}}}
    … …  
    618618{{{
    619619bbn-hn,[~],17:31(0)$ sudo ssh -i /opt/orca-12080/xcat/id_rsa root@bbn-w4
    620 [sudo] password for chaos: 
     620[sudo] password for chaos:
    621621Last login: Thu Jul  5 17:30:40 2012 from 10.100.0.1
    622 [root@bbn-w4 ~]# 
    623 }}}
    624 
    625 == Step 4: GPO inventories the rack based on our own processes == 
     622[root@bbn-w4 ~]#
     623}}}
     624
     625== Step 4: GPO inventories the rack based on our own processes ==
    626626
    627627=== Step 4A: Inventory and label physical rack contents ===
    … …  
    633633 * Use [https://wiki.exogeni.net/doku.php?id=public:hardware:rack_layout] to determine the name of each object
    634634 * If any objects can't be found there, compare to [gsw:ChaosSandbox/ExogeniRackNotes], and iterate with RENCI
    635  * Physically label each device in the rack with its name on front and back 
     635 * Physically label each device in the rack with its name on front and back
    636636 * Inventory all hardware details for rack contents on gsw:OpsHardwareInventory
    637637 * Add an ascii rack diagram to gsw:OpsHardwareInventory
    … …  
    844844 * The tactical overview in the top left lists 5 problems.
    845845   * Clicking on that number takes me to [https://bbn-hn.exogeni.net/rack_bbn/check_mk/view.py?view_name=svcproblems]
    846    * The first problem, with state CRIT, is [https://bbn-hn.exogeni.net/rack_bbn/check_mk/view.py?view_name=service&site=&service=Multipath%20360080e50002d03ac000002cc4f69a431&host=bbn-hn.exogeni.net], a check on the multipath device on bbn-hn. 
     846   * The first problem, with state CRIT, is [https://bbn-hn.exogeni.net/rack_bbn/check_mk/view.py?view_name=service&site=&service=Multipath%20360080e50002d03ac000002cc4f69a431&host=bbn-hn.exogeni.net], a check on the multipath device on bbn-hn.
    847847   * I am not able to figure out, either from that page, or from poking around on bbn-hn, what the command is which is being run to get that result.  I will follow up via e-mail and ask.
    848848 * I also notice that my login reports me as `cgolubit (guest)`, while my chaos account lists me as `chaos (admin)`.  Josh reports that he shows up as `jbs (admin)`.
    … …  
    853853     * When Josh tried to login, he gets the error:
    854854{{{
    855 Your username (jbs) is listed more than once in multisite.mk. 
     855Your username (jbs) is listed more than once in multisite.mk.
    856856This is not allowed. Please check your config.
    857857}}}
    … …  
    860860==== Results of testing step 5C: 2012-05-26 ====
    861861
    862 I'm tracking testing of the RCI login problem, which is still giving a weird error, on exoticket:29. 
     862I'm tracking testing of the RCI login problem, which is still giving a weird error, on exoticket:29.
    863863
    864864Returning to:
    … …  
    876876Date: Sat, 26 May 2012 11:09:48 -0400
    877877To: exogeni-design@geni.net
    878 Subject: Re: [exogeni-design] question about nagios/omd plugins                 
     878Subject: Re: [exogeni-design] question about nagios/omd plugins
    879879}}}
    880880
    … …  
    884884{{{
    885885bbn-hn,[~],18:57(0)$ sudo su - rack_bbn
    886 OMD[rack_bbn]:~$ 
     886OMD[rack_bbn]:~$
    887887}}}
    888888 * As rack_bbn, i can list the check_mk information about bbn-hn.exogeni.net:
    … …  
    890890OMD[rack_bbn]:~$ cmk -D bbn-hn.exogeni.net | head
    891891
    892 bbn-hn.exogeni.net (192.1.242.3)                                               
     892bbn-hn.exogeni.net (192.1.242.3)
    893893Tags:                   tcp, linux, nagios, hn
    894894Host groups:            linux, hn
    … …  
    897897Is aggregated:          no
    898898...
    899   multipath       360080e50002d03ac000002cc4f69a431 2                                                                                                                                                                                                                            Multipath 360080e50002d03ac000002cc4f69a431                             
     899  multipath       360080e50002d03ac000002cc4f69a431 2                                                                                                                                                                                                                            Multipath 360080e50002d03ac000002cc4f69a431
    900900...
    901901}}}
    … …  
    945945- State:       CRITICAL
    946946- Date:        2012-04-26 03:22:08
    947 - Output:      CRIT - (mpathb) paths expected: 4, paths active: 2               
     947- Output:      CRIT - (mpathb) paths expected: 4, paths active: 2
    948948-
    949949----------------------------------
    … …  
    980980Date: Wed, 23 May 2012 16:29:52 +0000
    981981To: chaos@bbn.com
    982 Subject: *** PROBLEM *** 8052.bbn.xo / Interface Ethernet40 is CRITICAL         
     982Subject: *** PROBLEM *** 8052.bbn.xo / Interface Ethernet40 is CRITICAL
    983983
    984984--SERVICE-ALERT-------------------
    … …  
    999999Date: Wed, 23 May 2012 16:30:52 +0000
    10001000To: chaos@bbn.com
    1001 Subject: *** PROBLEM *** 8052.bbn.xo / Interface Ethernet30 is CRITICAL         
     1001Subject: *** PROBLEM *** 8052.bbn.xo / Interface Ethernet30 is CRITICAL
    10021002
    10031003--SERVICE-ALERT-------------------
    … …  
    10181018Date: Wed, 23 May 2012 16:31:52 +0000
    10191019To: chaos@bbn.com
    1020 Subject: *** RECOVERY *** 8052.bbn.xo / Interface Ethernet40 is OK             
     1020Subject: *** RECOVERY *** 8052.bbn.xo / Interface Ethernet40 is OK
    10211021
    10221022--SERVICE-ALERT-------------------
    … …  
    10261026- Service:     Interface Ethernet40
    10271027- - - - - - - - - - - - - - - - -
    1028 - State:       OK     
     1028- State:       OK
    10291029- Date:        2012-05-23 16:31:52
    1030 - Output:      OK - [168] (up) 1GBit/s, in: 29.71B/s(0.0%), out: 73.21B/s(0.0%) 
     1030- Output:      OK - [168] (up) 1GBit/s, in: 29.71B/s(0.0%), out: 73.21B/s(0.0%)
    10311031-
    10321032----------------------------------
    … …  
    10371037Date: Wed, 23 May 2012 16:31:52 +0000
    10381038To: chaos@bbn.com
    1039 Subject: *** RECOVERY *** 8052.bbn.xo / Interface Ethernet30 is OK             
     1039Subject: *** RECOVERY *** 8052.bbn.xo / Interface Ethernet30 is OK
    10401040
    10411041--SERVICE-ALERT-------------------
    … …  
    10451045- Service:     Interface Ethernet30
    10461046- - - - - - - - - - - - - - - - -
    1047 - State:       OK     
     1047- State:       OK
    10481048- Date:        2012-05-23 16:31:52
    1049 - Output:      OK - [158] (up) 1GBit/s, in: 0.00B/s(0.0%), out: 30.94B/s(0.0%) 
     1049- Output:      OK - [158] (up) 1GBit/s, in: 0.00B/s(0.0%), out: 30.94B/s(0.0%)
    10501050-
    10511051----------------------------------
    … …  
    10601060Date: Wed, 23 May 2012 17:11:32 +0000
    10611061To: chaos@bbn.com
    1062 Subject: *** PROBLEM *** bbn-w1.local / Check_MK inventory is WARNING           
     1062Subject: *** PROBLEM *** bbn-w1.local / Check_MK inventory is WARNING
    10631063
    10641064--SERVICE-ALERT-------------------
    … …  
    10701070- State:       WARNING
    10711071- Date:        2012-05-23 17:11:32
    1072 - Output:      WARNING - 3 unchecked services (lnx_if:2, qemu:1)               
     1072- Output:      WARNING - 3 unchecked services (lnx_if:2, qemu:1)
    10731073-              lnx_if: Interface vnet2
    10741074lnx_if: Interface vnet3
    … …  
    10821082Date: Wed, 23 May 2012 17:13:32 +0000
    10831083To: chaos@bbn.com
    1084 Subject: *** RECOVERY *** bbn-w1.local / Check_MK inventory is OK               
     1084Subject: *** RECOVERY *** bbn-w1.local / Check_MK inventory is OK
    10851085
    10861086--SERVICE-ALERT-------------------
    … …  
    10901090- Service:     Check_MK inventory
    10911091- - - - - - - - - - - - - - - - -
    1092 - State:       OK     
     1092- State:       OK
    10931093- Date:        2012-05-23 17:13:32
    10941094- Output:      OK - no unchecked services found
    … …  
    11691169
    11701170We will want to revisit this test when GMOC has workflows in place to handle notifications for rack outages, and before there are additional rack sites and users who may need to be notified.
     1171
     1172==== Results of testing step 6C: 2013-02-14 ====
     1173
     1174Eldar confirmed via e-mail that the long-term plan is set and working:
     1175
     1176{{{
     1177From: "Urumbaev, Eldar" <eurumbae@indiana.edu>
     1178To: Josh Smift <jbs@bbn.com>, "exogeni-design@geni.net"
     1179        <exogeni-design@geni.net>
     1180Subject: Re: [exogeni-design] Outage reporting to GMOC
     1181Date: Thu, 14 Feb 2013 13:33:20 +0000
     1182
     1183Hi Josh,
     1184
     1185We are all synched up. GMOC is subscribed to the [GENI-ORCA-USERS]
     1186geni-orca-users@googlegroups.com mailing list. The ExoGENI team has been
     1187putting [OUTAGE] or [MAINTENANCE] in subject line to help identify events
     1188that are relevant to us and require our action for tracking rack
     1189outages/maintenances. This has been working ok so far.
     1190
     1191Thanks,
     1192
     1193Eldar
     1194}}}
     1195
     1196So, this is all set.