Changes between Version 61 and Version 62 of GENIRacksHome/ExogeniRacks/AcceptanceTestStatus/EG-ADM-1
- Timestamp:
- 02/19/13 11:37:22 (11 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
GENIRacksHome/ExogeniRacks/AcceptanceTestStatus/EG-ADM-1
v61 v62 5 5 ''This page is GPO's working page for performing EG-ADM-1. It is public for informational purposes, but it is not an official status report. See [wiki:GENIRacksHome/ExogeniRacks/AcceptanceTestStatus] for the current status of ExoGENI acceptance tests.'' 6 6 7 ''Last substantive edit of this page: 201 2-10-10''7 ''Last substantive edit of this page: 2013-02-19'' 8 8 9 9 == Page format == … … 12 12 * The high-level description from test plan contains text copied exactly from the public test plan and acceptance criteria pages. 13 13 * The steps contain things i will actually do/verify: 14 * Steps may be composed of related substeps where i find this useful for clarity 14 * Steps may be composed of related substeps where i find this useful for clarity 15 15 * Each step is identified as either "(prep)" or "(verify)": 16 16 * Prep steps are just things we have to do. They're not tests of the rack, but are prerequisites for subsequent verification steps … … 19 19 == Status of test == 20 20 21 || '''Step''' || '''State''' || '''Date completed''' || '''Open Tickets''' || '''Closed Tickets/Comments''' 22 || 1 || [[Color(green,Pass)]] || 2012-02-24 || || 23 || 2A || [[Color(green,Pass)]] || 2012-10-10 || || ([exoticket:11]) 24 || 2B || [[Color(green,Pass)]] || 2012-10-10 || || 25 || 2C || [[Color(green,Pass)]] || 2012-10-10 || || 26 || 3A || [[Color(green,Pass)]] || 2012-05-10 || || 21 || '''Step''' || '''State''' || '''Date completed''' || '''Open Tickets''' || '''Closed Tickets/Comments''' || 22 || 1 || [[Color(green,Pass)]] || 2012-02-24 || || || 23 || 2A || [[Color(green,Pass)]] || 2012-10-10 || || ([exoticket:11]) || 24 || 2B || [[Color(green,Pass)]] || 2012-10-10 || || || 25 || 2C || [[Color(green,Pass)]] || 2012-10-10 || || || 26 || 3A || [[Color(green,Pass)]] || 2012-05-10 || || || 27 27 || 3B || [[Color(green,Pass)]] || 2012-05-10 || || ([exoticket:10], [exoticket:20], [exoticket:32]) || 28 || 3C || [[Color(green,Pass)]] || 2012-05-10 || || 29 || 3D || [[Color(green,Pass)]] || 2012-05-11 || || 30 || 3E || [[Color(green,Pass)]] || 2012-07-05 || || 31 || 4A || [[Color(green,Pass)]] || 2012-10-10 || || ([exoticket:22], [exoticket:33]) clarify some outstanding DNS questions 32 || 4B || [[Color(green,Pass)]] || 2012-07-05 || || ([exoticket:23]) 33 || 4C || [[Color(green,Pass)]] || 2012-10-10 || || ([exoticket:71]) 34 || 4D || [[Color(green,Pass)]] || 2012-06-21 || || ([exoticket:12]) 35 || 5A || [[Color(green,Pass)]] || 2012-05-23 || || 28 || 3C || [[Color(green,Pass)]] || 2012-05-10 || || || 29 || 3D || [[Color(green,Pass)]] || 2012-05-11 || || || 30 || 3E || [[Color(green,Pass)]] || 2012-07-05 || || || 31 || 4A || [[Color(green,Pass)]] || 2012-10-10 || || ([exoticket:22], [exoticket:33]) clarify some outstanding DNS questions || 32 || 4B || [[Color(green,Pass)]] || 2012-07-05 || || ([exoticket:23]) || 33 || 4C || [[Color(green,Pass)]] || 2012-10-10 || || ([exoticket:71]) || 34 || 4D || [[Color(green,Pass)]] || 2012-06-21 || || ([exoticket:12]) || 35 || 5A || [[Color(green,Pass)]] || 2012-05-23 || || || 36 36 || 5B || [[Color(yellow,Complete)]] || || || ([exoticket:28]) This test was completed once, but will re-run once hybrid mode is available on the dataplane switch || 37 || 5C || [[Color(green,Pass)]] || 2012-07-23 || || ([exoticket:29]) 38 || 5D || [[Color(green,Pass)]] || 2012-07-23 || || ([exoticket:30]) 39 || 6A || [[Color(green,Pass)]] || 2012-05-23 || || 40 || 6B || [[Color(green,Pass)]] || 2012-05-23 || || 41 || 6C || [[Color( yellow,Complete)]] || || || Interim notifications during the test period have been agreed on; revisit longer-term plan for notifications later||37 || 5C || [[Color(green,Pass)]] || 2012-07-23 || || ([exoticket:29]) || 38 || 5D || [[Color(green,Pass)]] || 2012-07-23 || || ([exoticket:30]) || 39 || 6A || [[Color(green,Pass)]] || 2012-05-23 || || || 40 || 6B || [[Color(green,Pass)]] || 2012-05-23 || || || 41 || 6C || [[Color(green,Pass)]] || 2013-02-14 || || || 42 42 43 43 == High-level description from test plan == … … 141 141 ... 142 142 bbn-hn,[~],14:06(0)$ sudo whoami 143 [sudo] password for chaos: 143 [sudo] password for chaos: 144 144 root 145 bbn-hn,[~],14:07(0)$ 145 bbn-hn,[~],14:07(0)$ 146 146 }}} 147 147 * Josh reported successful public-key login and sudo from a BBN subnet (128.89.91.0/24) … … 171 171 {{{ 172 172 bbn-hn,[~],14:23(1)$ ssh 192.168.103.2 173 Enter radius password: 173 Enter radius password: 174 174 175 175 IBM Networking Operating System RackSwitch G8052. … … 185 185 switch-type "IBM Networking Operating System RackSwitch G8052" 186 186 ... 187 8052.bbn.xo#show mac-address-table 188 Mac address Aging Time: 300 187 8052.bbn.xo#show mac-address-table 188 Mac address Aging Time: 300 189 189 ... 190 190 8052.bbn.xo#exitReceived disconnect from 192.168.103.2: 11: Logged out. … … 193 193 {{{ 194 194 bbn-hn,[~],14:28(1)$ ssh 192.168.103.4 195 Enter radius password: 195 Enter radius password: 196 196 197 197 IBM Networking Operating System RackSwitch G8264. … … 208 208 ... 209 209 8264.bbn.xo#show mac-address-table 210 Mac address Aging Time: 300 210 Mac address Aging Time: 300 211 211 212 212 FDB is empty. … … 218 218 {{{ 219 219 capybara,[~/src/cvs/geni-inf/GENI-CVS.BBN.COM/puppet],10:14(0)$ ssh 192.1.242.4 220 Enter radius password: 220 Enter radius password: 221 221 222 222 IBM Networking Operating System RackSwitch G8052. … … 231 231 {{{ 232 232 [tupty@bbn-hn ~]$ ssh 192.168.103.2 233 Enter radius password: 233 Enter radius password: 234 234 Received disconnect from 192.168.103.2: 11: Logged out. 235 235 }}} 236 236 237 In summary, all of the access works for me because i am in `xoadmins`, but Tim is not able to login because `bbnadmins` does not have access. 237 In summary, all of the access works for me because i am in `xoadmins`, but Tim is not able to login because `bbnadmins` does not have access. 238 238 239 239 ==== Results of testing step 3B: 2012-05-26 ==== 240 240 241 Testing assertion that exoticket:20 has been resolved, so my site admin account, `cgolubit`, should be able to run this test. 241 Testing assertion that exoticket:20 has been resolved, so my site admin account, `cgolubit`, should be able to run this test. 242 242 243 243 * Per e-mail from Chris, the 8052 is 192.168.103.2, and the 8264 is 192.168.103.4. The 8052 also has the public IP address 192.1.242.4. … … 251 251 Are you sure you want to continue connecting (yes/no)? yes 252 252 Warning: Permanently added '192.168.103.2' (DSA) to the list of known hosts. 253 Enter radius password: 253 Enter radius password: 254 254 255 255 IBM Networking Operating System RackSwitch G8052. … … 275 275 {{{ 276 276 8052.bbn.xo>show mac-address-table 277 Mac address Aging Time: 300 277 Mac address Aging Time: 300 278 278 279 279 Total number of FDB entries : 26 … … 295 295 Are you sure you want to continue connecting (yes/no)? yes 296 296 Warning: Permanently added '192.168.103.4' (DSA) to the list of known hosts. 297 Enter radius password: 297 Enter radius password: 298 298 299 299 IBM Networking Operating System RackSwitch G8264. … … 317 317 * Mac address table (which is empty here) can be viewed: 318 318 {{{ 319 8264.bbn.xo>show mac-address-table 320 Mac address Aging Time: 300 319 8264.bbn.xo>show mac-address-table 320 Mac address Aging Time: 300 321 321 322 322 FDB is empty. … … 324 324 * Openflow informaton can be viewed, including DPID and controllers for an active instance: 325 325 {{{ 326 8264.bbn.xo>show openflow 1 326 8264.bbn.xo>show openflow 1 327 327 Open Flow Instance ID: 1 328 328 DataPath ID: 0x640817f4b52a00 329 329 ... 330 Configured Controllers: 330 Configured Controllers: 331 331 IP Address: 192.168.103.10 332 332 State: Active … … 340 340 {{{ 341 341 capybara,[~],09:35(255)$ ssh cgolubit@192.1.242.4 342 Enter radius password: 342 Enter radius password: 343 343 344 344 IBM Networking Operating System RackSwitch G8052. … … 411 411 ... 412 412 bbn-w1,[~],16:19(0)$ sudo whoami 413 [sudo] password for chaos: 413 [sudo] password for chaos: 414 414 root 415 415 }}} … … 421 421 ... 422 422 bbn-w2,[~],16:25(0)$ sudo whoami 423 [sudo] password for chaos: 423 [sudo] password for chaos: 424 424 root 425 425 }}} … … 430 430 ... 431 431 bbn-w3,[~],16:26(0)$ sudo whoami 432 [sudo] password for chaos: 432 [sudo] password for chaos: 433 433 root 434 434 }}} … … 485 485 Name: 192.168.100.2 486 486 Address: 192.168.100.2#53 487 Aliases: 487 Aliases: 488 488 489 489 bbn-w1.bbn.xo has address 192.168.103.101 … … 521 521 * Logout 522 522 * Now browse to `http://bbn-w2.bbn.xo`: 523 * Login as before 523 * Login as before 524 524 * Tasks -> Remote Control -> Start Remote Control in Multi-User Mode 525 525 * IMM Control -> Configuration File -> view the current configuration summary, and make a copy 526 526 * Now browse to `http://bbn-w3.bbn.xo`: 527 * Login as before 527 * Login as before 528 528 * Tasks -> Remote Control -> Start Remote Control in Multi-User Mode 529 529 * IMM Control -> Configuration File -> view the current configuration summary, and make a copy 530 530 * Now browse to `http://bbn-w4.bbn.xo`: 531 * Login as before 531 * Login as before 532 532 * Tasks -> Remote Control -> Start Remote Control in Multi-User Mode 533 533 * the console here shows that bbn-w4 is at a PXE boot prompt … … 564 564 }}} 565 565 * Now browse to `http://bbn-w4.bbn.xo`: 566 * Login as before 566 * Login as before 567 567 * IMM Control -> Configuration File -> view the current configuration summary, and make a copy 568 568 * This time, the configuration eventually loaded with no trouble 569 569 * Now browse to `http://bbn-hn.bbn.xo`: 570 * Login as before 570 * Login as before 571 571 * Tasks -> Remote Control -> Start Remote Control in Multi-User Mode 572 572 * IMM Control -> Configuration File -> view the current configuration summary, and make a copy … … 607 607 {{{ 608 608 bbn-hn,[~],16:55(0)$ ssh bbn-w4 609 chaos@bbn-w4's password: 609 chaos@bbn-w4's password: 610 610 Permission denied, please try again. 611 611 612 612 bbn-hn,[~],16:55(130)$ ssh cgolubit@bbn-w4 613 cgolubit@bbn-w4's password: 613 cgolubit@bbn-w4's password: 614 614 Permission denied, please try again. 615 615 }}} … … 618 618 {{{ 619 619 bbn-hn,[~],17:31(0)$ sudo ssh -i /opt/orca-12080/xcat/id_rsa root@bbn-w4 620 [sudo] password for chaos: 620 [sudo] password for chaos: 621 621 Last login: Thu Jul 5 17:30:40 2012 from 10.100.0.1 622 [root@bbn-w4 ~]# 623 }}} 624 625 == Step 4: GPO inventories the rack based on our own processes == 622 [root@bbn-w4 ~]# 623 }}} 624 625 == Step 4: GPO inventories the rack based on our own processes == 626 626 627 627 === Step 4A: Inventory and label physical rack contents === … … 633 633 * Use [https://wiki.exogeni.net/doku.php?id=public:hardware:rack_layout] to determine the name of each object 634 634 * If any objects can't be found there, compare to [gsw:ChaosSandbox/ExogeniRackNotes], and iterate with RENCI 635 * Physically label each device in the rack with its name on front and back 635 * Physically label each device in the rack with its name on front and back 636 636 * Inventory all hardware details for rack contents on gsw:OpsHardwareInventory 637 637 * Add an ascii rack diagram to gsw:OpsHardwareInventory … … 844 844 * The tactical overview in the top left lists 5 problems. 845 845 * Clicking on that number takes me to [https://bbn-hn.exogeni.net/rack_bbn/check_mk/view.py?view_name=svcproblems] 846 * The first problem, with state CRIT, is [https://bbn-hn.exogeni.net/rack_bbn/check_mk/view.py?view_name=service&site=&service=Multipath%20360080e50002d03ac000002cc4f69a431&host=bbn-hn.exogeni.net], a check on the multipath device on bbn-hn. 846 * The first problem, with state CRIT, is [https://bbn-hn.exogeni.net/rack_bbn/check_mk/view.py?view_name=service&site=&service=Multipath%20360080e50002d03ac000002cc4f69a431&host=bbn-hn.exogeni.net], a check on the multipath device on bbn-hn. 847 847 * I am not able to figure out, either from that page, or from poking around on bbn-hn, what the command is which is being run to get that result. I will follow up via e-mail and ask. 848 848 * I also notice that my login reports me as `cgolubit (guest)`, while my chaos account lists me as `chaos (admin)`. Josh reports that he shows up as `jbs (admin)`. … … 853 853 * When Josh tried to login, he gets the error: 854 854 {{{ 855 Your username (jbs) is listed more than once in multisite.mk. 855 Your username (jbs) is listed more than once in multisite.mk. 856 856 This is not allowed. Please check your config. 857 857 }}} … … 860 860 ==== Results of testing step 5C: 2012-05-26 ==== 861 861 862 I'm tracking testing of the RCI login problem, which is still giving a weird error, on exoticket:29. 862 I'm tracking testing of the RCI login problem, which is still giving a weird error, on exoticket:29. 863 863 864 864 Returning to: … … 876 876 Date: Sat, 26 May 2012 11:09:48 -0400 877 877 To: exogeni-design@geni.net 878 Subject: Re: [exogeni-design] question about nagios/omd plugins 878 Subject: Re: [exogeni-design] question about nagios/omd plugins 879 879 }}} 880 880 … … 884 884 {{{ 885 885 bbn-hn,[~],18:57(0)$ sudo su - rack_bbn 886 OMD[rack_bbn]:~$ 886 OMD[rack_bbn]:~$ 887 887 }}} 888 888 * As rack_bbn, i can list the check_mk information about bbn-hn.exogeni.net: … … 890 890 OMD[rack_bbn]:~$ cmk -D bbn-hn.exogeni.net | head 891 891 892 bbn-hn.exogeni.net (192.1.242.3) 892 bbn-hn.exogeni.net (192.1.242.3) 893 893 Tags: tcp, linux, nagios, hn 894 894 Host groups: linux, hn … … 897 897 Is aggregated: no 898 898 ... 899 multipath 360080e50002d03ac000002cc4f69a431 2 Multipath 360080e50002d03ac000002cc4f69a431 899 multipath 360080e50002d03ac000002cc4f69a431 2 Multipath 360080e50002d03ac000002cc4f69a431 900 900 ... 901 901 }}} … … 945 945 - State: CRITICAL 946 946 - Date: 2012-04-26 03:22:08 947 - Output: CRIT - (mpathb) paths expected: 4, paths active: 2 947 - Output: CRIT - (mpathb) paths expected: 4, paths active: 2 948 948 - 949 949 ---------------------------------- … … 980 980 Date: Wed, 23 May 2012 16:29:52 +0000 981 981 To: chaos@bbn.com 982 Subject: *** PROBLEM *** 8052.bbn.xo / Interface Ethernet40 is CRITICAL 982 Subject: *** PROBLEM *** 8052.bbn.xo / Interface Ethernet40 is CRITICAL 983 983 984 984 --SERVICE-ALERT------------------- … … 999 999 Date: Wed, 23 May 2012 16:30:52 +0000 1000 1000 To: chaos@bbn.com 1001 Subject: *** PROBLEM *** 8052.bbn.xo / Interface Ethernet30 is CRITICAL 1001 Subject: *** PROBLEM *** 8052.bbn.xo / Interface Ethernet30 is CRITICAL 1002 1002 1003 1003 --SERVICE-ALERT------------------- … … 1018 1018 Date: Wed, 23 May 2012 16:31:52 +0000 1019 1019 To: chaos@bbn.com 1020 Subject: *** RECOVERY *** 8052.bbn.xo / Interface Ethernet40 is OK 1020 Subject: *** RECOVERY *** 8052.bbn.xo / Interface Ethernet40 is OK 1021 1021 1022 1022 --SERVICE-ALERT------------------- … … 1026 1026 - Service: Interface Ethernet40 1027 1027 - - - - - - - - - - - - - - - - - 1028 - State: OK 1028 - State: OK 1029 1029 - Date: 2012-05-23 16:31:52 1030 - Output: OK - [168] (up) 1GBit/s, in: 29.71B/s(0.0%), out: 73.21B/s(0.0%) 1030 - Output: OK - [168] (up) 1GBit/s, in: 29.71B/s(0.0%), out: 73.21B/s(0.0%) 1031 1031 - 1032 1032 ---------------------------------- … … 1037 1037 Date: Wed, 23 May 2012 16:31:52 +0000 1038 1038 To: chaos@bbn.com 1039 Subject: *** RECOVERY *** 8052.bbn.xo / Interface Ethernet30 is OK 1039 Subject: *** RECOVERY *** 8052.bbn.xo / Interface Ethernet30 is OK 1040 1040 1041 1041 --SERVICE-ALERT------------------- … … 1045 1045 - Service: Interface Ethernet30 1046 1046 - - - - - - - - - - - - - - - - - 1047 - State: OK 1047 - State: OK 1048 1048 - Date: 2012-05-23 16:31:52 1049 - Output: OK - [158] (up) 1GBit/s, in: 0.00B/s(0.0%), out: 30.94B/s(0.0%) 1049 - Output: OK - [158] (up) 1GBit/s, in: 0.00B/s(0.0%), out: 30.94B/s(0.0%) 1050 1050 - 1051 1051 ---------------------------------- … … 1060 1060 Date: Wed, 23 May 2012 17:11:32 +0000 1061 1061 To: chaos@bbn.com 1062 Subject: *** PROBLEM *** bbn-w1.local / Check_MK inventory is WARNING 1062 Subject: *** PROBLEM *** bbn-w1.local / Check_MK inventory is WARNING 1063 1063 1064 1064 --SERVICE-ALERT------------------- … … 1070 1070 - State: WARNING 1071 1071 - Date: 2012-05-23 17:11:32 1072 - Output: WARNING - 3 unchecked services (lnx_if:2, qemu:1) 1072 - Output: WARNING - 3 unchecked services (lnx_if:2, qemu:1) 1073 1073 - lnx_if: Interface vnet2 1074 1074 lnx_if: Interface vnet3 … … 1082 1082 Date: Wed, 23 May 2012 17:13:32 +0000 1083 1083 To: chaos@bbn.com 1084 Subject: *** RECOVERY *** bbn-w1.local / Check_MK inventory is OK 1084 Subject: *** RECOVERY *** bbn-w1.local / Check_MK inventory is OK 1085 1085 1086 1086 --SERVICE-ALERT------------------- … … 1090 1090 - Service: Check_MK inventory 1091 1091 - - - - - - - - - - - - - - - - - 1092 - State: OK 1092 - State: OK 1093 1093 - Date: 2012-05-23 17:13:32 1094 1094 - Output: OK - no unchecked services found … … 1169 1169 1170 1170 We will want to revisit this test when GMOC has workflows in place to handle notifications for rack outages, and before there are additional rack sites and users who may need to be notified. 1171 1172 ==== Results of testing step 6C: 2013-02-14 ==== 1173 1174 Eldar confirmed via e-mail that the long-term plan is set and working: 1175 1176 {{{ 1177 From: "Urumbaev, Eldar" <eurumbae@indiana.edu> 1178 To: Josh Smift <jbs@bbn.com>, "exogeni-design@geni.net" 1179 <exogeni-design@geni.net> 1180 Subject: Re: [exogeni-design] Outage reporting to GMOC 1181 Date: Thu, 14 Feb 2013 13:33:20 +0000 1182 1183 Hi Josh, 1184 1185 We are all synched up. GMOC is subscribed to the [GENI-ORCA-USERS] 1186 geni-orca-users@googlegroups.com mailing list. The ExoGENI team has been 1187 putting [OUTAGE] or [MAINTENANCE] in subject line to help identify events 1188 that are relevant to us and require our action for tracking rack 1189 outages/maintenances. This has been working ok so far. 1190 1191 Thanks, 1192 1193 Eldar 1194 }}} 1195 1196 So, this is all set.