| 1 | = GR-ADM-3: Full Rack Reboot Test = |
| 2 | |
| 3 | This page captures status for the test case GR-ADM-3. For additional information see the [wiki:GENIRacksHome/GRAMRacks/AcceptanceTestStatusApr2014 Acceptance Test Status - April 2014] page overall status, or the [wiki:GENIRacksHome/AcceptanceTests/GRAMAcceptanceTestsPlan GRAM Acceptance Test Plan] for details about the planned evaluation. |
| 4 | |
| 5 | |
| 6 | || '''Step''' || '''State''' ||''' Notes ''' || '''Tickets''' || |
| 7 | || Step 1 ||[[span(Pass, style=background-color: green )]]|| ||~~117~~ || |
| 8 | || Step 2 ||[[span(Pass, style=background-color: green )]]|| || || |
| 9 | || Step 3 ||[[span(Pass, style=background-color: green )]]|| ||~~120~~ || |
| 10 | |
| 11 | [[BR]] |
| 12 | |
| 13 | || '''State Legend''' || '''Description''' || |
| 14 | ||[[span(Pass, style=background-color: green )]] || Test completed and met all criteria || |
| 15 | ||[[span(Pass: most criteria, style=background-color: #98FB98)]]|| Test completed and met most criteria. Exceptions documented || |
| 16 | ||[[span(Fail, style=background-color: red)]] || Test completed and failed to meet criteria. || |
| 17 | ||[[span(Complete, style=background-color: yellow)]] || Test completed but will require re-execution due to expected changes || |
| 18 | ||[[span(Blocked, style=background-color: orange)]] || Blocked by ticketed issue(s). || |
| 19 | ||[[span(In Progress, style=background-color: #63B8FF)]]|| Currently under test. || |
| 20 | |
| 21 | |
| 22 | == Test Plan Steps == |
| 23 | |
| 24 | === Step 1. Review Documentation === |
| 25 | |
| 26 | Review relevant rack documentation about shutdown options and make a plan for the order in which to shutdown each component. |
| 27 | |
| 28 | === Step 2. Shutdown all devices === |
| 29 | |
| 30 | Cleanly shutdown and/or hard-power-off all devices in the rack, and verify that everything in the rack is powered down. |
| 31 | |
| 32 | Used procedures described at https://superior.bbn.com/trac/bbn-rack/wiki/PowerDown page and followed the steps described: |
| 33 | |
| 34 | 1. Verified no active slivers existed. |
| 35 | 2. Stopped GRAM services: |
| 36 | - sudo service gram-am stop |
| 37 | - sudo service gram-ch stop |
| 38 | - sudo service gram-ctrl stop |
| 39 | - sudo service gram-vmoc stop |
| 40 | - sudo service gram-mon stop |
| 41 | 3. Shutdown devices in the order suggested: |
| 42 | - OpenFlow Switch - Dell Force10 |
| 43 | - Compute Nodes |
| 44 | - Control Node |
| 45 | - Management Switch - Dell Powerconnect 7048 |
| 46 | - UPS (not available) |
| 47 | |
| 48 | |
| 49 | === Step 3. Power up and restore services === |
| 50 | |
| 51 | Power on all devices, bring all logical components back online, and use monitoring and comprehensive health tests to verify that the rack is healthy again. |
| 52 | |
| 53 | 1. Powered on devices in order suggested: |
| 54 | - UPS (not available) |
| 55 | - Management Switch - Dell Powerconnect 7048 |
| 56 | - OpenFlow Switch - Dell Force10 |
| 57 | - Control Node |
| 58 | - Compute Nodes |
| 59 | 2. Checked GRAM Services: |
| 60 | {{{ |
| 61 | lnevers@bbn-cam-ctrl-1:~$ cd /home/gram/gram/src/gram/am/gram |
| 62 | lnevers@bbn-cam-ctrl-1:/home/gram/gram/src/gram/am/gram$ sudo bash |
| 63 | [sudo] password for lnevers: |
| 64 | root@bbn-cam-ctrl-1:/home/gram/gram/src/gram/am/gram# source /etc/novarc |
| 65 | root@bbn-cam-ctrl-1:/home/gram/gram/src/gram/am/gram# export PYTHONPATH=$PYTHONPATH:/home/gram/gram/src |
| 66 | root@bbn-cam-ctrl-1:/home/gram/gram/src/gram/am/gram# python healthcheck.py |
| 67 | Starting healthcheck |
| 68 | Checking GRAM services... |
| 69 | gram-am - running |
| 70 | gram-ctrl - running |
| 71 | gram-vmoc - running |
| 72 | gram-mon - running |
| 73 | gram-ch - running |
| 74 | checking OpenStack services... |
| 75 | nova-api - running |
| 76 | nova-cert - running |
| 77 | nova-conductor - running |
| 78 | nova-consoleauth - running |
| 79 | nova-novncproxy - running |
| 80 | nova-scheduler - running |
| 81 | quantum-dhcp-agent - running |
| 82 | quantum-metadata-agent - running |
| 83 | quantum-server - running |
| 84 | quantum-l3-agent - running |
| 85 | quantum-plugin-openvswitch-agent - running |
| 86 | glance-registry - running |
| 87 | glance-api - running |
| 88 | keystone - running |
| 89 | WARNING: Management namespace NOT found |
| 90 | Restarting Quantum-L3 service to attempt to recover the namespace - attempt 0 |
| 91 | Restarting Quantum-L3 service to attempt to recover the namespace - attempt 1 |
| 92 | Restarting Quantum-L3 service to attempt to recover the namespace - attempt 2 |
| 93 | Restarting Quantum-L3 service to attempt to recover the namespace - attempt 3 |
| 94 | Restarting Quantum-L3 service to attempt to recover the namespace - attempt 4 |
| 95 | Restarting Quantum-L3 service to attempt to recover the namespace - attempt 5 |
| 96 | Restarting Quantum-L3 service to attempt to recover the namespace - attempt 6 |
| 97 | Restarting Quantum-L3 service to attempt to recover the namespace - attempt 7 |
| 98 | Restarting Quantum-L3 service to attempt to recover the namespace - attempt 8 |
| 99 | Restarting Quantum-L3 service to attempt to recover the namespace - attempt 9 |
| 100 | Found management namespace and it matches config |
| 101 | Checking the status of the compute hosts: |
| 102 | |
| 103 | Binary Host Zone Status State Updated_At |
| 104 | nova-compute bbn-cam-cmpe-1 nova enabled :-) 2014-04-17 18:28:54 |
| 105 | nova-compute bbn-cam-cmpe-2 nova enabled :-) 2014-04-17 18:28:55 |
| 106 | |
| 107 | |
| 108 | Checking status of Openstack newtworking software modules: |
| 109 | |
| 110 | +--------------------------------------+--------------------+------------------------+-------+----------------+ |
| 111 | +--------------------------------------+--------------------+------------------------+-------+----------------+ |
| 112 | | id | agent_type | host | alive | admin_state_up | |
| 113 | +--------------------------------------+--------------------+------------------------+-------+----------------+ |
| 114 | WARNING: the followng agent is down or not properly configured (ignore if it is a duplicate entry): |
| 115 | |
| 116 | | 2e7b8626-f0b6-4e95-a58b-67f248544ac6 | Open vSwitch agent | bbn-cam-ctrl-1.bbn.com | xxx | True | |
| 117 | | 47648f6d-4b89-457a-b9e8-61b48f7a6e49 | DHCP agent | bbn-cam-ctrl-1 | :-) | True | |
| 118 | | 54ad9b0e-a92e-4ab0-bf99-55aabcc86237 | L3 agent | bbn-cam-ctrl-1.bbn.com | :-) | True | |
| 119 | | 74426394-c899-4632-9591-0afd7a004e14 | Open vSwitch agent | bbn-cam-ctrl-1 | :-) | True | |
| 120 | | 7e198c11-253b-448f-b6b2-b288446be95e | Open vSwitch agent | bbn-cam-cmpe-1 | :-) | True | |
| 121 | | 83c68b64-2223-47d3-b4d7-6b4c94e251e9 | Open vSwitch agent | bbn-cam-cmpe-2 | :-) | True | |
| 122 | WARNING: the followng agent is down or not properly configured (ignore if it is a duplicate entry): |
| 123 | |
| 124 | | 9a4c9f19-a8f5-4b4d-b10d-71eec433bc44 | Open vSwitch agent | bbn-cam-cmpe-2.bbn.com | xxx | True | |
| 125 | WARNING: the followng agent is down or not properly configured (ignore if it is a duplicate entry): |
| 126 | |
| 127 | | 9ed5f6aa-428b-4f95-98d6-75d78f9ed015 | L3 agent | bbn-cam-ctrl-1 | xxx | True | |
| 128 | WARNING: the followng agent is down or not properly configured (ignore if it is a duplicate entry): |
| 129 | |
| 130 | | a28d38c9-f088-4946-bb7e-8f63a853c5d8 | DHCP agent | bbn-cam-ctrl-1.bbn.com | xxx | True | |
| 131 | WARNING: the followng agent is down or not properly configured (ignore if it is a duplicate entry): |
| 132 | |
| 133 | | a33c944f-7d3a-4560-9485-137a515cbeba | Open vSwitch agent | bbn-cam-cmpe-1.bbn.com | xxx | True | |
| 134 | +--------------------------------------+--------------------+------------------------+-------+----------------+ |
| 135 | |
| 136 | |
| 137 | |
| 138 | Keystone - pass |
| 139 | Nova - pass |
| 140 | Glance - pass |
| 141 | Quantum - pass |
| 142 | AM is up : Get-Version succeeded at AM |
| 143 | Keystone - pass |
| 144 | Nova - pass |
| 145 | Glance - pass |
| 146 | Quantum - pass |
| 147 | AM is up : Get-Version succeeded at AM |
| 148 | Allocate - failure |
| 149 | 14:57:38 INFO omni: Loading agg_nick_cache file '/home/lnevers/.gcf/agg_nick_cache' |
| 150 | 14:57:38 INFO omni: Loading config file /home/lnevers/.gcf/omni_config |
| 151 | 14:57:38 INFO omni: Using control framework my_gcf |
| 152 | 14:57:38 WARNING omni: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires in <= 3 hours |
| 153 | 14:57:38 INFO omni: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires on 2014-04-17 19:36:42 UTC |
| 154 | 14:57:38 INFO omni: Allocate slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001: |
| 155 | 14:57:38 WARNING omni: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires in <= 3 hours on 2014-04-17 19:36:42 UTC |
| 156 | Allocation of slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001 failed: Error from Aggregate: code 102: No credential was found with appropriate privileges. Tried urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser. Last failure: Cred urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser fails: Credential doesn't grant rights to you (urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+lnevers), but to urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser (over object urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY).. |
| 157 | |
| 158 | 14:57:38 INFO omni: ------------------------------------------------------------ |
| 159 | 14:57:38 INFO omni: Completed allocate: |
| 160 | Args: allocate DUMMY /tmp/dummy.rspec |
| 161 | |
| 162 | Result Summary: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires in <= 3 hours on 2014-04-17 19:36:42 UTC |
| 163 | Allocation of slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001 failed: Error from Aggregate: code 102: No credential was found with appropriate privileges. Tried urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser. Last failure: Cred urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser fails: Credential doesn't grant rights to you (urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+lnevers), but to urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser (over object urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY).. |
| 164 | Allocate slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY failed at https://bbn-cam-ctrl-1:5001 |
| 165 | 14:57:38 INFO omni: ============================================================ |
| 166 | |
| 167 | Provision - failure |
| 168 | 14:57:39 INFO omni: Loading agg_nick_cache file '/home/lnevers/.gcf/agg_nick_cache' |
| 169 | 14:57:39 INFO omni: Loading config file /home/lnevers/.gcf/omni_config |
| 170 | 14:57:39 INFO omni: Using control framework my_gcf |
| 171 | 14:57:39 WARNING omni: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires in <= 3 hours |
| 172 | 14:57:39 INFO omni: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires on 2014-04-17 19:36:42 UTC |
| 173 | 14:57:39 ERROR omni: Failed to read user key from ~/.ssh/id_rsa.pub: [Errno 2] No such file or directory: '/home/lnevers/.ssh/id_rsa.pub' |
| 174 | 14:57:39 WARNING omni: Empty keys for user urn:publicid:IDN+geni:dell:gcf+user+lnevers |
| 175 | 14:57:39 INFO omni: Provision slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001 |
| 176 | 14:57:39 WARNING omni: Provision of slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001 failed: Error from Aggregate: code 102: No credential was found with appropriate privileges. Tried urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser. Last failure: Cred urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser fails: Credential doesn't grant rights to you (urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+lnevers), but to urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser (over object urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY). |
| 177 | 14:57:39 INFO omni: ------------------------------------------------------------ |
| 178 | 14:57:39 INFO omni: Completed provision: |
| 179 | Args: provision DUMMY |
| 180 | |
| 181 | Result Summary: Provision of slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001 failed: Error from Aggregate: code 102: No credential was found with appropriate privileges. Tried urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser. Last failure: Cred urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser fails: Credential doesn't grant rights to you (urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+lnevers), but to urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser (over object urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY). |
| 182 | Provision slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY failed at https://bbn-cam-ctrl-1:5001 |
| 183 | 14:57:39 INFO omni: ============================================================ |
| 184 | |
| 185 | Delete - failure |
| 186 | }}} |
| 187 | |
| 188 | Will verify the operations failure fix separately - A unique slice id must be used. |