= OG-ADM-3: Full Rack Reboot Test = This page captures status for the test case OG-ADM-3. For additional information see the [wiki:GENIRacksHome/OpenGENIRacks/AcceptanceTestStatusApr2014 Acceptance Test Status - April 2014] page overall status, or the [wiki:GENIRacksHome/AcceptanceTests/OpenGENIAcceptanceTestsPlan OpenGENI Acceptance Test Plan] for details about the planned evaluation. || '''Step''' || '''State''' ||''' Notes ''' || '''Tickets''' || || Step 1 ||[[span(Pass, style=background-color: green )]]|| ||~~117~~ || || Step 2 ||[[span(Pass, style=background-color: green )]]|| || || || Step 3 ||[[span(Pass, style=background-color: green )]]|| ||~~120~~ || [[BR]] || '''State Legend''' || '''Description''' || ||[[span(Pass, style=background-color: green )]] || Test completed and met all criteria || ||[[span(Pass: most criteria, style=background-color: #98FB98)]]|| Test completed and met most criteria. Exceptions documented || ||[[span(Fail, style=background-color: red)]] || Test completed and failed to meet criteria. || ||[[span(Complete, style=background-color: yellow)]] || Test completed but will require re-execution due to expected changes || ||[[span(Blocked, style=background-color: orange)]] || Blocked by ticketed issue(s). || ||[[span(In Progress, style=background-color: #63B8FF)]]|| Currently under test. || == Test Plan Steps == === Step 1. Review Documentation === Review relevant rack documentation about shutdown options and make a plan for the order in which to shutdown each component. === Step 2. Shutdown all devices === Cleanly shutdown and/or hard-power-off all devices in the rack, and verify that everything in the rack is powered down. Used procedures described at https://superior.bbn.com/trac/bbn-rack/wiki/PowerDown page and followed the steps described: 1. Verified no active slivers existed. 2. Stopped OpenGENI services: - sudo service gram-am stop - sudo service gram-ch stop - sudo service gram-ctrl stop - sudo service gram-vmoc stop - sudo service gram-mon stop 3. Shutdown devices in the order suggested: - OpenFlow Switch - Dell Force10 - Compute Nodes - Control Node - Management Switch - Dell Powerconnect 7048 - UPS (not available) === Step 3. Power up and restore services === Power on all devices, bring all logical components back online, and use monitoring and comprehensive health tests to verify that the rack is healthy again. 1. Powered on devices in order suggested: - UPS (not available) - Management Switch - Dell Powerconnect 7048 - OpenFlow Switch - Dell Force10 - Control Node - Compute Nodes 2. Checked OpenGENI Services: {{{ lnevers@bbn-cam-ctrl-1:~$ cd /home/gram/gram/src/gram/am/gram lnevers@bbn-cam-ctrl-1:/home/gram/gram/src/gram/am/gram$ sudo bash [sudo] password for lnevers: root@bbn-cam-ctrl-1:/home/gram/gram/src/gram/am/gram# source /etc/novarc root@bbn-cam-ctrl-1:/home/gram/gram/src/gram/am/gram# export PYTHONPATH=$PYTHONPATH:/home/gram/gram/src root@bbn-cam-ctrl-1:/home/gram/gram/src/gram/am/gram# python healthcheck.py Starting healthcheck Checking OpenGENI services... gram-am - running gram-ctrl - running gram-vmoc - running gram-mon - running gram-ch - running checking OpenStack services... nova-api - running nova-cert - running nova-conductor - running nova-consoleauth - running nova-novncproxy - running nova-scheduler - running quantum-dhcp-agent - running quantum-metadata-agent - running quantum-server - running quantum-l3-agent - running quantum-plugin-openvswitch-agent - running glance-registry - running glance-api - running keystone - running WARNING: Management namespace NOT found Restarting Quantum-L3 service to attempt to recover the namespace - attempt 0 Restarting Quantum-L3 service to attempt to recover the namespace - attempt 1 Restarting Quantum-L3 service to attempt to recover the namespace - attempt 2 Restarting Quantum-L3 service to attempt to recover the namespace - attempt 3 Restarting Quantum-L3 service to attempt to recover the namespace - attempt 4 Restarting Quantum-L3 service to attempt to recover the namespace - attempt 5 Restarting Quantum-L3 service to attempt to recover the namespace - attempt 6 Restarting Quantum-L3 service to attempt to recover the namespace - attempt 7 Restarting Quantum-L3 service to attempt to recover the namespace - attempt 8 Restarting Quantum-L3 service to attempt to recover the namespace - attempt 9 Found management namespace and it matches config Checking the status of the compute hosts: Binary Host Zone Status State Updated_At nova-compute bbn-cam-cmpe-1 nova enabled :-) 2014-04-17 18:28:54 nova-compute bbn-cam-cmpe-2 nova enabled :-) 2014-04-17 18:28:55 Checking status of Openstack newtworking software modules: +--------------------------------------+--------------------+------------------------+-------+----------------+ +--------------------------------------+--------------------+------------------------+-------+----------------+ | id | agent_type | host | alive | admin_state_up | +--------------------------------------+--------------------+------------------------+-------+----------------+ WARNING: the followng agent is down or not properly configured (ignore if it is a duplicate entry): | 2e7b8626-f0b6-4e95-a58b-67f248544ac6 | Open vSwitch agent | bbn-cam-ctrl-1.bbn.com | xxx | True | | 47648f6d-4b89-457a-b9e8-61b48f7a6e49 | DHCP agent | bbn-cam-ctrl-1 | :-) | True | | 54ad9b0e-a92e-4ab0-bf99-55aabcc86237 | L3 agent | bbn-cam-ctrl-1.bbn.com | :-) | True | | 74426394-c899-4632-9591-0afd7a004e14 | Open vSwitch agent | bbn-cam-ctrl-1 | :-) | True | | 7e198c11-253b-448f-b6b2-b288446be95e | Open vSwitch agent | bbn-cam-cmpe-1 | :-) | True | | 83c68b64-2223-47d3-b4d7-6b4c94e251e9 | Open vSwitch agent | bbn-cam-cmpe-2 | :-) | True | WARNING: the followng agent is down or not properly configured (ignore if it is a duplicate entry): | 9a4c9f19-a8f5-4b4d-b10d-71eec433bc44 | Open vSwitch agent | bbn-cam-cmpe-2.bbn.com | xxx | True | WARNING: the followng agent is down or not properly configured (ignore if it is a duplicate entry): | 9ed5f6aa-428b-4f95-98d6-75d78f9ed015 | L3 agent | bbn-cam-ctrl-1 | xxx | True | WARNING: the followng agent is down or not properly configured (ignore if it is a duplicate entry): | a28d38c9-f088-4946-bb7e-8f63a853c5d8 | DHCP agent | bbn-cam-ctrl-1.bbn.com | xxx | True | WARNING: the followng agent is down or not properly configured (ignore if it is a duplicate entry): | a33c944f-7d3a-4560-9485-137a515cbeba | Open vSwitch agent | bbn-cam-cmpe-1.bbn.com | xxx | True | +--------------------------------------+--------------------+------------------------+-------+----------------+ Keystone - pass Nova - pass Glance - pass Quantum - pass AM is up : Get-Version succeeded at AM Keystone - pass Nova - pass Glance - pass Quantum - pass AM is up : Get-Version succeeded at AM Allocate - failure 14:57:38 INFO omni: Loading agg_nick_cache file '/home/lnevers/.gcf/agg_nick_cache' 14:57:38 INFO omni: Loading config file /home/lnevers/.gcf/omni_config 14:57:38 INFO omni: Using control framework my_gcf 14:57:38 WARNING omni: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires in <= 3 hours 14:57:38 INFO omni: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires on 2014-04-17 19:36:42 UTC 14:57:38 INFO omni: Allocate slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001: 14:57:38 WARNING omni: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires in <= 3 hours on 2014-04-17 19:36:42 UTC Allocation of slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001 failed: Error from Aggregate: code 102: No credential was found with appropriate privileges. Tried urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser. Last failure: Cred urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser fails: Credential doesn't grant rights to you (urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+lnevers), but to urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser (over object urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY).. 14:57:38 INFO omni: ------------------------------------------------------------ 14:57:38 INFO omni: Completed allocate: Args: allocate DUMMY /tmp/dummy.rspec Result Summary: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires in <= 3 hours on 2014-04-17 19:36:42 UTC Allocation of slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001 failed: Error from Aggregate: code 102: No credential was found with appropriate privileges. Tried urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser. Last failure: Cred urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser fails: Credential doesn't grant rights to you (urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+lnevers), but to urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser (over object urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY).. Allocate slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY failed at https://bbn-cam-ctrl-1:5001 14:57:38 INFO omni: ============================================================ Provision - failure 14:57:39 INFO omni: Loading agg_nick_cache file '/home/lnevers/.gcf/agg_nick_cache' 14:57:39 INFO omni: Loading config file /home/lnevers/.gcf/omni_config 14:57:39 INFO omni: Using control framework my_gcf 14:57:39 WARNING omni: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires in <= 3 hours 14:57:39 INFO omni: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires on 2014-04-17 19:36:42 UTC 14:57:39 ERROR omni: Failed to read user key from ~/.ssh/id_rsa.pub: [Errno 2] No such file or directory: '/home/lnevers/.ssh/id_rsa.pub' 14:57:39 WARNING omni: Empty keys for user urn:publicid:IDN+geni:dell:gcf+user+lnevers 14:57:39 INFO omni: Provision slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001 14:57:39 WARNING omni: Provision of slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001 failed: Error from Aggregate: code 102: No credential was found with appropriate privileges. Tried urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser. Last failure: Cred urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser fails: Credential doesn't grant rights to you (urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+lnevers), but to urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser (over object urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY). 14:57:39 INFO omni: ------------------------------------------------------------ 14:57:39 INFO omni: Completed provision: Args: provision DUMMY Result Summary: Provision of slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001 failed: Error from Aggregate: code 102: No credential was found with appropriate privileges. Tried urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser. Last failure: Cred urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser fails: Credential doesn't grant rights to you (urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+lnevers), but to urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser (over object urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY). Provision slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY failed at https://bbn-cam-ctrl-1:5001 14:57:39 INFO omni: ============================================================ Delete - failure }}} Will verify the operations failure fix separately - A unique slice id must be used.