Changes between Initial Version and Version 1 of GENIRacksHome/OpenGENIRacks/AcceptanceTestStatusApr2014/OG-ADM-3


Ignore:
Timestamp:
05/23/14 11:33:26 (10 years ago)
Author:
lnevers@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • GENIRacksHome/OpenGENIRacks/AcceptanceTestStatusApr2014/OG-ADM-3

    v1 v1  
     1= GR-ADM-3: Full Rack Reboot Test =
     2
     3This page captures status for the test case GR-ADM-3. For additional information see the [wiki:GENIRacksHome/GRAMRacks/AcceptanceTestStatusApr2014 Acceptance Test Status - April 2014] page overall status, or the [wiki:GENIRacksHome/AcceptanceTests/GRAMAcceptanceTestsPlan GRAM Acceptance Test Plan] for details about the planned evaluation.
     4
     5
     6|| '''Step''' || '''State'''                                  ||''' Notes '''             || '''Tickets'''  ||
     7|| Step 1     ||[[span(Pass, style=background-color: green )]]||                          ||~~117~~            ||
     8|| Step 2     ||[[span(Pass, style=background-color: green )]]||                          ||                ||
     9|| Step 3     ||[[span(Pass, style=background-color: green )]]||                          ||~~120~~                ||
     10
     11[[BR]]
     12
     13|| '''State Legend'''                                   || '''Description'''                                                    ||
     14||[[span(Pass, style=background-color: green )]]        || Test completed and met all criteria                                  ||
     15||[[span(Pass: most criteria, style=background-color: #98FB98)]]|| Test completed and met most criteria. Exceptions documented  ||
     16||[[span(Fail, style=background-color: red)]]           || Test completed and failed to meet criteria.                          ||
     17||[[span(Complete, style=background-color: yellow)]]    || Test completed but will require re-execution due to expected changes ||
     18||[[span(Blocked, style=background-color: orange)]]     || Blocked by ticketed issue(s).                                        ||
     19||[[span(In Progress, style=background-color: #63B8FF)]]|| Currently under test.                                                ||
     20
     21
     22== Test Plan Steps ==
     23
     24=== Step 1. Review Documentation ===
     25
     26Review relevant rack documentation about shutdown options and make a plan for the order in which to shutdown each component.
     27
     28=== Step 2. Shutdown all devices ===
     29
     30Cleanly shutdown and/or hard-power-off all devices in the rack, and verify that everything in the rack is powered down.
     31
     32Used procedures described at https://superior.bbn.com/trac/bbn-rack/wiki/PowerDown page and followed the steps described:
     33
     34  1. Verified no active slivers existed.
     35  2. Stopped GRAM services:
     36    - sudo service gram-am stop
     37    - sudo service gram-ch stop
     38    - sudo service gram-ctrl stop
     39    - sudo service gram-vmoc stop
     40    - sudo service gram-mon stop
     41  3. Shutdown devices in the order suggested:
     42    - OpenFlow Switch - Dell Force10
     43    - Compute Nodes
     44    - Control Node
     45    - Management Switch - Dell Powerconnect 7048
     46    - UPS (not available) 
     47 
     48
     49=== Step 3. Power up and restore services ===
     50
     51Power on all devices, bring all logical components back online, and use monitoring and comprehensive health tests to verify that the rack is healthy again.
     52
     53 1. Powered on devices in order suggested:
     54   - UPS (not available)
     55   - Management Switch - Dell Powerconnect 7048
     56   - OpenFlow Switch - Dell Force10
     57   - Control Node
     58   - Compute Nodes
     59 2. Checked GRAM Services:
     60 {{{
     61lnevers@bbn-cam-ctrl-1:~$ cd /home/gram/gram/src/gram/am/gram
     62lnevers@bbn-cam-ctrl-1:/home/gram/gram/src/gram/am/gram$ sudo bash
     63[sudo] password for lnevers:
     64root@bbn-cam-ctrl-1:/home/gram/gram/src/gram/am/gram# source /etc/novarc
     65root@bbn-cam-ctrl-1:/home/gram/gram/src/gram/am/gram# export PYTHONPATH=$PYTHONPATH:/home/gram/gram/src
     66root@bbn-cam-ctrl-1:/home/gram/gram/src/gram/am/gram# python healthcheck.py
     67Starting healthcheck
     68Checking GRAM services...
     69gram-am - running
     70gram-ctrl - running
     71gram-vmoc - running
     72gram-mon - running
     73gram-ch - running
     74checking OpenStack services...
     75nova-api - running
     76nova-cert - running
     77nova-conductor - running
     78nova-consoleauth  - running
     79nova-novncproxy - running
     80nova-scheduler - running
     81quantum-dhcp-agent - running
     82quantum-metadata-agent - running
     83quantum-server - running
     84quantum-l3-agent - running
     85quantum-plugin-openvswitch-agent - running
     86glance-registry - running
     87glance-api - running
     88keystone - running
     89WARNING: Management namespace NOT found
     90Restarting Quantum-L3 service to attempt to recover the namespace - attempt 0
     91Restarting Quantum-L3 service to attempt to recover the namespace - attempt 1
     92Restarting Quantum-L3 service to attempt to recover the namespace - attempt 2
     93Restarting Quantum-L3 service to attempt to recover the namespace - attempt 3
     94Restarting Quantum-L3 service to attempt to recover the namespace - attempt 4
     95Restarting Quantum-L3 service to attempt to recover the namespace - attempt 5
     96Restarting Quantum-L3 service to attempt to recover the namespace - attempt 6
     97Restarting Quantum-L3 service to attempt to recover the namespace - attempt 7
     98Restarting Quantum-L3 service to attempt to recover the namespace - attempt 8
     99Restarting Quantum-L3 service to attempt to recover the namespace - attempt 9
     100Found management namespace and it matches config
     101Checking the status of the compute hosts:
     102
     103Binary           Host                                 Zone             Status     State Updated_At
     104nova-compute     bbn-cam-cmpe-1                       nova             enabled    :-)   2014-04-17 18:28:54
     105nova-compute     bbn-cam-cmpe-2                       nova             enabled    :-)   2014-04-17 18:28:55
     106
     107
     108Checking status of Openstack newtworking software modules:
     109
     110+--------------------------------------+--------------------+------------------------+-------+----------------+
     111+--------------------------------------+--------------------+------------------------+-------+----------------+
     112| id                                   | agent_type         | host                   | alive | admin_state_up |
     113+--------------------------------------+--------------------+------------------------+-------+----------------+
     114WARNING: the followng agent is down or not properly configured (ignore if it is a duplicate entry):
     115
     116| 2e7b8626-f0b6-4e95-a58b-67f248544ac6 | Open vSwitch agent | bbn-cam-ctrl-1.bbn.com | xxx   | True           |
     117| 47648f6d-4b89-457a-b9e8-61b48f7a6e49 | DHCP agent         | bbn-cam-ctrl-1         | :-)   | True           |
     118| 54ad9b0e-a92e-4ab0-bf99-55aabcc86237 | L3 agent           | bbn-cam-ctrl-1.bbn.com | :-)   | True           |
     119| 74426394-c899-4632-9591-0afd7a004e14 | Open vSwitch agent | bbn-cam-ctrl-1         | :-)   | True           |
     120| 7e198c11-253b-448f-b6b2-b288446be95e | Open vSwitch agent | bbn-cam-cmpe-1         | :-)   | True           |
     121| 83c68b64-2223-47d3-b4d7-6b4c94e251e9 | Open vSwitch agent | bbn-cam-cmpe-2         | :-)   | True           |
     122WARNING: the followng agent is down or not properly configured (ignore if it is a duplicate entry):
     123
     124| 9a4c9f19-a8f5-4b4d-b10d-71eec433bc44 | Open vSwitch agent | bbn-cam-cmpe-2.bbn.com | xxx   | True           |
     125WARNING: the followng agent is down or not properly configured (ignore if it is a duplicate entry):
     126
     127| 9ed5f6aa-428b-4f95-98d6-75d78f9ed015 | L3 agent           | bbn-cam-ctrl-1         | xxx   | True           |
     128WARNING: the followng agent is down or not properly configured (ignore if it is a duplicate entry):
     129
     130| a28d38c9-f088-4946-bb7e-8f63a853c5d8 | DHCP agent         | bbn-cam-ctrl-1.bbn.com | xxx   | True           |
     131WARNING: the followng agent is down or not properly configured (ignore if it is a duplicate entry):
     132
     133| a33c944f-7d3a-4560-9485-137a515cbeba | Open vSwitch agent | bbn-cam-cmpe-1.bbn.com | xxx   | True           |
     134+--------------------------------------+--------------------+------------------------+-------+----------------+
     135
     136
     137
     138Keystone - pass
     139Nova - pass
     140Glance - pass
     141Quantum - pass
     142AM is up : Get-Version succeeded at AM
     143Keystone - pass
     144Nova - pass
     145Glance - pass
     146Quantum - pass
     147AM is up : Get-Version succeeded at AM
     148Allocate - failure
     14914:57:38 INFO     omni: Loading agg_nick_cache file '/home/lnevers/.gcf/agg_nick_cache'
     15014:57:38 INFO     omni: Loading config file /home/lnevers/.gcf/omni_config
     15114:57:38 INFO     omni: Using control framework my_gcf
     15214:57:38 WARNING  omni: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires in <= 3 hours
     15314:57:38 INFO     omni: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires on 2014-04-17 19:36:42 UTC
     15414:57:38 INFO     omni: Allocate slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001:
     15514:57:38 WARNING  omni: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires in <= 3 hours on 2014-04-17 19:36:42 UTC
     156Allocation of slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001 failed: Error from Aggregate: code 102: No credential was found with appropriate privileges. Tried urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser. Last failure: Cred urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser fails: Credential doesn't grant rights to you (urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+lnevers), but to urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser (over object urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY)..
     157
     15814:57:38 INFO     omni:  ------------------------------------------------------------
     15914:57:38 INFO     omni:  Completed allocate:
     160Args: allocate DUMMY /tmp/dummy.rspec
     161
     162  Result Summary: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires in <= 3 hours on 2014-04-17 19:36:42 UTC
     163Allocation of slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001 failed: Error from Aggregate: code 102: No credential was found with appropriate privileges. Tried urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser. Last failure: Cred urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser fails: Credential doesn't grant rights to you (urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+lnevers), but to urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser (over object urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY)..
     164Allocate slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY failed at https://bbn-cam-ctrl-1:5001
     16514:57:38 INFO     omni:  ============================================================
     166
     167Provision - failure
     16814:57:39 INFO     omni: Loading agg_nick_cache file '/home/lnevers/.gcf/agg_nick_cache'
     16914:57:39 INFO     omni: Loading config file /home/lnevers/.gcf/omni_config
     17014:57:39 INFO     omni: Using control framework my_gcf
     17114:57:39 WARNING  omni: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires in <= 3 hours
     17214:57:39 INFO     omni: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires on 2014-04-17 19:36:42 UTC
     17314:57:39 ERROR    omni: Failed to read user key from ~/.ssh/id_rsa.pub: [Errno 2] No such file or directory: '/home/lnevers/.ssh/id_rsa.pub'
     17414:57:39 WARNING  omni: Empty keys for user urn:publicid:IDN+geni:dell:gcf+user+lnevers
     17514:57:39 INFO     omni: Provision slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001
     17614:57:39 WARNING  omni: Provision of slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001 failed: Error from Aggregate: code 102: No credential was found with appropriate privileges. Tried urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser. Last failure: Cred urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser fails: Credential doesn't grant rights to you (urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+lnevers), but to urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser (over object urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY).
     17714:57:39 INFO     omni:  ------------------------------------------------------------
     17814:57:39 INFO     omni:  Completed provision:
     179Args: provision DUMMY
     180
     181  Result Summary: Provision of slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001 failed: Error from Aggregate: code 102: No credential was found with appropriate privileges. Tried urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser. Last failure: Cred urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser fails: Credential doesn't grant rights to you (urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+lnevers), but to urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser (over object urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY).
     182Provision slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY failed at https://bbn-cam-ctrl-1:5001
     18314:57:39 INFO     omni:  ============================================================
     184
     185Delete - failure
     186}}}
     187
     188Will verify the operations failure fix separately - A unique slice id must be used.