wiki:GENIRacksHome/OpenGENIRacks/AcceptanceTestStatusApr2014/OG-ADM-3

Version 3 (modified by lnevers@bbn.com, 10 years ago) (diff)

--

OG-ADM-3: Full Rack Reboot Test

This page captures status for the test case OG-ADM-3. For additional information see the Acceptance Test Status - April 2014 page overall status, or the OpenGENI Acceptance Test Plan for details about the planned evaluation.

Step State Notes Tickets
Step 1 Pass 117
Step 2 Pass
Step 3 Pass 120


State Legend Description
Pass Test completed and met all criteria
Pass: most criteria Test completed and met most criteria. Exceptions documented
Fail Test completed and failed to meet criteria.
Complete Test completed but will require re-execution due to expected changes
Blocked Blocked by ticketed issue(s).
In Progress Currently under test.

Test Plan Steps

Step 1. Review Documentation

Review relevant rack documentation about shutdown options and make a plan for the order in which to shutdown each component.

Step 2. Shutdown all devices

Cleanly shutdown and/or hard-power-off all devices in the rack, and verify that everything in the rack is powered down.

Used procedures described at http://groups.geni.net/geni/wiki/GENIRacksHome/OpenGENIRacks/PowerDown page and followed the steps described:

  1. Verified no active slivers existed.
  2. Stopped OpenGENI services:
    • sudo service gram-am stop
    • sudo service gram-ch stop
    • sudo service gram-ctrl stop
    • sudo service gram-vmoc stop
    • sudo service gram-mon stop
  3. Shutdown devices in the order suggested:
    • OpenFlow Switch - Dell Force10
    • Compute Nodes
    • Control Node
    • Management Switch - Dell Powerconnect 7048
    • UPS (not available)

Step 3. Power up and restore services

Power on all devices, bring all logical components back online, and use monitoring and comprehensive health tests to verify that the rack is healthy again.

  1. Powered on devices in order suggested:
    • UPS (not available)
    • Management Switch - Dell Powerconnect 7048
    • OpenFlow Switch - Dell Force10
    • Control Node
    • Compute Nodes
  2. Checked OpenGENI Services:
    lnevers@bbn-cam-ctrl-1:~$ cd /home/gram/gram/src/gram/am/gram
    lnevers@bbn-cam-ctrl-1:/home/gram/gram/src/gram/am/gram$ sudo bash
    [sudo] password for lnevers: 
    root@bbn-cam-ctrl-1:/home/gram/gram/src/gram/am/gram# source /etc/novarc
    root@bbn-cam-ctrl-1:/home/gram/gram/src/gram/am/gram# export PYTHONPATH=$PYTHONPATH:/home/gram/gram/src
    root@bbn-cam-ctrl-1:/home/gram/gram/src/gram/am/gram# python healthcheck.py
    Starting healthcheck
    Checking OpenGENI services...
    gram-am - running
    gram-ctrl - running
    gram-vmoc - running
    gram-mon - running
    gram-ch - running
    checking OpenStack services...
    nova-api - running
    nova-cert - running
    nova-conductor - running
    nova-consoleauth  - running
    nova-novncproxy - running
    nova-scheduler - running
    quantum-dhcp-agent - running
    quantum-metadata-agent - running
    quantum-server - running
    quantum-l3-agent - running
    quantum-plugin-openvswitch-agent - running
    glance-registry - running
    glance-api - running
    keystone - running
    WARNING: Management namespace NOT found
    Restarting Quantum-L3 service to attempt to recover the namespace - attempt 0
    Restarting Quantum-L3 service to attempt to recover the namespace - attempt 1
    Restarting Quantum-L3 service to attempt to recover the namespace - attempt 2
    Restarting Quantum-L3 service to attempt to recover the namespace - attempt 3
    Restarting Quantum-L3 service to attempt to recover the namespace - attempt 4
    Restarting Quantum-L3 service to attempt to recover the namespace - attempt 5
    Restarting Quantum-L3 service to attempt to recover the namespace - attempt 6
    Restarting Quantum-L3 service to attempt to recover the namespace - attempt 7
    Restarting Quantum-L3 service to attempt to recover the namespace - attempt 8
    Restarting Quantum-L3 service to attempt to recover the namespace - attempt 9
    Found management namespace and it matches config
    Checking the status of the compute hosts: 
    
    Binary           Host                                 Zone             Status     State Updated_At
    nova-compute     bbn-cam-cmpe-1                       nova             enabled    :-)   2014-04-17 18:28:54
    nova-compute     bbn-cam-cmpe-2                       nova             enabled    :-)   2014-04-17 18:28:55
    
    
    Checking status of Openstack newtworking software modules: 
    
    +--------------------------------------+--------------------+------------------------+-------+----------------+
    +--------------------------------------+--------------------+------------------------+-------+----------------+
    | id                                   | agent_type         | host                   | alive | admin_state_up |
    +--------------------------------------+--------------------+------------------------+-------+----------------+
    WARNING: the followng agent is down or not properly configured (ignore if it is a duplicate entry): 
    
    | 2e7b8626-f0b6-4e95-a58b-67f248544ac6 | Open vSwitch agent | bbn-cam-ctrl-1.bbn.com | xxx   | True           |
    | 47648f6d-4b89-457a-b9e8-61b48f7a6e49 | DHCP agent         | bbn-cam-ctrl-1         | :-)   | True           |
    | 54ad9b0e-a92e-4ab0-bf99-55aabcc86237 | L3 agent           | bbn-cam-ctrl-1.bbn.com | :-)   | True           |
    | 74426394-c899-4632-9591-0afd7a004e14 | Open vSwitch agent | bbn-cam-ctrl-1         | :-)   | True           |
    | 7e198c11-253b-448f-b6b2-b288446be95e | Open vSwitch agent | bbn-cam-cmpe-1         | :-)   | True           |
    | 83c68b64-2223-47d3-b4d7-6b4c94e251e9 | Open vSwitch agent | bbn-cam-cmpe-2         | :-)   | True           |
    WARNING: the followng agent is down or not properly configured (ignore if it is a duplicate entry): 
    
    | 9a4c9f19-a8f5-4b4d-b10d-71eec433bc44 | Open vSwitch agent | bbn-cam-cmpe-2.bbn.com | xxx   | True           |
    WARNING: the followng agent is down or not properly configured (ignore if it is a duplicate entry): 
    
    | 9ed5f6aa-428b-4f95-98d6-75d78f9ed015 | L3 agent           | bbn-cam-ctrl-1         | xxx   | True           |
    WARNING: the followng agent is down or not properly configured (ignore if it is a duplicate entry): 
    
    | a28d38c9-f088-4946-bb7e-8f63a853c5d8 | DHCP agent         | bbn-cam-ctrl-1.bbn.com | xxx   | True           |
    WARNING: the followng agent is down or not properly configured (ignore if it is a duplicate entry): 
    
    | a33c944f-7d3a-4560-9485-137a515cbeba | Open vSwitch agent | bbn-cam-cmpe-1.bbn.com | xxx   | True           |
    +--------------------------------------+--------------------+------------------------+-------+----------------+
    
    
    
    Keystone - pass
    Nova - pass
    Glance - pass
    Quantum - pass
    AM is up : Get-Version succeeded at AM
    Keystone - pass
    Nova - pass
    Glance - pass
    Quantum - pass
    AM is up : Get-Version succeeded at AM
    Allocate - failure
    14:57:38 INFO     omni: Loading agg_nick_cache file '/home/lnevers/.gcf/agg_nick_cache'
    14:57:38 INFO     omni: Loading config file /home/lnevers/.gcf/omni_config
    14:57:38 INFO     omni: Using control framework my_gcf
    14:57:38 WARNING  omni: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires in <= 3 hours
    14:57:38 INFO     omni: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires on 2014-04-17 19:36:42 UTC
    14:57:38 INFO     omni: Allocate slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001:
    14:57:38 WARNING  omni: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires in <= 3 hours on 2014-04-17 19:36:42 UTC
    Allocation of slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001 failed: Error from Aggregate: code 102: No credential was found with appropriate privileges. Tried urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser. Last failure: Cred urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser fails: Credential doesn't grant rights to you (urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+lnevers), but to urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser (over object urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY)..
    
    14:57:38 INFO     omni:  ------------------------------------------------------------
    14:57:38 INFO     omni:  Completed allocate:
    Args: allocate DUMMY /tmp/dummy.rspec
    
      Result Summary: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires in <= 3 hours on 2014-04-17 19:36:42 UTC
    Allocation of slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001 failed: Error from Aggregate: code 102: No credential was found with appropriate privileges. Tried urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser. Last failure: Cred urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser fails: Credential doesn't grant rights to you (urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+lnevers), but to urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser (over object urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY)..
    Allocate slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY failed at https://bbn-cam-ctrl-1:5001 
    14:57:38 INFO     omni:  ============================================================
    
    Provision - failure
    14:57:39 INFO     omni: Loading agg_nick_cache file '/home/lnevers/.gcf/agg_nick_cache'
    14:57:39 INFO     omni: Loading config file /home/lnevers/.gcf/omni_config
    14:57:39 INFO     omni: Using control framework my_gcf
    14:57:39 WARNING  omni: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires in <= 3 hours
    14:57:39 INFO     omni: Slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY expires on 2014-04-17 19:36:42 UTC
    14:57:39 ERROR    omni: Failed to read user key from ~/.ssh/id_rsa.pub: [Errno 2] No such file or directory: '/home/lnevers/.ssh/id_rsa.pub'
    14:57:39 WARNING  omni: Empty keys for user urn:publicid:IDN+geni:dell:gcf+user+lnevers
    14:57:39 INFO     omni: Provision slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001
    14:57:39 WARNING  omni: Provision of slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001 failed: Error from Aggregate: code 102: No credential was found with appropriate privileges. Tried urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser. Last failure: Cred urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser fails: Credential doesn't grant rights to you (urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+lnevers), but to urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser (over object urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY).
    14:57:39 INFO     omni:  ------------------------------------------------------------
    14:57:39 INFO     omni:  Completed provision:
    Args: provision DUMMY
    
      Result Summary: Provision of slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY at https://bbn-cam-ctrl-1:5001 failed: Error from Aggregate: code 102: No credential was found with appropriate privileges. Tried urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser. Last failure: Cred urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser fails: Credential doesn't grant rights to you (urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+lnevers), but to urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser (over object urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY).
    Provision slivers in slice urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+DUMMY failed at https://bbn-cam-ctrl-1:5001 
    14:57:39 INFO     omni:  ============================================================
    
    Delete - failure
    

Will verify the operations failure fix separately - A unique slice id must be used.