= GRAM Rack Power Up Sequence =

It is recommended that GRAM rack devices be powered in the following order:

   1.  UPS (if included)
   2.  Management Switch - Dell Powerconnect 7048
   3.  !OpenFlow Switch - Dell Force10
   4.  Control Node
   5.  Compute Nodes 

Make sure that the above sequence is followed in starting the rack devices. If not followed, GRAM services may fail to start.


== Checking GRAM Services ==

Once devices have powered up, the local administrator shoul check the services status and restore any suspended VMs.  This section outlines the produre to be used:

 1. Login to the Control Node as an admin. For example, type "ssh -Y -i ~/.ssh/id_rsa ''admin_user''@<control addr>"
 2. Run the GRAM tool healthcheck.py as directed in the [wiki:GENIRacksHome/OpenGENI/AdministrationTools Administration Tools] page.
 3. If processes have been started properly, you should expected output similar to the following, but specific to your site naming:
 {{{
 gram-mon - running
 gram-ch - running
 checking OpenStack services...
 nova-api - running
 nova-cert - running
 nova-conductor - running
 nova-consoleauth  - running
 nova-novncproxy - running
 nova-scheduler - running
 quantum-dhcp-agent - running
 quantum-metadata-agent - running
 quantum-server - running
 quantum-l3-agent - running
 quantum-plugin-openvswitch-agent - running
 glance-registry - running
 glance-api - running
 keystone - running
WARNING: Management namespace NOT found
Restarting Quantum-L3 service to attempt to recover the namespace - attempt 0
Restarting Quantum-L3 service to attempt to recover the namespace - attempt 1
Restarting Quantum-L3 service to attempt to recover the namespace - attempt 2
Restarting Quantum-L3 service to attempt to recover the namespace - attempt 3
Restarting Quantum-L3 service to attempt to recover the namespace - attempt 4
Restarting Quantum-L3 service to attempt to recover the namespace - attempt 5
Restarting Quantum-L3 service to attempt to recover the namespace - attempt 6
Restarting Quantum-L3 service to attempt to recover the namespace - attempt 7
Restarting Quantum-L3 service to attempt to recover the namespace - attempt 8
Restarting Quantum-L3 service to attempt to recover the namespace - attempt 9
Found management namespace and it matches config
Checking the status of the compute hosts: 
 
 Binary           Host                                 Zone             Status     State Updated_At
 nova-compute     boscompute2                          nova             enabled    :-)   2014-04-15 13:39:58
 nova-compute     boscompute1                          nova             enabled    :-)   2014-04-15 13:39:58
 
 Checking status of Openstack newtworking software modules: 

 +--------------------------------------+--------------------+-----------------------------------+-------+----------------+
 | id                                   | agent_type         | host                              | alive | admin_state_up |
 +--------------------------------------+--------------------+-----------------------------------+-------+----------------+
 | 97c56061-2dc7-4760-90c4-77fffea0e10b | Open vSwitch agent | boscompute1.gram.gpolab.bbn.com   | :-)   | True           |
 | a6b774d4-1e9d-4db3-b7f7-5ed39324598d | DHCP agent         | boscontroller.gram.gpolab.bbn.com | :-)   | True           |
 | c3cfb1ce-1cb0-4651-854c-65a1280a54d5 | L3 agent           | boscontroller.gram.gpolab.bbn.com | :-)   | True           |
 | cae16682-40db-488b-8b43-12812c8618a7 | Open vSwitch agent | boscompute2.gram.gpolab.bbn.com   | :-)   | True           |
 | d2387a09-1767-4798-a754-196bd2e842cc | Open vSwitch agent | boscontroller.gram.gpolab.bbn.com | :-)   | True           |
 +--------------------------------------+--------------------+-----------------------------------+-------+----------------+

 Keystone - pass
 Nova - pass
 Glance - pass
 Quantum - pass
 AM is up : Get-Version succeeded at AM
 Allocate - success
 Provision - success
 Delete - success
 AM is functioning
 }}}
   ''Note: '''AM is functioning''' signifies that the rack is operational.  If the AM is not functioning, the healthcheck script will print out out warnings and attempts to fix them. ''

 4. Resume VMs that were suspended on the powering down of the rack, they must be restored to their original state. For each ID suspended, you should issue a "nova resume ''ID''".  For example, "nova resume b840058e-4511-420c-aaf4-577562b2dce6". So, when typing "nova list --all-tenants", the state of this ID is back to '''ACTIVE'''.

 {{{
 +--------------------------------------+------+--------+--------------------------------------------+
 | ID                                   | Name | Status | Networks                                   |
 +--------------------------------------+------+--------+--------------------------------------------+
 | b840058e-4511-420c-aaf4-577562b2dce6 | VM-1 | ACTIVE | GRAM-mgmt-net=192.168.10.3; lan0=10.0.37.1 |
 | cfa1aa58-e68f-4176-beed-60e9a4257ab3 | VM-2 | ACTIVE | GRAM-mgmt-net=192.168.10.4; lan0=10.0.37.2 |
 +--------------------------------------+------+--------+--------------------------------------------+
 }}}