Changes between Initial Version and Version 1 of GENIRacksHome/OpenGENIRacks/PowerDown


Ignore:
Timestamp:
05/23/14 11:36:53 (7 years ago)
Author:
Jeanne Ohren
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • GENIRacksHome/OpenGENIRacks/PowerDown

    v1 v1  
     1[[PageOutline]]
     2
     3= GRAM Rack Power Down Sequence =
     4
     5Before a GRAM rack can be shutdown, GENI experimenters '''__must__''' be notified of the outage window.  To schedule a site outage you can send an email to the GMOC (gmoc@grnoc.iu.edu) or you can schedule an outage by submitting a GMOC ticket [http://gmoc.grnoc.iu.edu/gmoc/support/report-a-problem.html here].  You can also choose to subscribe to the [http://lists.geni.net/mailman/listinfo/response-team GENI Response Team] mailing list to all GENI outage and maintenance notifications from the GMOC.  The GMOC also posts [http://gmoc.grnoc.iu.edu/gmoc/index/support/gmoc-operations-calendars.html notifications] on the GMOC calendar.
     6
     7
     8All GRAM services must be shut down before the rack devices are shutdown. [[BR]]
     9
     10
     11'' Note:'' In the instructions replace instances of ''admin_user'' with the your admin account.   Logging into the Control Node with
     12SSH key you provided and make sure that you replace identity with the correct path name if your SSH keys are not in a standard location.   
     13You can login to the Compute Nodes from the Control Node with SSH keys installed when your account was created.   
     14
     15== Shutting Down GRAM Services  ==
     16
     17 1. Login to the Control Node with your administrative account: For example, type "ssh -Y -i ~/.ssh/id_rsa ''admin_user''@<control addr>"
     18
     19 2. Make sure that no experiments are running and check for VMs that are in an '''ACTIVE''' state:
     20     a.  Type "source /etc/novarc"
     21     b.  Type "nova list --all-tenants".  If all experiment have been stopped you will see no resources listed. In the event that experimenters are still running you will see a list of resources, like this:     
     22        {{{
     23        +--------------------------------------+------+--------+--------------------------------------------+
     24        | ID                                   | Name | Status | Networks                                   |
     25        +--------------------------------------+------+--------+--------------------------------------------+
     26        | b840058e-4511-420c-aaf4-577562b2dce6 | VM-1 | ACTIVE | GRAM-mgmt-net=192.168.10.3; lan0=10.0.37.1 |
     27        | cfa1aa58-e68f-4176-beed-60e9a4257ab3 | VM-2 | ACTIVE | GRAM-mgmt-net=192.168.10.4; lan0=10.0.37.2 |
     28        +--------------------------------------+------+--------+--------------------------------------------+
     29        }}}
     30     c.  You can use the "suspend" command to store the content of the VMs on disk.  Type "nova suspend ''ID''."  For example, "nova suspend b840058e-4511-420c-aaf4-577562b2dce6".   Expected output of "nova list --all-tenants" would then be:
     31        {{{
     32        +--------------------------------------+------+-----------+--------------------------------------------+
     33        | ID                                   | Name | Status    | Networks                                   |
     34        +--------------------------------------+------+-----------+--------------------------------------------+
     35        | b840058e-4511-420c-aaf4-577562b2dce6 | VM-1 | SUSPENDED | GRAM-mgmt-net=192.168.10.3; lan0=10.0.37.1 |
     36        | cfa1aa58-e68f-4176-beed-60e9a4257ab3 | VM-2 | ACTIVE    | GRAM-mgmt-net=192.168.10.4; lan0=10.0.37.2 |
     37        +--------------------------------------+------+-----------+--------------------------------------------+
     38        }}}
     39     d.  Suspended VMs can be resumed on startup.  So, keep track of the IDs that were suspended so that they can be resumed once a rack is back up and operational.
     40
     41         
     42 3. Stop all GRAM processes on the Control Node before shutting down any of the rack devices:
     43        a.  "sudo service gram-am stop"
     44        b.  "sudo service gram-ch stop"
     45        c.  "sudo service gram-ctrl stop"
     46        d.  "sudo service gram-vmoc stop"
     47        e.  "sudo service gram-mon stop"
     48
     49== Shutting Down Rack Devices ==
     50
     51Once the GRAM services have been stopped, you can shutdown the GRAM devices in the following order:
     52     1. !OpenFlow Switch - Dell Force10
     53     2. Compute Nodes
     54     3. Control Node   
     55     4. Management Switch - Dell Powerconnect 7048
     56     5. UPS (if included) 
     57
     58Make sure you follow this order, as you may loose access if the ordering is not followed:
     59
     60 1. Shutting Down Force10 !OpenFlow Switch:         
     61        a. From the control node, type "ssh 10.10.8.200"
     62        b. From the Force 10 console, type "enable"
     63        c. Then, "type reload"
     64        d. And then detach the power cord.
     65
     66 2. Shutting Down Compute Nodes:
     67        a. From the Control Node, type "ssh <hostname>".  The hostnames for the compute nodes can be found in /etc/hosts.[[BR]]
     68        b. Then type "sudo sync; sudo init 0"
     69
     70 3. Shutting Down Control Node:
     71        a. After shutting down all other resources, on the Control Node type "sudo sync; sudo init 0"
     72
     73 4. Shutting Down !PowerConnect 7048:
     74        a. Detach the power cord.
     75
     76 5. Shutting down the UPS (if included)
     77        a. Turn off power switch