| 1 | [[PageOutline]] |
| 2 | |
| 3 | = GRAM Rack Power Down Sequence = |
| 4 | |
| 5 | Before a GRAM rack can be shutdown, GENI experimenters '''__must__''' be notified of the outage window. To schedule a site outage you can send an email to the GMOC (gmoc@grnoc.iu.edu) or you can schedule an outage by submitting a GMOC ticket [http://gmoc.grnoc.iu.edu/gmoc/support/report-a-problem.html here]. You can also choose to subscribe to the [http://lists.geni.net/mailman/listinfo/response-team GENI Response Team] mailing list to all GENI outage and maintenance notifications from the GMOC. The GMOC also posts [http://gmoc.grnoc.iu.edu/gmoc/index/support/gmoc-operations-calendars.html notifications] on the GMOC calendar. |
| 6 | |
| 7 | |
| 8 | All GRAM services must be shut down before the rack devices are shutdown. [[BR]] |
| 9 | |
| 10 | |
| 11 | '' Note:'' In the instructions replace instances of ''admin_user'' with the your admin account. Logging into the Control Node with |
| 12 | SSH key you provided and make sure that you replace identity with the correct path name if your SSH keys are not in a standard location. |
| 13 | You can login to the Compute Nodes from the Control Node with SSH keys installed when your account was created. |
| 14 | |
| 15 | == Shutting Down GRAM Services == |
| 16 | |
| 17 | 1. Login to the Control Node with your administrative account: For example, type "ssh -Y -i ~/.ssh/id_rsa ''admin_user''@<control addr>" |
| 18 | |
| 19 | 2. Make sure that no experiments are running and check for VMs that are in an '''ACTIVE''' state: |
| 20 | a. Type "source /etc/novarc" |
| 21 | b. Type "nova list --all-tenants". If all experiment have been stopped you will see no resources listed. In the event that experimenters are still running you will see a list of resources, like this: |
| 22 | {{{ |
| 23 | +--------------------------------------+------+--------+--------------------------------------------+ |
| 24 | | ID | Name | Status | Networks | |
| 25 | +--------------------------------------+------+--------+--------------------------------------------+ |
| 26 | | b840058e-4511-420c-aaf4-577562b2dce6 | VM-1 | ACTIVE | GRAM-mgmt-net=192.168.10.3; lan0=10.0.37.1 | |
| 27 | | cfa1aa58-e68f-4176-beed-60e9a4257ab3 | VM-2 | ACTIVE | GRAM-mgmt-net=192.168.10.4; lan0=10.0.37.2 | |
| 28 | +--------------------------------------+------+--------+--------------------------------------------+ |
| 29 | }}} |
| 30 | c. You can use the "suspend" command to store the content of the VMs on disk. Type "nova suspend ''ID''." For example, "nova suspend b840058e-4511-420c-aaf4-577562b2dce6". Expected output of "nova list --all-tenants" would then be: |
| 31 | {{{ |
| 32 | +--------------------------------------+------+-----------+--------------------------------------------+ |
| 33 | | ID | Name | Status | Networks | |
| 34 | +--------------------------------------+------+-----------+--------------------------------------------+ |
| 35 | | b840058e-4511-420c-aaf4-577562b2dce6 | VM-1 | SUSPENDED | GRAM-mgmt-net=192.168.10.3; lan0=10.0.37.1 | |
| 36 | | cfa1aa58-e68f-4176-beed-60e9a4257ab3 | VM-2 | ACTIVE | GRAM-mgmt-net=192.168.10.4; lan0=10.0.37.2 | |
| 37 | +--------------------------------------+------+-----------+--------------------------------------------+ |
| 38 | }}} |
| 39 | d. Suspended VMs can be resumed on startup. So, keep track of the IDs that were suspended so that they can be resumed once a rack is back up and operational. |
| 40 | |
| 41 | |
| 42 | 3. Stop all GRAM processes on the Control Node before shutting down any of the rack devices: |
| 43 | a. "sudo service gram-am stop" |
| 44 | b. "sudo service gram-ch stop" |
| 45 | c. "sudo service gram-ctrl stop" |
| 46 | d. "sudo service gram-vmoc stop" |
| 47 | e. "sudo service gram-mon stop" |
| 48 | |
| 49 | == Shutting Down Rack Devices == |
| 50 | |
| 51 | Once the GRAM services have been stopped, you can shutdown the GRAM devices in the following order: |
| 52 | 1. !OpenFlow Switch - Dell Force10 |
| 53 | 2. Compute Nodes |
| 54 | 3. Control Node |
| 55 | 4. Management Switch - Dell Powerconnect 7048 |
| 56 | 5. UPS (if included) |
| 57 | |
| 58 | Make sure you follow this order, as you may loose access if the ordering is not followed: |
| 59 | |
| 60 | 1. Shutting Down Force10 !OpenFlow Switch: |
| 61 | a. From the control node, type "ssh 10.10.8.200" |
| 62 | b. From the Force 10 console, type "enable" |
| 63 | c. Then, "type reload" |
| 64 | d. And then detach the power cord. |
| 65 | |
| 66 | 2. Shutting Down Compute Nodes: |
| 67 | a. From the Control Node, type "ssh <hostname>". The hostnames for the compute nodes can be found in /etc/hosts.[[BR]] |
| 68 | b. Then type "sudo sync; sudo init 0" |
| 69 | |
| 70 | 3. Shutting Down Control Node: |
| 71 | a. After shutting down all other resources, on the Control Node type "sudo sync; sudo init 0" |
| 72 | |
| 73 | 4. Shutting Down !PowerConnect 7048: |
| 74 | a. Detach the power cord. |
| 75 | |
| 76 | 5. Shutting down the UPS (if included) |
| 77 | a. Turn off power switch |