Changes between Initial Version and Version 1 of GENIRacksHome/OpenGENIRacks/Monitoring


Ignore:
Timestamp:
05/23/14 11:36:46 (10 years ago)
Author:
Jeanne Ohren
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • GENIRacksHome/OpenGENIRacks/Monitoring

    v1 v1  
     1== Login Access ==
     2
     31. Control and Compute Nodes:   The admin should have an rsa private key for the gram user.
     4          ssh -i ~/.ssh/id_rsa gram@<hostname/IP address>
     5
     62. Force 10 switch:  The switch can only be accessed from the control network, 10.10.8, which means login is only possible from the control or compute nodes. Alternately, use screen on the node that has the serial cable to get access to the console.
     7          ssh gram@10.10.8.200
     8
     93. !PowerConnect 7048:  The switch can only be accessed from the control network, 10.10.8, which means login is only possible from the control or compute nodes. Alternately, use screen on the node that has the serial cable to get access to the console.
     10         ssh gram@10.10.8.201
     11== Monitoring Log ==
     12
     13        There is a gram-mon daemon that puts a log in /var/log/upstart/gram-mon.log.   Relevant information from the gram snapshot is logged in a format for easier readability.  The upstart logging capabilities in terms of log management are utilized by the gram-mon daemon.
     14
     15== Tools ==
     16
     17        The gram-mon.py can be used to translate a vlan, mac, or ip address to relevant slice information.
     18
     19        Usage: 
     20                gram-mon.py -i <ipaddress>[[BR]]
     21                gram-mon.py -m <macaddr>[[BR]]
     22                gram-mon.py -v <vlantag>[[BR]]
     23
     24        Example Output:
     25{{{
     26                               Figure 1
     27
     28
     29> python gram-mon.py -v 2
     30Searching for Slice Information for vlan 2
     31Latest snapshot file: /etc/gram/snapshots/gram/2014_01_16_21_40_14_0.json
     32
     33Diagnostic Information for vlan = 2:
     34        Slice URN : urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+d3
     35        User URN  : urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser
     36        VM UUID   : 7b1c5788-7c3d-4383-af41-709785cf2cc8 on bbn-cam-cmpe-1
     37        NIC UUID  : 88fa651e-efef-4f99-a6a8-4dfc2dceef1f
     38        NIC UUID  : 71eea5fc-d217-4944-99de-31a142ef9aa1
     39        Link UUID : 561b94ed-dd12-45ad-90b7-8780915fd11a
     40        VM UUID   : 69af44a1-9fea-44e1-9086-a351519404b8 on bbn-cam-cmpe-2
     41
     42
     43> python gram-mon.py -i 128.89.72.122
     44Searching for Slice Information for ipaddr 128.89.72.122
     45Latest snapshot file: /etc/gram/snapshots/gram/2014_01_16_21_40_14_0.json
     46
     47Diagnostic Information for ip = 128.89.72.122:
     48        Slice URN : urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+d3
     49        User URN  : urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser
     50        VM UUID   : 7b1c5788-7c3d-4383-af41-709785cf2cc8 on bbn-cam-cmpe-1
     51        NIC UUID  : 88fa651e-efef-4f99-a6a8-4dfc2dceef1f
     52        NIC UUID  : 71eea5fc-d217-4944-99de-31a142ef9aa1
     53        Link UUID : 561b94ed-dd12-45ad-90b7-8780915fd11a
     54        VM UUID   : 69af44a1-9fea-44e1-9086-a351519404b8 on bbn-cam-cmpe-2
     55
     56> python gram-mon.py -m fa:16:3e:8f:ff:78
     57Searching for Slice Information for mac fa:16:3e:8f:ff:78
     58Latest snapshot file: /etc/gram/snapshots/gram/2014_01_16_21_40_14_0.json
     59
     60Diagnostic Information for mac = fa:16:3e:8f:ff:78:
     61        Slice URN : urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+d3
     62        User URN  : urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser
     63        VM UUID   : 7b1c5788-7c3d-4383-af41-709785cf2cc8 on bbn-cam-cmpe-1
     64        NIC UUID  : 88fa651e-efef-4f99-a6a8-4dfc2dceef1f
     65        NIC UUID  : 71eea5fc-d217-4944-99de-31a142ef9aa1
     66        Link UUID : 561b94ed-dd12-45ad-90b7-8780915fd11a
     67        VM UUID   : 69af44a1-9fea-44e1-9086-a351519404b8 on bbn-cam-cmpe-2
     68
     69}}}
     70
     71== Procedures For Shutting Down Resources  ==
     72
     731.  Shutdown Servers:
     74    * Login to the console as an admin user,
     75      * For example for the BBN rack, type "ssh -Y -i ~gramdell_rsa.priv gram@<control addr>"
     76      * Then type "sudo shutdown now"
     77    * Or use the iDRAC
     78      * Find the IP address of your IDRAC which should be off the control network 10.10.8
     79      * From the control node on the rack:[[BR]]
     80      * Type "ssh -i ~/.ssh/id_dsa gram@<IDRAC IP ADDRESS>"
     81      * At the iDRAC command prompt, get into ''racadm'' shell by:
     82      * Typing "racadm" [[BR]
     83      * Then typing "serveraction powerdown"
     84
     85
     862. Shutdown a slice
     87 * If you as the admin have GENI privileges over the slice and you have the sliceName, then you can
     88   * Login to the console as an admin user,
     89   * For example for the BBN rack, "ssh -Y -i ~gramdell_rsa.priv gram@<control addr>"
     90   * Then set up the environment appropriately:
     91     * Type "export PYTHONPATH=/opt/gcf/src:/home/gram/gram/src"
     92     * Type "export PATH=/opt/gcf/src:$PATH"
     93     * Type "omni.py -a <Rack Aggregate URL or Nickname> -V 3 delete <slicename>"
     94
     95 * If you have no affiliation with GENI:  You will need VM UUIDs as shown in Figure 1 above.
     96   * Login to the console as an admin user,
     97   * For example for the BBN rack, "ssh -Y -i ~gramdell_rsa.priv gram@<control addr>"
     98   * Then set up the environment appropriately:
     99     * Type "source /etc/novarc"
     100     * For each VM UUID, type "nova stop <VM UUID>
     101     * The above command should keep the VM around, but if you want to get rid of the VM, type "nova delete <VM UUID>
     102
     1033. Shutdown a port in the network on the data switch assuming you know the port to shutdown.  In the following example, for the BBN rack, the port that is connected to bbn-cam-cmpe-2 is being shutdown based on current rack information.
     104   * Login to the Force10 Switch, by typing "ssh gram@10.10.8.200" and entering the gram password
     105   * Type the following Cisco-like IOS commands:[[BR]]
     106     "enable"[[BR]]
     107     "config"[[BR]]
     108     "interface !TenGigabitEthernet 0/2"[[BR]]
     109     "shutdown"
     110   * Alternately, the openflow instance could be disabled[[BR]]
     111     "enable"[[BR]]
     112     "config"[[BR]]
     113     "openflow of-instance 1"[[BR]]
     114     "shutdown"
     115
     116== New Notes ==
     117
     1181.  Have to configure /home/gram/gram/opsmon/opsmon-config.json   
     119{{{
     120
     121        "base_address" : "https://128.89.118.100",
     122        "aggregate_id" : "gram-bbncam",
     123        "aggregate_urn" : "urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+authority+am"
     124
     125}}}
     126
     127Edit Hosts Information:  Must find CONTROL ethernet interface and DATA ethernet MAC address.   Must also
     128get the correct urn.  What if we don't have an external IP address?
     129{{{
     130
     131 "hosts" : [
     132                {"id" : "bbn-cam-cmpe-1", "address" : "128.89.118.101",
     133                "urn" : "urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+node+bbn-cam-cmpe-1",
     134                "interfaces" : {"eth3" : {"address" : "10.10.8.101",
     135                                 "type" : "IPv4", "role" : "CONTROL", "max_bps" : "1000000000"},
     136                                "eth2" : {"address" : "a0:36:9f:11:0d:60",
     137                                 "type" : "MAC", "role" : "DATA", "max_bps" : "1000000000"}}},
     138                {"id" : "bbn-cam-cmpe-2", "address" : "128.89.118.102",
     139                "urn" : "urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+node+bbn-cam-cmpe-2",
     140                "interfaces" : {"eth3" : {"address" : "10.10.8.102",
     141                                 "type" : "IPv4", "role" : "CONTROL", "max_bps" : "1000000000"},
     142                                "eth2" : {"address" : "a0:36:9f:20:74:cc",
     143                                 "type" : "MAC", "role" : "DATA", "max_bps" : "1000000000"}}}
     144        ],
     145
     146
     147
     148}}}