wiki:GENIRacksHome/OpenGENIRacks/Monitoring

Version 2 (modified by lnevers@bbn.com, 10 years ago) (diff)

--

Login Access

Control and Compute Nodes: The admin should have an SSH RSA private key for the gram user.

          ssh -i ~/.ssh/id_rsa gram@<hostname/IP address>

Force 10 switch: The switch can only be accessed from the control network, 10.10.8, which means login is only possible from the control or compute nodes. Alternately, use screen on the node that has the serial cable to get access to the console.

          ssh gram@10.10.8.200

PowerConnect 7048 switch: The switch can only be accessed from the control network, 10.10.8, which means login is only possible from the control or compute nodes. Alternately, use screen on the node that has the serial cable to get access to the console.

         ssh gram@10.10.8.201

Monitoring Log

There is a gram-mon daemon that creates a log in /var/log/upstart/gram-mon.log. Relevant information from the gram snapshot is logged in a format for easier readability. The upstart logging capabilities in terms of log management are utilized by the gram-mon daemon.

Tools

The gram-mon.py can be used to translate a VLAN, MAC, or IP address to relevant slice information.

Usage:

gram-mon.py -i <ipaddress>
gram-mon.py -m <macaddr>
gram-mon.py -v <vlantag>

Example Output:

                               Figure 1


> python gram-mon.py -v 2
Searching for Slice Information for vlan 2
Latest snapshot file: /etc/gram/snapshots/gram/2014_01_16_21_40_14_0.json

Diagnostic Information for vlan = 2:
        Slice URN : urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+d3
        User URN  : urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser
        VM UUID   : 7b1c5788-7c3d-4383-af41-709785cf2cc8 on bbn-cam-cmpe-1
        NIC UUID  : 88fa651e-efef-4f99-a6a8-4dfc2dceef1f
        NIC UUID  : 71eea5fc-d217-4944-99de-31a142ef9aa1
        Link UUID : 561b94ed-dd12-45ad-90b7-8780915fd11a
        VM UUID   : 69af44a1-9fea-44e1-9086-a351519404b8 on bbn-cam-cmpe-2


> python gram-mon.py -i 128.89.72.122
Searching for Slice Information for ipaddr 128.89.72.122
Latest snapshot file: /etc/gram/snapshots/gram/2014_01_16_21_40_14_0.json

Diagnostic Information for ip = 128.89.72.122:
        Slice URN : urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+d3
        User URN  : urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser
        VM UUID   : 7b1c5788-7c3d-4383-af41-709785cf2cc8 on bbn-cam-cmpe-1
        NIC UUID  : 88fa651e-efef-4f99-a6a8-4dfc2dceef1f
        NIC UUID  : 71eea5fc-d217-4944-99de-31a142ef9aa1
        Link UUID : 561b94ed-dd12-45ad-90b7-8780915fd11a
        VM UUID   : 69af44a1-9fea-44e1-9086-a351519404b8 on bbn-cam-cmpe-2

> python gram-mon.py -m fa:16:3e:8f:ff:78
Searching for Slice Information for mac fa:16:3e:8f:ff:78
Latest snapshot file: /etc/gram/snapshots/gram/2014_01_16_21_40_14_0.json

Diagnostic Information for mac = fa:16:3e:8f:ff:78:
        Slice URN : urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+slice+d3
        User URN  : urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf+user+gramuser
        VM UUID   : 7b1c5788-7c3d-4383-af41-709785cf2cc8 on bbn-cam-cmpe-1
        NIC UUID  : 88fa651e-efef-4f99-a6a8-4dfc2dceef1f
        NIC UUID  : 71eea5fc-d217-4944-99de-31a142ef9aa1
        Link UUID : 561b94ed-dd12-45ad-90b7-8780915fd11a
        VM UUID   : 69af44a1-9fea-44e1-9086-a351519404b8 on bbn-cam-cmpe-2

Procedures For Shutting Down Resources

  1. Shutdown Servers:
    • Login to the console as an admin user,
      • For example for the BBN rack, type "ssh -Y -i ~gramdell_rsa.priv gram@<control addr>"
      • Then type "sudo shutdown now"
    • Or use the iDRAC
      • Find the IP address of your IDRAC which should be off the control network 10.10.8
      • From the control node on the rack:
      • Type "ssh -i ~/.ssh/id_dsa gram@<IDRAC IP ADDRESS>"
      • At the iDRAC command prompt, get into racadm shell by:
      • Typing "racadm" [[BR]
      • Then typing "serveraction powerdown"
  1. Shutdown a slice
    • If you as the admin have GENI privileges over the slice and you have the sliceName, then you can
      • Login to the console as an admin user,
      • For example for the BBN rack, "ssh -Y -i ~gramdell_rsa.priv gram@<control addr>"
      • Then set up the environment appropriately:
        • Type "export PYTHONPATH=/opt/gcf/src:/home/gram/gram/src"
        • Type "export PATH=/opt/gcf/src:$PATH"
        • Type "omni.py -a <Rack Aggregate URL or Nickname> -V 3 delete <slicename>"
  • If you have no affiliation with GENI: You will need VM UUIDs as shown in Figure 1 above.
    • Login to the console as an admin user,
    • For example for the BBN rack, "ssh -Y -i ~gramdell_rsa.priv gram@<control addr>"
    • Then set up the environment appropriately:
      • Type "source /etc/novarc"
      • For each VM UUID, type "nova stop <VM UUID>
      • The above command should keep the VM around, but if you want to get rid of the VM, type "nova delete <VM UUID>
  1. Shutdown a port in the network on the data switch assuming you know the port to shutdown. In the following example, for the BBN rack, the port that is connected to bbn-cam-cmpe-2 is being shutdown based on current rack information.
    • Login to the Force10 Switch, by typing "ssh gram@10.10.8.200" and entering the gram password
    • Type the following Cisco-like IOS commands:
      "enable"
      "config"
      "interface TenGigabitEthernet 0/2"
      "shutdown"
    • Alternately, the openflow instance could be disabled
      "enable"
      "config"
      "openflow of-instance 1"
      "shutdown"

New Notes

  1. Have to configure /home/gram/gram/opsmon/opsmon-config.json
            "base_address" : "https://128.89.118.100",
            "aggregate_id" : "gram-bbncam",
            "aggregate_urn" : "urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+authority+am"
    
    

Edit Hosts Information: Must find CONTROL ethernet interface and DATA ethernet MAC address. Must also get the correct urn. What if we don't have an external IP address?

 "hosts" : [
                {"id" : "bbn-cam-cmpe-1", "address" : "128.89.118.101",
                "urn" : "urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+node+bbn-cam-cmpe-1",
                "interfaces" : {"eth3" : {"address" : "10.10.8.101",
                                 "type" : "IPv4", "role" : "CONTROL", "max_bps" : "1000000000"},
                                "eth2" : {"address" : "a0:36:9f:11:0d:60",
                                 "type" : "MAC", "role" : "DATA", "max_bps" : "1000000000"}}},
                {"id" : "bbn-cam-cmpe-2", "address" : "128.89.118.102",
                "urn" : "urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+node+bbn-cam-cmpe-2",
                "interfaces" : {"eth3" : {"address" : "10.10.8.102",
                                 "type" : "IPv4", "role" : "CONTROL", "max_bps" : "1000000000"},
                                "eth2" : {"address" : "a0:36:9f:20:74:cc",
                                 "type" : "MAC", "role" : "DATA", "max_bps" : "1000000000"}}}
        ],