wiki:GENIRacksHome/OpenGENIRacks/AcceptanceTestStatusApr2014/OG-MON-1

Version 2 (modified by lnevers@bbn.com, 5 years ago) (diff)

--

OG-MON-1: Control Network Software and VLAN Inspection Test

This page captures status for the test case OG-MON-1. For additional information see the Acceptance Test Status - April 2014 page overall status, or the OpenGENI Acceptance Test Plan for details about the planned evaluation.

Step State Notes Tickets
Step 1 Pass: most criteria git internal, No version code_tag 66, 115
Step 2 Pass
Step 3 Pass


State Legend Description
Pass Test completed and met all criteria
Pass: most criteria Test completed and met most criteria. Exceptions documented
Fail Test completed and failed to meet criteria.
Complete Test completed but will require re-execution due to expected changes
Blocked Blocked by ticketed issue(s).
In Progress Currently under test.

Test Plan Steps

Step 1. Identify processes and versions

A site administrator enumerates processes on each of the server host, the Control VM, the VMOC VM, etc. and an experimental node configured for OpenStack, which listen for network connections from other nodes, identifies what version of what software package is in use for each, and verifies that we know the source of each piece of software and could get access to its source code.

First get supported versions for the AM API and verify access to resources:

$ omni.py getversion -a https://clemson-clemson-control-1.clemson.edu:5001  
18:45:29 INFO     omni: Loading agg_nick_cache file '/home/lnevers/.gcf/agg_nick_cache'
18:45:29 INFO     omni: Loading config file /home/lnevers/.gcf/omni_config
18:45:29 INFO     omni: Setting option useSliceMembers based on omni_config setting
18:45:29 INFO     omni: Using control framework portal
18:45:29 INFO     omni: Member Authority is https://ch.geni.net/MA (from config)
18:45:29 INFO     omni: Slice Authority is https://ch.geni.net/SA (from config)
18:45:30 INFO     omni: AM URN: unspecified_AM_URN (url: https://clemson-clemson-control-1.clemson.edu:5001) has version:
18:45:30 INFO     omni: {   'geni_ad_rspec_versions': [   {   'extensions': [],
                                      'namespace': 'http://www.geni.net/resources/rspec/3',
                                      'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd',
                                      'type': 'GENI',
                                      'version': '3'}],
    'geni_api': 3,
    'geni_api_versions': {   '2': 'https://clemson-clemson-control-1.clemson.edu:5002',
                             '3': 'https://clemson-clemson-control-1.clemson.edu:5001'},
    'geni_credential_types': [{   'geni_type': 'geni_sfa',
                                     'geni_version': '3'}],
    'geni_request_rspec_versions': [   {   'extensions': [],
                                           'namespace': 'http://www.geni.net/resources/rspec/3',
                                           'schema': 'http://www.geni.net/resources/rspec/3/request.xsd',
                                           'type': 'GENI',
                                           'version': '3'}]}
18:45:30 INFO     omni:  ------------------------------------------------------
18:45:30 INFO     omni:  Completed getversion:
Args: getversion

  Result Summary: 
Got version for https://clemson-clemson-control-1.clemson.edu:5001
 
18:45:30 INFO     omni:  ======================================================

There is no aggregate version information (ticket #115)

Then verify that resources are available for experimenters:

$ omni.py listresources -a gram  -o

the following are major services for OpenGENI:

  • gram-am - the aggregate manager that handles AM API calls for resources
  • gram-vnoc - VMOC controller that manages OpenFlow traffic for switch(es) and experimenter controller(s).
  • gram-ctrl- Default controller that performs layer 2 MAC to port learning.
  • gram-mon- Data collection, not sure how it fits in to current plan.

On the Clemson rack, the OpenGENI software is installed in /home/gram/gram on the control node named "clemson-clemson-control-1" (130.127.88.98) and the OpenGENI services listed above are started by scripts in the /home/gram/gram/src/services directory:

$ ls -ltr /home/gram/gram/src/services
total 24
-rwxrwxr-x 1 gram gram 1802 Apr  8 12:21 gram-vmoc.conf
-rwxrwxr-x 1 gram gram 1499 Apr  8 12:21 gram-mon.conf
-rwxrwxr-x 1 gram gram 1898 Apr  8 12:21 gram-ctrl.conf
-rwxrwxr-x 1 gram gram 1514 Apr  8 12:21 gram-cni.conf
-rwxrwxr-x 1 gram gram 1476 Apr  8 12:21 gram-ch.conf
-rwxrwxr-x 1 gram gram 1655 Apr  8 12:21 gram-am.conf

All services were verified to be running:

gram      2428     1  0 Apr08 ?        00:00:00 su -s /bin/sh -c exec python /home/gram/gram/src/monitor/gram-mon.py gram
gram      2561  2428  0 Apr08 ?        00:00:27 python /home/gram/gram/src/monitor/gram-mon.py
gram     17739     1  0 10:41 ?        00:00:00 su -s /bin/sh -c exec python /home/gram/gram/src/gram-am.py gram
gram     17740 17739  0 10:41 ?        00:00:08 python /home/gram/gram/src/gram-am.py
gram     26544 27283  0 19:05 pts/3    00:00:00 egrep --color=auto -i gram-am|gram-mon|vmoc|gram-ch
gram     29434     1  0 13:24 ?        00:00:44 python2.7 -u /opt/pox/pox.py log.level --DEBUG vmoc.VMOC --management_port=7001 --default_controller_url=https://localhost:9000
gram     29970     1  0 13:24 ?        00:00:40 python2.7 -u /opt/pox/pox.py log.level --DEBUG openflow.of_01 --port=9000 vmoc.l2_simple_learning

Check that all processes are listening:

$ sudo netstat -anp | egrep ':\*.+LISTEN' | egrep -v '127.0.0.1' 
tcp        0      0 0.0.0.0:8774            0.0.0.0:*               LISTEN      2455/python     
tcp        0      0 0.0.0.0:8775            0.0.0.0:*               LISTEN      2455/python     
tcp        0      0 0.0.0.0:9191            0.0.0.0:*               LISTEN      2383/python     
tcp        0      0 0.0.0.0:9000            0.0.0.0:*               LISTEN      29970/python2.7 
tcp        0      0 0.0.0.0:5000            0.0.0.0:*               LISTEN      19190/python    
tcp        0      0 0.0.0.0:8776            0.0.0.0:*               LISTEN      2454/python     
tcp        0      0 0.0.0.0:6633            0.0.0.0:*               LISTEN      29434/python2.7 
tcp        0      0 130.127.88.98:5001      0.0.0.0:*               LISTEN      17740/python    
tcp        0      0 130.127.88.98:5002      0.0.0.0:*               LISTEN      17740/python    
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      2637/mysqld     
tcp        0      0 0.0.0.0:36171           0.0.0.0:*               LISTEN      3872/beam.smp   
tcp        0      0 0.0.0.0:9292            0.0.0.0:*               LISTEN      2393/python     
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      4306/apache2    
tcp        0      0 0.0.0.0:4369            0.0.0.0:*               LISTEN      3415/epmd       
tcp        0      0 0.0.0.0:53              0.0.0.0:*               LISTEN      2460/dnsmasq    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      3384/sshd       
tcp        0      0 0.0.0.0:7001            0.0.0.0:*               LISTEN      29434/python2.7 
tcp        0      0 0.0.0.0:3260            0.0.0.0:*               LISTEN      2546/tgtd       
tcp        0      0 0.0.0.0:35357           0.0.0.0:*               LISTEN      19190/python    
tcp        0      0 130.127.88.98:8000      0.0.0.0:*               LISTEN      2094/python     
tcp        0      0 0.0.0.0:9696            0.0.0.0:*               LISTEN      2443/python     
tcp        0      0 0.0.0.0:6080            0.0.0.0:*               LISTEN      2479/python     
tcp        0      0 0.0.0.0:8773            0.0.0.0:*               LISTEN      2455/python     
tcp6       0      0 :::5672                 :::*                    LISTEN      3872/beam.smp   
tcp6       0      0 :::53                   :::*                    LISTEN      2460/dnsmasq    
tcp6       0      0 :::22                   :::*                    LISTEN      3384/sshd       
tcp6       0      0 :::3260                 :::*                    LISTEN      2546/tgtd       

Stripping list of processes to get a list of PIDs, and determine what the processes that are listening:

$ sudo netstat -anp | egrep ':\*.+LISTEN' | egrep -v '127.0.0.1' >netstat-raw.txt
$ awk '{print $NF}' netstat-raw.txt | sed -r -e 's|/.*||' | sort -n | uniq > netstat-pids.txt
$ for pid in $(cat netstat-pids.txt) ; do ps -efwww | egrep "^[^ ]+ +$pid" ; done >& netstat-commands.txt
$ lnevers@bbn-cam-ctrl-1:~$ cat netstat-commands.txt
gram      2094  2093  0 Apr08 ?        00:00:10 python /opt/gcf/src/gcf-ch.py
glance    2383     1  0 Apr08 ?        00:00:00 /usr/bin/python /usr/bin/glance-registry
glance    2393     1  0 Apr08 ?        00:00:00 /usr/bin/python /usr/bin/glance-api
quantum   2443     1  0 Apr08 ?        00:09:09 python /usr/bin/quantum-server --config-file /etc/quantum/quantum.conf --log-file /var/log/quantum/server.log --config-file /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini
cinder    2454     1  0 Apr08 ?        00:00:00 /usr/bin/python /usr/bin/cinder-api --config-file=/etc/cinder/cinder.conf --log-file=/var/log/cinder/cinder-api.log
nova      2455     1  0 Apr08 ?        00:00:00 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf
dnsmasq   2460     1  0 Apr08 ?        00:00:00 /usr/sbin/dnsmasq -x /var/run/dnsmasq/dnsmasq.pid -u dnsmasq -r /var/run/dnsmasq/resolv.conf -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new
nova      2479     1  0 Apr08 ?        00:00:13 /usr/bin/python /usr/bin/nova-novncproxy --config-file=/etc/nova/nova.conf
root      2546     1  0 Apr08 ?        00:00:07 tgtd
mysql     2637     1 39 Apr08 ?        10:56:03 /usr/sbin/mysqld
root      3384     1  0 Apr08 ?        00:00:00 /usr/sbin/sshd -D
rabbitmq  3415     1  0 Apr08 ?        00:00:00 /usr/lib/erlang/erts-5.8.5/bin/epmd -daemon
rabbitmq  3872     1  1 Apr08 ?        00:29:47 /usr/lib/erlang/erts-5.8.5/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -noshell -noinput -sname rabbit@clemson-clemson-control-1 -boot /var/lib/rabbitmq/mnesia/rabbit@clemson-clemson-control-1-plugins-expand/rabbit -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@clemson-clemson-control-1.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@clemson-clemson-control-1-sasl.log"} -os_mon start_cpu_sup true -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@clemson-clemson-control-1"
root      4306     1  0 Apr08 ?        00:00:02 /usr/sbin/apache2 -k start
gram     17740 17739  0 10:41 ?        00:00:08 python /home/gram/gram/src/gram-am.py
keystone 19190     1  0 06:30 ?        00:02:45 /usr/bin/python /usr/bin/keystone-all
gram     29434     1  0 13:24 ?        00:00:44 python2.7 -u /opt/pox/pox.py log.level --DEBUG vmoc.VMOC --management_port=7001 --default_controller_url=https://localhost:9000
gram     29970     1  0 13:24 ?        00:00:40 python2.7 -u /opt/pox/pox.py log.level --DEBUG openflow.of_01 --port=9000 vmoc.l2_simple_learning

Additionally OpenGENI software configuration is captured on the control node:

$ more  /etc/gram/config.json 
{
   "__001": "This file contains OpenGENI installation-specific", 
   "__002": "parameters and structures. The values contained here",
   "__003": "override values in the gram.am.gram.config module.",
   "__004": "That is, a key/value pair  K, V  in this file", 
   "__005": "sets gram.am.config.K = V",
   "__006": "An error is given if the field (K) does not exist, ",
   "__007": "Or the type of the value cannot be coerced",
   "__008": "Further any parameter whose value is None in config.py", 
   "__009": "and set as null here must be set in this file:",
   "__010": "no default is provided",

   "default_VM_flavor": "m1.small",
   "default_OS_image": "ubuntu-12.04",
   "default_OS_type": "Linux",
   "default_OS_version": "12",

   "aggregate_id" : "urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+authority+am",

   "disk_image_metadata": 
   {
   "ubuntu-12.04":
       {
        "os": "Linux",
        "version": "12.0",
        "description":" Standard Ubuntu"
        },
   "fedora-19":
       {
        "os": "Linux",
        "version": "19.0",
        "description":"Standard Fedora"
        }
   },

   "external_interface": "ext",
   "external_address": "130.127.88.98",
   "external_netmask": "255.255.255.224",

   "control_address": "10.10.8.100",
   "control_interface": "ctl",
   "data_interface": "dat",
   "data_address": "10.10.5.100",
   "management_interface": "mgt",
   "management_address": "10.10.6.100",
   "management_network_name": "GRAM-mgmt-net",
   "management_network_cidr": "192.168.10.0/24", 
   "management_network_vlan": "2500", 
   "mysql_user": "admin",
   "mysql_password": "os_pwd",
   "rabbit_password": "guest",
   "nova_password": "os_pwd",
   "glance_password": "os_pwd",
   "keystone_password": "os_pwd",
   "quantum_password": "os_pwd",
   "os_tenant_name": "admin",
   "os_username": "admin",
   "os_password": "admin_pass",
   "os_auth_url": "http://localhost:5000/v2.0/",
   "os_region_name" : "RegionOne", 
   "os_no_cache" : "1",
   "service_token" : "clemson-clemson-control-1",
   "service_endpoint" : "http://localhost:35357/v2.0",

   "public_gateway_ip" : "130.127.88.97",
   "public_dns_nameservers" : "130.127.255.250 130.127.255.251",
   "public_subnet_cidr" : "130.127.88.96/27",
   "public_subnet_start_ip" : "130.127.88.101",
   "public_subnet_end_ip" : "130.127.88.110",

   "metadata_port" : "8775",

   "backup_directory" : "/home/gram/.backup",
   "control_host" : "clemson-clemson-control-1.clemson.edu",
   "control_host_addr" : "10.10.8.100",
   "compute_hosts": {
		"clemson-clemson-compute-1":"10.10.8.101",
		"clemson-clemson-compute-2":"10.10.8.102"},
   "glance_images" : [],
   "internal_vlans": "1001-2000",
   "host_file_entries": {
		"clemson-clemson-control-1":"130.127.88.98",
		"clemson-clemson-compute-1":"130.127.88.99",
		"clemson-clemson-compute-2":"130.127.88.100"},
   "stitching_info" : {
       "aggregate_id" : "urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+authority+am",
        "aggregate_url" : "https://clemson-clemson-control-1.clemson.edu:5001",
        "edge_points" : [
                {
                "local_switch" : "urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+node+procurve2",
                "port" : "urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+stitchport+procurve2:16",
                "local_link" : "urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+interface+procurve2:16",
                "remote_switch": "urn:publicid:IDN+ion.internet2.edu+interface+rtr.atla:xe-0/3/0:clemson-gr",
                "vlans" : "3000-3009",
                "traffic_engineering_metric" : "22",
                "capacity": "1000000",
                "stitching_granularity":"1",
                "maximum_reservable_capacity": "1000000",
                "minimum_reservable_capacity": "1000",
                "interface_mtu": "1500"
                }
        ]
   },
   "public_ip": "130.127.88.101",
   "mgmt_ns": "qrouter-a03127ba-7184-42e6-bd5d-2e5957193b86"
}

GRAM GIT Repository was successfully accessed internally:

LNM:~$  git clone git@superior.bbn.com:gram.git
Cloning into 'gram'...
remote: Counting objects: 12215, done.
remote: Compressing objects: 100% (3988/3988), done.
remote: Total 12215 (delta 8852), reused 11130 (delta 8026)
Receiving objects: 100% (12215/12215), 470.86 MiB | 441.00 KiB/s, done.
Resolving deltas: 100% (8852/8852), done.
Checking connectivity... done

There is no BBN-external access to the GRAM Repository - ticket #66

GRAM Software version installed on control node:

$ dpkg -p gram
Package: gram
Priority: optional
Maintainer: GRAM Developers <gram-dev@bbn.com>
Architecture: all
Version: 3.32
Depends: git, module-assistant, cloud-init, xmlsec1, ntp, vlan, bridge-utils
Size: 2195036
Description: GRAM installation package

Step 2. Review control network configuration

A site administrator reviews the configuration of the rack control plane switch and verifies that each experimental node's control and console access interfaces are on the expected VLANs. According to Details on Clemson OpenGENI Rack Installation and Configuration document interfaces are configured as follows on the OpenGENI nodes:

  • eth0: Control network configured for 10.10.8.0
  • eth1 : Data plane network configured for 10.10.5.0
  • eth3 : Management network configured for 10.10.6.0

Subnet masks for network are not listed.

Control Node

The following interfaces are configured on the control node (clemson-clemson-control-1/130.127.88.98):

gram@clemson-clemson-control-1:~$ /sbin/ifconfig
br-ex     Link encap:Ethernet  HWaddr a0:36:9f:11:0a:b2  
          inet addr:130.127.88.98  Bcast:130.127.88.127  Mask:255.255.255.224
          inet6 addr: fe80::a236:9fff:fe11:ab2/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1767593 errors:0 dropped:329916 overruns:0 frame:0
          TX packets:2647553 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:286297387 (286.2 MB)  TX bytes:2530830293 (2.5 GB)

ctl       Link encap:Ethernet  HWaddr 90:b1:1c:0f:d6:49  
          inet addr:10.10.8.100  Bcast:10.10.8.255  Mask:255.255.255.0
          inet6 addr: fe80::92b1:1cff:fe0f:d649/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:7567719 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6985519 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1624052195 (1.6 GB)  TX bytes:3613462297 (3.6 GB)
          Interrupt:37 Memory:d4000000-d47fffff 

dat       Link encap:Ethernet  HWaddr a0:36:9f:11:0a:b0  
          inet addr:10.10.5.100  Bcast:10.10.5.255  Mask:255.255.255.0
          inet6 addr: fe80::a236:9fff:fe11:ab0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1559187 errors:0 dropped:17 overruns:0 frame:0
          TX packets:399137 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:149773816 (149.7 MB)  TX bytes:81043906 (81.0 MB)

ext       Link encap:Ethernet  HWaddr a0:36:9f:11:0a:b2  
          inet6 addr: fe80::a236:9fff:fe11:ab2/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:2933086 errors:0 dropped:265 overruns:0 frame:0
          TX packets:4313453 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1413216222 (1.4 GB)  TX bytes:2647059093 (2.6 GB)

int-br-dat Link encap:Ethernet  HWaddr ca:dc:26:eb:65:79  
          inet6 addr: fe80::c8dc:26ff:feeb:6579/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1393949 errors:0 dropped:0 overruns:0 frame:0
          TX packets:404422 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:141213388 (141.2 MB)  TX bytes:81611296 (81.6 MB)

int-br-mgt Link encap:Ethernet  HWaddr be:da:ce:fe:46:ff  
          inet6 addr: fe80::bcda:ceff:fefe:46ff/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:578605 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1065223 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:77999612 (77.9 MB)  TX bytes:1155087194 (1.1 GB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:36888536 errors:0 dropped:0 overruns:0 frame:0
          TX packets:36888536 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:11093262695 (11.0 GB)  TX bytes:11093262695 (11.0 GB)

mgt       Link encap:Ethernet  HWaddr 90:b1:1c:0f:d6:47  
          inet addr:10.10.6.100  Bcast:10.10.6.255  Mask:255.255.255.0
          inet6 addr: fe80::92b1:1cff:fe0f:d647/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:930178 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1061053 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:107818204 (107.8 MB)  TX bytes:1163036100 (1.1 GB)
          Interrupt:36 Memory:d3000000-d37fffff 

phy-br-dat Link encap:Ethernet  HWaddr 72:8f:86:3b:36:84  
          inet6 addr: fe80::708f:86ff:fe3b:3684/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:404422 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1393949 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:81611296 (81.6 MB)  TX bytes:141213388 (141.2 MB)

phy-br-mgt Link encap:Ethernet  HWaddr 06:d7:7e:68:95:8a  
          inet6 addr: fe80::4d7:7eff:fe68:958a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1065223 errors:0 dropped:0 overruns:0 frame:0
          TX packets:578605 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1155087194 (1.1 GB)  TX bytes:77999612 (77.9 MB)

gram@clemson-clemson-control-1:~$ 

All connections on Control Node:

gram@clemson-clemson-control-1:~$ sockstat
USER     PROCESS              PID      PROTO  SOURCE ADDRESS            FOREIGN ADDRESS           STATE
gram     python               4835     tcp4   130.127.88.98:8000        *:*                       LISTEN
gram     python               5428     tcp4   130.127.88.98:5002        *:*                       LISTEN
gram     python               5428     tcp4   130.127.88.98:5001        *:*                       LISTEN
gram     python2.7            23275    tcp4   *:6633                    *:*                       LISTEN
gram     python2.7            23275    tcp4   10.10.8.100:6633          10.10.8.200:55511         ESTABLISHED
gram     python2.7            23275    tcp4   *:7001                    *:*                       LISTEN
gram     python2.7            23275    tcp4   130.127.88.98:36894       192.1.249.185:33019       ESTABLISHED
gram     python2.7            23275    tcp4   130.127.88.98:52614       130.127.88.98:9000        ESTABLISHED
gram     python2.7            23275    tcp4   130.127.88.98:52615       130.127.88.98:9000        ESTABLISHED
gram     python2.7            23275    tcp4   127.0.0.1:33549           127.0.0.1:9000            ESTABLISHED
gram     python2.7            32360    tcp4   *:9000                    *:*                       LISTEN
gram     python2.7            32360    tcp4   127.0.0.1:9000            127.0.0.1:33549           ESTABLISHED
gram     python2.7            32360    tcp4   130.127.88.98:9000        130.127.88.98:52614       ESTABLISHED
gram     python2.7            32360    tcp4   130.127.88.98:9000        130.127.88.98:52615       ESTABLISHED

All Know connection are explained: {{ gram@clemson-clemson-control-1:~$ for i in sockstat | awk '{print $3}' | grep -v PID | sort -u; do ps -eaf|egrep $i|egrep -v grep; done gram 23275 1 0 Apr19 ? 00:05:53 python2.7 -u -O /opt/pox/pox.py log.level --DEBUG vmoc.VMOC --management_port=7001 --default_controller_url=https://localhost:9000 gram 32360 1 0 14:10 ? 00:00:19 python2.7 -u -O /opt/pox/pox.py log.level --DEBUG openflow.of_01 --port=9000 vmoc.l2_simple_learning gram 4835 4828 0 Apr21 ? 00:00:12 python /opt/gcf/src/gcf-ch.py gram 5428 5421 0 Apr21 ? 00:00:35 python /home/gram/gram/src/gram-am.py gram@clemson-clemson-control-1:~$ for i in sockstat | awk '{print $3}' | grep -v PID | sort -u; do ps -eaf|egrep $i|egrep -v grep; done gram 23275 1 0 Apr19 ? 00:05:53 python2.7 -u -O /opt/pox/pox.py log.level --DEBUG vmoc.VMOC --management_port=7001 --default_controller_url=https://localhost:9000 gram 32360 1 0 14:10 ? 00:00:19 python2.7 -u -O /opt/pox/pox.py log.level --DEBUG openflow.of_01 --port=9000 vmoc.l2_simple_learning gram 4835 4828 0 Apr21 ? 00:00:12 python /opt/gcf/src/gcf-ch.py gram 5428 5421 0 Apr21 ? 00:00:35 python /home/gram/gram/src/gram-am.py

}}}

Compute Nodes

Information for host clemson-clemson-compute-1 (130.127.88.99):

  System information as of Tue Apr 22 18:53:57 EDT 2014

  System load:  0.05               Users logged in:      0
  Usage of /:   1.3% of 297.06GB   IP address for dat:   10.10.5.101
  Memory usage: 1%                 IP address for mgt:   10.10.6.101
  Swap usage:   0%                 IP address for ctl:   10.10.8.101
  Processes:    254                IP address for br-ex: 130.127.88.99

There are no connections:

gram@clemson-clemson-compute-1:~$ sockstat
USER     PROCESS              PID      PROTO  SOURCE ADDRESS            FOREIGN ADDRESS           STATE
gram@clemson-clemson-compute-1:~$ 

Information for host clemson-clemson-compute-2 (130.127.88.100):

  System information as of Tue Apr 22 18:56:21 EDT 2014

  System load:  0.0                Users logged in:      0
  Usage of /:   1.4% of 297.06GB   IP address for dat:   10.10.5.102
  Memory usage: 1%                 IP address for mgt:   10.10.6.102
  Swap usage:   0%                 IP address for ctl:   10.10.8.102
  Processes:    257                IP address for br-ex: 130.127.88.100

There are no active connections:

gram@clemson-clemson-compute-2:~$ sockstat
USER     PROCESS              PID      PROTO  SOURCE ADDRESS            FOREIGN ADDRESS           STATE
gram@clemson-clemson-compute-2:~$ 

Step3. Review MAC address

On Control Node get list of MAC addresses:

gram@clemson-clemson-control-1:~$ arp -a
? (10.10.8.102) at 90:b1:1c:0f:f4:79 [ether] on ctl
? (10.10.8.101) at 90:b1:1c:0f:8a:79 [ether] on ctl
? (10.10.8.201) at d0:67:e5:a9:b1:9f [ether] on ctl
? (130.127.88.97) at 00:23:5e:a9:5c:ca [ether] on br-ex
? (10.10.8.200) at 00:01:e8:d7:58:9d [ether] on ctl
gram@clemson-clemson-control-1:~$ 

On Compute Node 1:

gram@clemson-clemson-compute-1:~$ arp -a
clemson-clemson-control-1 (10.10.8.100) at 90:b1:1c:0f:d6:49 [ether] on ctl
? (130.127.88.97) at 00:23:5e:a9:5c:ca [ether] on br-ex
gram@clemson-clemson-compute-1:~$ 

On Compute Node 2:

gram@clemson-clemson-compute-2:~$ arp -a
? (130.127.88.97) at 00:23:5e:a9:5c:ca [ether] on br-ex
clemson-clemson-control-1 (10.10.8.100) at 90:b1:1c:0f:d6:49 [ether] on ctl
gram@clemson-clemson-compute-2:~$ 

Verified MAC addresses for the Control Network interfaces on the switch.