= OG-MON-1: Control Network Software and VLAN Inspection Test = This page captures status for the test case OG-MON-1. For additional information see the [wiki:GENIRacksHome/OpenGENIRacks/AcceptanceTestStatusApr2014 Acceptance Test Status - April 2014] page overall status, or the [wiki:GENIRacksHome/AcceptanceTests/OpenGENIAcceptanceTestsPlan OpenGENI Acceptance Test Plan] for details about the planned evaluation. || '''Step''' || '''State'''||''' Notes ''' || '''Tickets''' || || Step 1 ||[[span(Pass: most criteria, style=background-color: #98FB98)]]|| git internal, ~~No version code_tag~~ || 66, ~~115~~ || || Step 2 ||[[span(Pass, style=background-color: green )]] || || || || Step 3 ||[[span(Pass, style=background-color: green )]] || || || [[BR]] || '''State Legend''' || '''Description''' || ||[[span(Pass, style=background-color: green )]] || Test completed and met all criteria || ||[[span(Pass: most criteria, style=background-color: #98FB98)]]|| Test completed and met most criteria. Exceptions documented || ||[[span(Fail, style=background-color: red)]] || Test completed and failed to meet criteria. || ||[[span(Complete, style=background-color: yellow)]] || Test completed but will require re-execution due to expected changes || ||[[span(Blocked, style=background-color: orange)]] || Blocked by ticketed issue(s). || ||[[span(In Progress, style=background-color: #63B8FF)]]|| Currently under test. || = Test Plan Steps = == Step 1. Identify processes and versions == A site administrator enumerates processes on each of the server host, the Control VM, the VMOC VM, etc. and an experimental node configured for OpenStack, which listen for network connections from other nodes, identifies what version of what software package is in use for each, and verifies that we know the source of each piece of software and could get access to its source code. First get supported versions for the AM API and verify access to resources: {{{ $ omni.py getversion -a https://clemson-clemson-control-1.clemson.edu:5001 18:45:29 INFO omni: Loading agg_nick_cache file '/home/lnevers/.gcf/agg_nick_cache' 18:45:29 INFO omni: Loading config file /home/lnevers/.gcf/omni_config 18:45:29 INFO omni: Setting option useSliceMembers based on omni_config setting 18:45:29 INFO omni: Using control framework portal 18:45:29 INFO omni: Member Authority is https://ch.geni.net/MA (from config) 18:45:29 INFO omni: Slice Authority is https://ch.geni.net/SA (from config) 18:45:30 INFO omni: AM URN: unspecified_AM_URN (url: https://clemson-clemson-control-1.clemson.edu:5001) has version: 18:45:30 INFO omni: { 'geni_ad_rspec_versions': [ { 'extensions': [], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd', 'type': 'GENI', 'version': '3'}], 'geni_api': 3, 'geni_api_versions': { '2': 'https://clemson-clemson-control-1.clemson.edu:5002', '3': 'https://clemson-clemson-control-1.clemson.edu:5001'}, 'geni_credential_types': [{ 'geni_type': 'geni_sfa', 'geni_version': '3'}], 'geni_request_rspec_versions': [ { 'extensions': [], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/request.xsd', 'type': 'GENI', 'version': '3'}]} 18:45:30 INFO omni: ------------------------------------------------------ 18:45:30 INFO omni: Completed getversion: Args: getversion Result Summary: Got version for https://clemson-clemson-control-1.clemson.edu:5001 18:45:30 INFO omni: ====================================================== }}} There is no aggregate version information (ticket #115) Then verify that resources are available for experimenters: {{{ $ omni.py listresources -a gram -o }}} the following are major services for OpenGENI: - ''gram-am'' - the aggregate manager that handles AM API calls for resources - ''gram-vnoc'' - VMOC controller that manages OpenFlow traffic for switch(es) and experimenter controller(s). - ''gram-ctrl''- Default controller that performs layer 2 MAC to port learning. - ''gram-mon''- Data collection, not sure how it fits in to current plan. On the Clemson rack, the OpenGENI software is installed in /home/gram/gram on the control node named "clemson-clemson-control-1" (130.127.88.98) and the OpenGENI services listed above are started by scripts in the /home/gram/gram/src/services directory: {{{ $ ls -ltr /home/gram/gram/src/services total 24 -rwxrwxr-x 1 gram gram 1802 Apr 8 12:21 gram-vmoc.conf -rwxrwxr-x 1 gram gram 1499 Apr 8 12:21 gram-mon.conf -rwxrwxr-x 1 gram gram 1898 Apr 8 12:21 gram-ctrl.conf -rwxrwxr-x 1 gram gram 1514 Apr 8 12:21 gram-cni.conf -rwxrwxr-x 1 gram gram 1476 Apr 8 12:21 gram-ch.conf -rwxrwxr-x 1 gram gram 1655 Apr 8 12:21 gram-am.conf }}} All services were verified to be running: {{{ gram 2428 1 0 Apr08 ? 00:00:00 su -s /bin/sh -c exec python /home/gram/gram/src/monitor/gram-mon.py gram gram 2561 2428 0 Apr08 ? 00:00:27 python /home/gram/gram/src/monitor/gram-mon.py gram 17739 1 0 10:41 ? 00:00:00 su -s /bin/sh -c exec python /home/gram/gram/src/gram-am.py gram gram 17740 17739 0 10:41 ? 00:00:08 python /home/gram/gram/src/gram-am.py gram 26544 27283 0 19:05 pts/3 00:00:00 egrep --color=auto -i gram-am|gram-mon|vmoc|gram-ch gram 29434 1 0 13:24 ? 00:00:44 python2.7 -u /opt/pox/pox.py log.level --DEBUG vmoc.VMOC --management_port=7001 --default_controller_url=https://localhost:9000 gram 29970 1 0 13:24 ? 00:00:40 python2.7 -u /opt/pox/pox.py log.level --DEBUG openflow.of_01 --port=9000 vmoc.l2_simple_learning }}} Check that all processes are listening: {{{ $ sudo netstat -anp | egrep ':\*.+LISTEN' | egrep -v '127.0.0.1' tcp 0 0 0.0.0.0:8774 0.0.0.0:* LISTEN 2455/python tcp 0 0 0.0.0.0:8775 0.0.0.0:* LISTEN 2455/python tcp 0 0 0.0.0.0:9191 0.0.0.0:* LISTEN 2383/python tcp 0 0 0.0.0.0:9000 0.0.0.0:* LISTEN 29970/python2.7 tcp 0 0 0.0.0.0:5000 0.0.0.0:* LISTEN 19190/python tcp 0 0 0.0.0.0:8776 0.0.0.0:* LISTEN 2454/python tcp 0 0 0.0.0.0:6633 0.0.0.0:* LISTEN 29434/python2.7 tcp 0 0 130.127.88.98:5001 0.0.0.0:* LISTEN 17740/python tcp 0 0 130.127.88.98:5002 0.0.0.0:* LISTEN 17740/python tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 2637/mysqld tcp 0 0 0.0.0.0:36171 0.0.0.0:* LISTEN 3872/beam.smp tcp 0 0 0.0.0.0:9292 0.0.0.0:* LISTEN 2393/python tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 4306/apache2 tcp 0 0 0.0.0.0:4369 0.0.0.0:* LISTEN 3415/epmd tcp 0 0 0.0.0.0:53 0.0.0.0:* LISTEN 2460/dnsmasq tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 3384/sshd tcp 0 0 0.0.0.0:7001 0.0.0.0:* LISTEN 29434/python2.7 tcp 0 0 0.0.0.0:3260 0.0.0.0:* LISTEN 2546/tgtd tcp 0 0 0.0.0.0:35357 0.0.0.0:* LISTEN 19190/python tcp 0 0 130.127.88.98:8000 0.0.0.0:* LISTEN 2094/python tcp 0 0 0.0.0.0:9696 0.0.0.0:* LISTEN 2443/python tcp 0 0 0.0.0.0:6080 0.0.0.0:* LISTEN 2479/python tcp 0 0 0.0.0.0:8773 0.0.0.0:* LISTEN 2455/python tcp6 0 0 :::5672 :::* LISTEN 3872/beam.smp tcp6 0 0 :::53 :::* LISTEN 2460/dnsmasq tcp6 0 0 :::22 :::* LISTEN 3384/sshd tcp6 0 0 :::3260 :::* LISTEN 2546/tgtd }}} Stripping list of processes to get a list of PIDs, and determine what the processes that are listening: {{{ $ sudo netstat -anp | egrep ':\*.+LISTEN' | egrep -v '127.0.0.1' >netstat-raw.txt $ awk '{print $NF}' netstat-raw.txt | sed -r -e 's|/.*||' | sort -n | uniq > netstat-pids.txt $ for pid in $(cat netstat-pids.txt) ; do ps -efwww | egrep "^[^ ]+ +$pid" ; done >& netstat-commands.txt $ lnevers@bbn-cam-ctrl-1:~$ cat netstat-commands.txt gram 2094 2093 0 Apr08 ? 00:00:10 python /opt/gcf/src/gcf-ch.py glance 2383 1 0 Apr08 ? 00:00:00 /usr/bin/python /usr/bin/glance-registry glance 2393 1 0 Apr08 ? 00:00:00 /usr/bin/python /usr/bin/glance-api quantum 2443 1 0 Apr08 ? 00:09:09 python /usr/bin/quantum-server --config-file /etc/quantum/quantum.conf --log-file /var/log/quantum/server.log --config-file /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini cinder 2454 1 0 Apr08 ? 00:00:00 /usr/bin/python /usr/bin/cinder-api --config-file=/etc/cinder/cinder.conf --log-file=/var/log/cinder/cinder-api.log nova 2455 1 0 Apr08 ? 00:00:00 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf dnsmasq 2460 1 0 Apr08 ? 00:00:00 /usr/sbin/dnsmasq -x /var/run/dnsmasq/dnsmasq.pid -u dnsmasq -r /var/run/dnsmasq/resolv.conf -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new nova 2479 1 0 Apr08 ? 00:00:13 /usr/bin/python /usr/bin/nova-novncproxy --config-file=/etc/nova/nova.conf root 2546 1 0 Apr08 ? 00:00:07 tgtd mysql 2637 1 39 Apr08 ? 10:56:03 /usr/sbin/mysqld root 3384 1 0 Apr08 ? 00:00:00 /usr/sbin/sshd -D rabbitmq 3415 1 0 Apr08 ? 00:00:00 /usr/lib/erlang/erts-5.8.5/bin/epmd -daemon rabbitmq 3872 1 1 Apr08 ? 00:29:47 /usr/lib/erlang/erts-5.8.5/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -noshell -noinput -sname rabbit@clemson-clemson-control-1 -boot /var/lib/rabbitmq/mnesia/rabbit@clemson-clemson-control-1-plugins-expand/rabbit -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@clemson-clemson-control-1.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@clemson-clemson-control-1-sasl.log"} -os_mon start_cpu_sup true -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@clemson-clemson-control-1" root 4306 1 0 Apr08 ? 00:00:02 /usr/sbin/apache2 -k start gram 17740 17739 0 10:41 ? 00:00:08 python /home/gram/gram/src/gram-am.py keystone 19190 1 0 06:30 ? 00:02:45 /usr/bin/python /usr/bin/keystone-all gram 29434 1 0 13:24 ? 00:00:44 python2.7 -u /opt/pox/pox.py log.level --DEBUG vmoc.VMOC --management_port=7001 --default_controller_url=https://localhost:9000 gram 29970 1 0 13:24 ? 00:00:40 python2.7 -u /opt/pox/pox.py log.level --DEBUG openflow.of_01 --port=9000 vmoc.l2_simple_learning }}} Additionally OpenGENI software configuration is captured on the control node: {{{ $ more /etc/gram/config.json { "__001": "This file contains OpenGENI installation-specific", "__002": "parameters and structures. The values contained here", "__003": "override values in the gram.am.gram.config module.", "__004": "That is, a key/value pair K, V in this file", "__005": "sets gram.am.config.K = V", "__006": "An error is given if the field (K) does not exist, ", "__007": "Or the type of the value cannot be coerced", "__008": "Further any parameter whose value is None in config.py", "__009": "and set as null here must be set in this file:", "__010": "no default is provided", "default_VM_flavor": "m1.small", "default_OS_image": "ubuntu-12.04", "default_OS_type": "Linux", "default_OS_version": "12", "aggregate_id" : "urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+authority+am", "disk_image_metadata": { "ubuntu-12.04": { "os": "Linux", "version": "12.0", "description":" Standard Ubuntu" }, "fedora-19": { "os": "Linux", "version": "19.0", "description":"Standard Fedora" } }, "external_interface": "ext", "external_address": "130.127.88.98", "external_netmask": "255.255.255.224", "control_address": "10.10.8.100", "control_interface": "ctl", "data_interface": "dat", "data_address": "10.10.5.100", "management_interface": "mgt", "management_address": "10.10.6.100", "management_network_name": "GRAM-mgmt-net", "management_network_cidr": "192.168.10.0/24", "management_network_vlan": "2500", "mysql_user": "admin", "mysql_password": "os_pwd", "rabbit_password": "guest", "nova_password": "os_pwd", "glance_password": "os_pwd", "keystone_password": "os_pwd", "quantum_password": "os_pwd", "os_tenant_name": "admin", "os_username": "admin", "os_password": "admin_pass", "os_auth_url": "http://localhost:5000/v2.0/", "os_region_name" : "RegionOne", "os_no_cache" : "1", "service_token" : "clemson-clemson-control-1", "service_endpoint" : "http://localhost:35357/v2.0", "public_gateway_ip" : "130.127.88.97", "public_dns_nameservers" : "130.127.255.250 130.127.255.251", "public_subnet_cidr" : "130.127.88.96/27", "public_subnet_start_ip" : "130.127.88.101", "public_subnet_end_ip" : "130.127.88.110", "metadata_port" : "8775", "backup_directory" : "/home/gram/.backup", "control_host" : "clemson-clemson-control-1.clemson.edu", "control_host_addr" : "10.10.8.100", "compute_hosts": { "clemson-clemson-compute-1":"10.10.8.101", "clemson-clemson-compute-2":"10.10.8.102"}, "glance_images" : [], "internal_vlans": "1001-2000", "host_file_entries": { "clemson-clemson-control-1":"130.127.88.98", "clemson-clemson-compute-1":"130.127.88.99", "clemson-clemson-compute-2":"130.127.88.100"}, "stitching_info" : { "aggregate_id" : "urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+authority+am", "aggregate_url" : "https://clemson-clemson-control-1.clemson.edu:5001", "edge_points" : [ { "local_switch" : "urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+node+procurve2", "port" : "urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+stitchport+procurve2:16", "local_link" : "urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+interface+procurve2:16", "remote_switch": "urn:publicid:IDN+ion.internet2.edu+interface+rtr.atla:xe-0/3/0:clemson-gr", "vlans" : "3000-3009", "traffic_engineering_metric" : "22", "capacity": "1000000", "stitching_granularity":"1", "maximum_reservable_capacity": "1000000", "minimum_reservable_capacity": "1000", "interface_mtu": "1500" } ] }, "public_ip": "130.127.88.101", "mgmt_ns": "qrouter-a03127ba-7184-42e6-bd5d-2e5957193b86" } }}} GRAM GIT Repository was successfully accessed internally: {{{ LNM:~$ git clone git@superior.bbn.com:gram.git Cloning into 'gram'... remote: Counting objects: 12215, done. remote: Compressing objects: 100% (3988/3988), done. remote: Total 12215 (delta 8852), reused 11130 (delta 8026) Receiving objects: 100% (12215/12215), 470.86 MiB | 441.00 KiB/s, done. Resolving deltas: 100% (8852/8852), done. Checking connectivity... done }}} There is no BBN-external access to the GRAM Repository - ticket #66 GRAM Software version installed on control node: {{{ $ dpkg -p gram Package: gram Priority: optional Maintainer: GRAM Developers Architecture: all Version: 3.32 Depends: git, module-assistant, cloud-init, xmlsec1, ntp, vlan, bridge-utils Size: 2195036 Description: GRAM installation package }}} == Step 2. Review control network configuration == A site administrator reviews the configuration of the rack control plane switch and verifies that each experimental node's control and console access interfaces are on the expected VLANs. According to [https://superior.bbn.com/trac/bbn-rack/wiki/ClemsonRackDetails Details on Clemson OpenGENI Rack Installation and Configuration] document interfaces are configured as follows on the OpenGENI nodes: - eth0: Control network configured for 10.10.8.0 - eth1 : Data plane network configured for 10.10.5.0 - eth3 : Management network configured for 10.10.6.0 Subnet masks for network are not listed. === Control Node === The following interfaces are configured on the control node (clemson-clemson-control-1/130.127.88.98): {{{ gram@clemson-clemson-control-1:~$ /sbin/ifconfig br-ex Link encap:Ethernet HWaddr a0:36:9f:11:0a:b2 inet addr:130.127.88.98 Bcast:130.127.88.127 Mask:255.255.255.224 inet6 addr: fe80::a236:9fff:fe11:ab2/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1767593 errors:0 dropped:329916 overruns:0 frame:0 TX packets:2647553 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:286297387 (286.2 MB) TX bytes:2530830293 (2.5 GB) ctl Link encap:Ethernet HWaddr 90:b1:1c:0f:d6:49 inet addr:10.10.8.100 Bcast:10.10.8.255 Mask:255.255.255.0 inet6 addr: fe80::92b1:1cff:fe0f:d649/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:7567719 errors:0 dropped:0 overruns:0 frame:0 TX packets:6985519 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1624052195 (1.6 GB) TX bytes:3613462297 (3.6 GB) Interrupt:37 Memory:d4000000-d47fffff dat Link encap:Ethernet HWaddr a0:36:9f:11:0a:b0 inet addr:10.10.5.100 Bcast:10.10.5.255 Mask:255.255.255.0 inet6 addr: fe80::a236:9fff:fe11:ab0/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1559187 errors:0 dropped:17 overruns:0 frame:0 TX packets:399137 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:149773816 (149.7 MB) TX bytes:81043906 (81.0 MB) ext Link encap:Ethernet HWaddr a0:36:9f:11:0a:b2 inet6 addr: fe80::a236:9fff:fe11:ab2/64 Scope:Link UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1 RX packets:2933086 errors:0 dropped:265 overruns:0 frame:0 TX packets:4313453 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1413216222 (1.4 GB) TX bytes:2647059093 (2.6 GB) int-br-dat Link encap:Ethernet HWaddr ca:dc:26:eb:65:79 inet6 addr: fe80::c8dc:26ff:feeb:6579/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1393949 errors:0 dropped:0 overruns:0 frame:0 TX packets:404422 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:141213388 (141.2 MB) TX bytes:81611296 (81.6 MB) int-br-mgt Link encap:Ethernet HWaddr be:da:ce:fe:46:ff inet6 addr: fe80::bcda:ceff:fefe:46ff/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:578605 errors:0 dropped:0 overruns:0 frame:0 TX packets:1065223 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:77999612 (77.9 MB) TX bytes:1155087194 (1.1 GB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:36888536 errors:0 dropped:0 overruns:0 frame:0 TX packets:36888536 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:11093262695 (11.0 GB) TX bytes:11093262695 (11.0 GB) mgt Link encap:Ethernet HWaddr 90:b1:1c:0f:d6:47 inet addr:10.10.6.100 Bcast:10.10.6.255 Mask:255.255.255.0 inet6 addr: fe80::92b1:1cff:fe0f:d647/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:930178 errors:0 dropped:0 overruns:0 frame:0 TX packets:1061053 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:107818204 (107.8 MB) TX bytes:1163036100 (1.1 GB) Interrupt:36 Memory:d3000000-d37fffff phy-br-dat Link encap:Ethernet HWaddr 72:8f:86:3b:36:84 inet6 addr: fe80::708f:86ff:fe3b:3684/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:404422 errors:0 dropped:0 overruns:0 frame:0 TX packets:1393949 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:81611296 (81.6 MB) TX bytes:141213388 (141.2 MB) phy-br-mgt Link encap:Ethernet HWaddr 06:d7:7e:68:95:8a inet6 addr: fe80::4d7:7eff:fe68:958a/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1065223 errors:0 dropped:0 overruns:0 frame:0 TX packets:578605 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1155087194 (1.1 GB) TX bytes:77999612 (77.9 MB) gram@clemson-clemson-control-1:~$ }}} All connections on Control Node: {{{ gram@clemson-clemson-control-1:~$ sockstat USER PROCESS PID PROTO SOURCE ADDRESS FOREIGN ADDRESS STATE gram python 4835 tcp4 130.127.88.98:8000 *:* LISTEN gram python 5428 tcp4 130.127.88.98:5002 *:* LISTEN gram python 5428 tcp4 130.127.88.98:5001 *:* LISTEN gram python2.7 23275 tcp4 *:6633 *:* LISTEN gram python2.7 23275 tcp4 10.10.8.100:6633 10.10.8.200:55511 ESTABLISHED gram python2.7 23275 tcp4 *:7001 *:* LISTEN gram python2.7 23275 tcp4 130.127.88.98:36894 192.1.249.185:33019 ESTABLISHED gram python2.7 23275 tcp4 130.127.88.98:52614 130.127.88.98:9000 ESTABLISHED gram python2.7 23275 tcp4 130.127.88.98:52615 130.127.88.98:9000 ESTABLISHED gram python2.7 23275 tcp4 127.0.0.1:33549 127.0.0.1:9000 ESTABLISHED gram python2.7 32360 tcp4 *:9000 *:* LISTEN gram python2.7 32360 tcp4 127.0.0.1:9000 127.0.0.1:33549 ESTABLISHED gram python2.7 32360 tcp4 130.127.88.98:9000 130.127.88.98:52614 ESTABLISHED gram python2.7 32360 tcp4 130.127.88.98:9000 130.127.88.98:52615 ESTABLISHED }}} All Know connection are explained: {{ gram@clemson-clemson-control-1:~$ for i in `sockstat | awk '{print $3}' | grep -v PID | sort -u`; do ps -eaf|egrep $i|egrep -v grep; done gram 23275 1 0 Apr19 ? 00:05:53 python2.7 -u -O /opt/pox/pox.py log.level --DEBUG vmoc.VMOC --management_port=7001 --default_controller_url=https://localhost:9000 gram 32360 1 0 14:10 ? 00:00:19 python2.7 -u -O /opt/pox/pox.py log.level --DEBUG openflow.of_01 --port=9000 vmoc.l2_simple_learning gram 4835 4828 0 Apr21 ? 00:00:12 python /opt/gcf/src/gcf-ch.py gram 5428 5421 0 Apr21 ? 00:00:35 python /home/gram/gram/src/gram-am.py gram@clemson-clemson-control-1:~$ for i in `sockstat | awk '{print $3}' | grep -v PID | sort -u`; do ps -eaf|egrep $i|egrep -v grep; done gram 23275 1 0 Apr19 ? 00:05:53 python2.7 -u -O /opt/pox/pox.py log.level --DEBUG vmoc.VMOC --management_port=7001 --default_controller_url=https://localhost:9000 gram 32360 1 0 14:10 ? 00:00:19 python2.7 -u -O /opt/pox/pox.py log.level --DEBUG openflow.of_01 --port=9000 vmoc.l2_simple_learning gram 4835 4828 0 Apr21 ? 00:00:12 python /opt/gcf/src/gcf-ch.py gram 5428 5421 0 Apr21 ? 00:00:35 python /home/gram/gram/src/gram-am.py }}} === Compute Nodes === Information for host clemson-clemson-compute-1 (130.127.88.99): {{{ System information as of Tue Apr 22 18:53:57 EDT 2014 System load: 0.05 Users logged in: 0 Usage of /: 1.3% of 297.06GB IP address for dat: 10.10.5.101 Memory usage: 1% IP address for mgt: 10.10.6.101 Swap usage: 0% IP address for ctl: 10.10.8.101 Processes: 254 IP address for br-ex: 130.127.88.99 }}} There are no connections: {{{ gram@clemson-clemson-compute-1:~$ sockstat USER PROCESS PID PROTO SOURCE ADDRESS FOREIGN ADDRESS STATE gram@clemson-clemson-compute-1:~$ }}} Information for host clemson-clemson-compute-2 (130.127.88.100): {{{ System information as of Tue Apr 22 18:56:21 EDT 2014 System load: 0.0 Users logged in: 0 Usage of /: 1.4% of 297.06GB IP address for dat: 10.10.5.102 Memory usage: 1% IP address for mgt: 10.10.6.102 Swap usage: 0% IP address for ctl: 10.10.8.102 Processes: 257 IP address for br-ex: 130.127.88.100 }}} There are no active connections: {{{ gram@clemson-clemson-compute-2:~$ sockstat USER PROCESS PID PROTO SOURCE ADDRESS FOREIGN ADDRESS STATE gram@clemson-clemson-compute-2:~$ }}} == Step3. Review MAC address == On Control Node get list of MAC addresses: {{{ gram@clemson-clemson-control-1:~$ arp -a ? (10.10.8.102) at 90:b1:1c:0f:f4:79 [ether] on ctl ? (10.10.8.101) at 90:b1:1c:0f:8a:79 [ether] on ctl ? (10.10.8.201) at d0:67:e5:a9:b1:9f [ether] on ctl ? (130.127.88.97) at 00:23:5e:a9:5c:ca [ether] on br-ex ? (10.10.8.200) at 00:01:e8:d7:58:9d [ether] on ctl gram@clemson-clemson-control-1:~$ }}} On Compute Node 1: {{{ gram@clemson-clemson-compute-1:~$ arp -a clemson-clemson-control-1 (10.10.8.100) at 90:b1:1c:0f:d6:49 [ether] on ctl ? (130.127.88.97) at 00:23:5e:a9:5c:ca [ether] on br-ex gram@clemson-clemson-compute-1:~$ }}} On Compute Node 2: {{{ gram@clemson-clemson-compute-2:~$ arp -a ? (130.127.88.97) at 00:23:5e:a9:5c:ca [ether] on br-ex clemson-clemson-control-1 (10.10.8.100) at 90:b1:1c:0f:d6:49 [ether] on ctl gram@clemson-clemson-compute-2:~$ }}} Verified MAC addresses for the Control Network interfaces on the switch.