wiki:GENIRacksHome/OpenGENIRacks/AcceptanceTestStatusMay2013/OG-MON-1

Version 1 (modified by lnevers@bbn.com, 5 years ago) (diff)

--

GR-MON-1: Control Network Software and VLAN Inspection Test

This page captures status for the test case GR-MON-1. For additional information see the Acceptance Test Status - May 2013? page overall status, or the GRAM Acceptance Test Plan? for details about the planned evaluation.

Last Update: 2013/05/15"

Step State Notes Tickets
Step 1 Pass: most criteriaboscompute4 not working, git internal #62 #66
Step 2 Pass
Step 3 Pass


State Legend Description
Pass Test completed and met all criteria
Pass: most criteria Test completed and met most criteria. Exceptions documented
Fail Test completed and failed to meet criteria.
Complete Test completed but will require re-execution due to expected changes
Blocked Blocked by ticketed issue(s).
In Progress Currently under test.

Test Plan Steps

Step 1. Identify processes and versions

A site administrator enumerates processes on each of the server host, the Control VM, the VMOC VM, etc. and an experimental node configured for OpenStack, which listen for network connections from other nodes, identifies what version of what software package is in use for each, and verifies that we know the source of each piece of software and could get access to its source code.

First get supported versions for the AM API and verify access to resources:

$ omni.py getversion -a gram -V3
INFO:omni:Loading config file /home/lnevers/.gcf/omni_config
INFO:omni:Using control framework gram
INFO:omni:Substituting AM nickname gram with URL https://128.89.91.170:5001, URN unspecified_AM_URN
INFO:omni:Substituting AM nickname gram with URL https://128.89.91.170:5001, URN unspecified_AM_URN
INFO:omni:AM URN: unspecified_AM_URN (url: https://128.89.91.170:5001) has version:
INFO:omni:{   'geni_ad_rspec_versions': [   {   'extensions': [],
                                      'namespace': 'http://www.geni.net/resources/rspec/3',
                                      'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd',
                                      'type': 'GENI',
                                      'version': '3'}],
    'geni_api': 3,
    'geni_api_versions': {   '2': 'https://boscontroller.gpolab.bbn.com:5002',
                             '3': 'https://boscontroller.gpolab.bbn.com:5001'},
    'geni_credential_types': [{   'geni_type': 'geni_sfa',
                                     'geni_version': '3'}],
    'geni_request_rspec_versions': [   {   'extensions': [],
                                           'namespace': 'http://www.geni.net/resources/rspec/3',
                                           'schema': 'http://www.geni.net/resources/rspec/3/request.xsd',
                                           'type': 'GENI',
                                           'version': '3'}]}
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed getversion:

  Options as run:
		aggregate: ['gram']
		api_version: 3
		framework: gram
		noGetVersionCache: True

  Args: getversion

  Result Summary: 
Got version for https://128.89.91.170:5001
 
INFO:omni: ============================================================

Then verify that resources are available for experimenters:

$ omni.py listresources -a gram  -o

Note: Fails with ticket #53

Based on the GRAM Network and Software Architecture Description document, the following are major services for GRAM:

  • gram-ch - a test clearinghouse to be replaced by GENI clearinghouse.
  • gram-am - the aggregate manager that handles AM API calls for resources
  • gram-vnoc - VMOC controller that manages OpenFlow traffic for switch(es) and experimenter controller(s).
  • gram-ctrl- Default controller that performs layer 2 MAC to port learning.

The GRAM software is installed in /home/gram/gram on the control node (128.89.91.170) and the GRAM services listed above are started by scripts in the /home/gram/gram/src/services directory. All services were verified to be running.

The gram-vmoc processes:

gram     18694     1  0 May10 ?        00:09:55 python2.7 -u -O /opt/pox/pox.py log.level --DEBUG vmoc.VMOC --management_port=7001 --default_controller_url=https://localhost:9000

The gram-ctl processes default controller:

gram     18680     1  0 May10 ?        00:05:31 python2.7 -u -O /opt/pox/pox.py log.level --DEBUG openflow.of_01 --port=9000 vmoc.l2_simple_learning

The gram-am processes goth V3 (default) and V2:

gram     17475     1  0 07:00 ?        00:00:00 su -s /bin/sh -c exec python /home/gram/gram/src/gram-am.py gram
gram     17482 17475  0 07:00 ?        00:00:00 python /home/gram/gram/src/gram-am.py
gram     17526     1  0 07:00 ?        00:00:00 su -s /bin/sh -c exec python /home/gram/gram/src/gram-am.py -V 2 gram
gram     17533 17526  0 07:00 ?        00:00:00 python /home/gram/gram/src/gram-am.py -V 2

The clearinghouse process:

gram     17360     1  0 07:00 ?        00:00:00 su -s /bin/sh -c exec python /opt/gcf/src/gcf-ch.py gram

To check for all processes that are listening:

lnevers@boscontroller:~$ sudo netstat -anp | egrep ':\*.+LISTEN' | egrep -v '127.0.0.1' >netstat-raw.txt
lnevers@boscontroller:~$ cat netstat-raw.txt 
tcp        0      0 0.0.0.0:9292            0.0.0.0:*               LISTEN      8982/python     
tcp        0      0 0.0.0.0:4369            0.0.0.0:*               LISTEN      7530/epmd       
tcp        0      0 10.0.36.2:53            0.0.0.0:*               LISTEN      17692/dnsmasq   
tcp        0      0 192.168.10.2:53         0.0.0.0:*               LISTEN      9444/dnsmasq    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      18090/sshd      
tcp        0      0 0.0.0.0:7001            0.0.0.0:*               LISTEN      18694/python2.7 
tcp        0      0 0.0.0.0:35357           0.0.0.0:*               LISTEN      8631/python     
tcp        0      0 128.89.91.170:8000      0.0.0.0:*               LISTEN      25968/python    
tcp        0      0 0.0.0.0:9696            0.0.0.0:*               LISTEN      18143/python    
tcp        0      0 0.0.0.0:6080            0.0.0.0:*               LISTEN      4428/python     
tcp        0      0 0.0.0.0:36356           0.0.0.0:*               LISTEN      7562/beam.smp   
tcp        0      0 0.0.0.0:8773            0.0.0.0:*               LISTEN      9058/python     
tcp        0      0 0.0.0.0:8774            0.0.0.0:*               LISTEN      9058/python     
tcp        0      0 0.0.0.0:8775            0.0.0.0:*               LISTEN      9058/python     
tcp        0      0 0.0.0.0:9191            0.0.0.0:*               LISTEN      8990/python     
tcp        0      0 0.0.0.0:9000            0.0.0.0:*               LISTEN      18680/python2.7 
tcp        0      0 0.0.0.0:5000            0.0.0.0:*               LISTEN      8631/python     
tcp        0      0 128.89.91.170:5001      0.0.0.0:*               LISTEN      25822/python    
tcp        0      0 0.0.0.0:6633            0.0.0.0:*               LISTEN      18694/python2.7 
tcp        0      0 128.89.91.170:5002      0.0.0.0:*               LISTEN      25887/python    
tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      8427/mysqld     
tcp6       0      0 fe80::f816:3eff:feed:53 :::*                    LISTEN      17692/dnsmasq   
tcp6       0      0 fe80::f816:3eff:fe72:53 :::*                    LISTEN      9444/dnsmasq    
tcp6       0      0 :::22                   :::*                    LISTEN      18090/sshd      
tcp6       0      0 ::1:631                 :::*                    LISTEN      1034/cupsd      
tcp6       0      0 ::1:6010                :::*                    LISTEN      30396/22        
tcp6       0      0 :::5672                 :::*                    LISTEN      7562/beam.smp   

Stripping list of processes to get a list of PIDs, and determine what the processes that are listening:

lnevers@boscontroller:~$ sudo netstat -anp | egrep ':\*.+LISTEN' | egrep -v '127.0.0.1' >netstat-raw.txt
lnevers@boscontroller:~$ awk '{print $NF}' netstat-raw.txt | sed -r -e 's|/.*||' | sort -n | uniq > netstat-pids.txt
lnevers@boscontroller:~$ for pid in $(cat netstat-pids.txt) ; do ps -efwww | egrep "^[^ ]+ +$pid" ; done >& netstat-commands.txt
lnevers@boscontroller:~$ cat netstat-commands.txt
root      1034     1  0 May10 ?        00:00:00 /usr/sbin/cupsd -F
nova      4428  4420  0 May10 ?        00:00:21 /usr/bin/python /usr/bin/nova-novncproxy --config-file=/etc/nova/nova.conf
rabbitmq  7530     1  0 May10 ?        00:00:02 /usr/lib/erlang/erts-5.8.5/bin/epmd -daemon
rabbitmq  7562     1  0 May10 ?        00:06:00 /usr/lib/erlang/erts-5.8.5/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -noshell -noinput -sname rabbit@boscontroller -boot /var/lib/rabbitmq/mnesia/rabbit@boscontroller-plugins-expand/rabbit -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@boscontroller.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@boscontroller-sasl.log"} -os_mon start_cpu_sup true -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@boscontroller"
mysql     8427     1  1 May10 ?        01:52:51 /usr/sbin/mysqld
keystone  8631  8623  5 May10 ?        06:13:28 /usr/bin/python /usr/bin/keystone-all
glance    8982  8963  0 May10 ?        00:00:00 /usr/bin/python /usr/bin/glance-api
glance    8990  8981  0 May10 ?        00:00:00 /usr/bin/python /usr/bin/glance-registry
nova      9058  9039  0 May10 ?        00:00:01 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf
nobody    9444     1  0 May10 ?        00:00:01 /usr/sbin/dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap350960c0-bf --except-interface=lo --domain=openstacklocal --pid-file=/var/lib/quantum/dhcp/f80c6090-4321-4ffa-8f8b-eb9f897bd4ca/pid --dhcp-hostsfile=/var/lib/quantum/dhcp/f80c6090-4321-4ffa-8f8b-eb9f897bd4ca/host --dhcp-optsfile=/var/lib/quantum/dhcp/f80c6090-4321-4ffa-8f8b-eb9f897bd4ca/opts --dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update --leasefile-ro --dhcp-range=set:tag0,192.168.10.0,static,120s --conf-file= --server=8.8.8.8
nobody   17692     1  0 May14 ?        00:00:01 /usr/sbin/dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tapb48adcd2-64 --except-interface=lo --domain=openstacklocal --pid-file=/var/lib/quantum/dhcp/1072ae37-d6e2-46a6-be76-234db45cc907/pid --dhcp-hostsfile=/var/lib/quantum/dhcp/1072ae37-d6e2-46a6-be76-234db45cc907/host --dhcp-optsfile=/var/lib/quantum/dhcp/1072ae37-d6e2-46a6-be76-234db45cc907/opts --dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update --leasefile-ro --dhcp-range=set:tag0,10.0.36.0,static,120s --conf-file= --server=8.8.8.8
root     18090     1  0 May10 ?        00:00:00 /usr/sbin/sshd -D
quantum  18143     1  1 May10 ?        01:42:41 python /usr/bin/quantum-server --config-file /etc/quantum/quantum.conf --log-file /var/log/quantum/server.log --config-file /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini
gram     18680     1  0 May10 ?        00:05:37 python2.7 -u -O /opt/pox/pox.py log.level --DEBUG openflow.of_01 --port=9000 vmoc.l2_simple_learning
gram     18694     1  0 May10 ?        00:10:07 python2.7 -u -O /opt/pox/pox.py log.level --DEBUG vmoc.VMOC --management_port=7001 --default_controller_url=https://localhost:9000
gram     25822 25815  0 08:46 ?        00:00:01 python /home/gram/gram/src/gram-am.py
gram     25887 25880  0 08:46 ?        00:00:01 python /home/gram/gram/src/gram-am.py -V 2
gram     25968 25961  0 08:46 ?        00:00:01 python /opt/gcf/src/gcf-ch.py
lnevers  30396 30235  0 09:41 ?        00:00:00 sshd: lnevers@pts/22
lnevers@boscontroller:~$ 

Additionally GRAM software configuration is captured on the control node:

lnevers@boscontroller:~$ more  /etc/gram/config.json 
{
   "__001": "This file contains GRAM installation-specific", 
   "__002": "parameters and structures. The values contained here",
   "__003": "override values in the gram.am.gram.config module.",
   "__004": "That is, a key/value pair  K, V  in this file", 
   "__005": "sets gram.am.config.K = V",
   "__006": "An error is given if the field (K) does not exist, ",
   "__007": "Or the type of the value cannot be coerced",
   "__008": "Further any parameter whose value is None in config.py", 
   "__009": "and set as null here must be set in this file:",
   "__010": "no default is provided",


   "disk_image_metadata": 
   {
   "ubuntu-2nic": 
       {
        "os": "Linux", 
        "version": "12.0", 
        "description":"Ubuntu image with 2 NICs configured"
	},
   "cirros-2nic-x86_64": 
       {
        "os": "Linux", 
        "version": "12.0", 
        "description":"Cirros image with 2 NICs configured"
	}
   }, 

   "external_interface": "eth2", 
   "external_address": "128.89.91.170", 
   "external_netmask": "255.255.255.224", 
   "control_address": "10.10.8.100", 
   "control_interface": "eth0", 
   "control_netmask": "255.255.255.0", 
   "data_interface": "eth1", 
   "management_interface": "eth3",
   "management_address": "10.10.6.100",
   "management_netmask": "255.255.255.0",
   "mysql_user": "admin", 
   "mysql_password": "XXX", 
   "rabbit_password": "XXX", 
   "nova_password": "XXX", 
   "glance_password": "XXXX", 
   "keystone_password": "XXXX", 
   "quantum_password": "XXXX", 
   "os_tenant_name": "admin", 
   "os_username": "admin", 
   "os_password": "XXXX", 
   "os_auth_url": "http://localhost:5000/v2.0/",
   "os_region_name" : "RegionOne", 
   "os_no_cache" : "1",
   "service_token" : "bos",
   "service_endpoint" : "http://localhost:35357/v2.0",

   "public_gateway_ip" : "128.89.91.162",
   "public_subnet_cidr" : "128.89.91.160/27",
   "public_subnet_start_ip" : "128.89.91.163",
   "public_subnet_end_ip" : "128.89.91.169",

   "metadata_port" : "8775",

   "backup_directory" : "/home/gram/.backup",
   "control_host" : "boscontroller",
   "compute_hosts": {
   	    "boscompute1": "10.10.8.101", 
	    "boscompute2": "10.10.8.102", 
	    "boscompute4": "10.10.8.104"},
   "glance_images" : []

}

Note: There are 3 configured compute nodes, but only two are being used. boscompute4 not working and needs to be restored.

Access to code is possible as follows:

Note: GIT repository is currently internal only

GRAM Software version:

lnevers@boscontroller:~$ dpkg -p gram
Package: gram
Priority: optional
Maintainer: GRAM Developers <gram-dev@bbn.com>
Architecture: all
Version: 05102013B
Depends: git, cloud-utils, kvm, xmlsec1, python-software-properties, ubuntu-cloud-keyring, ntp, mysql-server, python-mysqldb, rabbitmq-server, keystone, python-keystone, p
Size: 1921726
Description: GRAM installation package
lnevers@boscontroller:~$ 

Step 2. Review control network configuration

A site administrator reviews the configuration of the rack control plane switch and verifies that each experimental node's control and console access interfaces are on the expected VLANs. According to GRAM Network and Software Architecture Description document interfaces are configured as follows on the GRAM nodes:

  • eth0: Control network configured for 10.10.8.0/24
  • eth1 : Data network configured for 10.10.5.0/24
  • eth2 : External network configured for 128.89.91.0/27
  • eth3 : Management network configured for 10.10.6.0/24

Control Node

The following interfaces are configured on the control node (boscontroller/128.89.91.170):

eth0      Link encap:Ethernet  HWaddr 78:2b:cb:50:68:01  
          inet addr:10.10.8.100  Bcast:10.10.8.255  Mask:255.255.255.0   <<<<== Control Network Interface
          inet6 addr: fe80::7a2b:cbff:fe50:6801/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2558290 errors:0 dropped:188657 overruns:0 frame:0
          TX packets:2770380 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1005762310 (1.0 GB)  TX bytes:2334137468 (2.3 GB)

eth1      Link encap:Ethernet  HWaddr 78:2b:cb:50:68:02  
          inet addr:10.10.5.100  Bcast:10.10.5.255  Mask:255.255.255.0   <<<<== Data Network Interface
          inet6 addr: fe80::7a2b:cbff:fe50:6802/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:73830 errors:0 dropped:137 overruns:0 frame:0
          TX packets:3228 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:8326147 (8.3 MB)  TX bytes:562478 (562.4 KB)

eth3      Link encap:Ethernet  HWaddr 00:1b:21:bc:27:11  
          inet addr:10.10.6.100  Bcast:10.10.6.255  Mask:255.255.255.0  <<<<== Management Network Interface
          inet6 addr: fe80::21b:21ff:febc:2711/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1141956 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6392 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:91674129 (91.6 MB)  TX bytes:814927 (814.9 KB)
          Memory:dc7e0000-dc800000 

Compute Nodes

The following interfaces are configured on the compute node 1 (boscompute1/128.89.91.171):

eth0      Link encap:Ethernet  HWaddr 78:2b:cb:4f:62:6c    
          inet addr:10.10.8.101  Bcast:10.10.8.255  Mask:255.255.255.0       <<<<== Control Network Interface
          inet6 addr: fe80::7a2b:cbff:fe4f:626c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1415472 errors:0 dropped:61316 overruns:0 frame:0
          TX packets:728707 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:732279308 (732.2 MB)  TX bytes:325014011 (325.0 MB)

eth1      Link encap:Ethernet  HWaddr 78:2b:cb:4f:62:6d  
          inet addr:10.10.5.101  Bcast:10.10.5.255  Mask:255.255.255.0       <<<<== Data Network Interface
          inet6 addr: fe80::7a2b:cbff:fe4f:626d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:93698 errors:0 dropped:4435 overruns:0 frame:0
          TX packets:1397 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:12247041 (12.2 MB)  TX bytes:254376 (254.3 KB)

eth2      Link encap:Ethernet  HWaddr 00:1b:21:bc:23:f0  
          inet addr:128.89.91.171  Bcast:128.89.91.191  Mask:255.255.255.224    <<<<== External Network Interface
          inet6 addr: fe80::21b:21ff:febc:23f0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:753699 errors:0 dropped:0 overruns:0 frame:0
          TX packets:161176 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:583999876 (583.9 MB)  TX bytes:13922812 (13.9 MB)
          Memory:dc7c0000-dc7e0000 

eth3      Link encap:Ethernet  HWaddr 00:1b:21:bc:23:f1  
          inet addr:10.10.6.101  Bcast:10.10.6.255  Mask:255.255.255.0       <<<<== Management Network Interface
          inet6 addr: fe80::21b:21ff:febc:23f1/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1202254 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4896 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:97559060 (97.5 MB)  TX bytes:1043945 (1.0 MB)
          Memory:dc7e0000-dc800000 

The following interfaces are configured on the compute node 2 (boscompute2/128.89.91.172):

eth0      Link encap:Ethernet  HWaddr 78:2b:cb:50:5c:00  
          inet addr:10.10.8.102  Bcast:10.10.8.255  Mask:255.255.255.0      <<<<== Control Network Interface
          inet6 addr: fe80::7a2b:cbff:fe50:5c00/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1934002 errors:0 dropped:64759 overruns:0 frame:0
          TX packets:1051218 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1171016212 (1.1 GB)  TX bytes:438688993 (438.6 MB)

eth1      Link encap:Ethernet  HWaddr 78:2b:cb:50:5c:01  
          inet addr:10.10.5.102  Bcast:10.10.5.255  Mask:255.255.255.0      <<<<== Data Network Interface
          inet6 addr: fe80::7a2b:cbff:fe50:5c01/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:143088 errors:0 dropped:4494 overruns:0 frame:0
          TX packets:6321 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:17382441 (17.3 MB)  TX bytes:1056197 (1.0 MB)

eth2      Link encap:Ethernet  HWaddr 00:1b:21:4b:3e:b4  
          inet addr:128.89.91.172  Bcast:128.89.91.191  Mask:255.255.255.224   <<<<== External Network Interface
          inet6 addr: fe80::21b:21ff:fe4b:3eb4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:766841 errors:0 dropped:0 overruns:0 frame:0
          TX packets:165589 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:566156761 (566.1 MB)  TX bytes:16091017 (16.0 MB)
          Memory:df3c0000-df3e0000 

eth3      Link encap:Ethernet  HWaddr 00:1b:21:4b:3e:b5  
          inet addr:10.10.6.102  Bcast:0.0.0.0  Mask:255.255.255.255          <<<<== Management Network Interface
          inet6 addr: fe80::21b:21ff:fe4b:3eb5/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1438112 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14301 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:128257526 (128.2 MB)  TX bytes:2862471 (2.8 MB)
          Memory:df3e0000-df400000 

The following interfaces are configured on the compute node 4 (boscompute4/128.89.91.174):

eth0      Link encap:Ethernet  HWaddr 78:2b:cb:51:a6:82  
          inet addr:10.10.8.104  Bcast:10.10.8.255  Mask:255.255.255.0           <<<<== Control Network Interface
          inet6 addr: fe80::7a2b:cbff:fe51:a682/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:772020 errors:0 dropped:7466 overruns:0 frame:0
          TX packets:516318 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:550568324 (550.5 MB)  TX bytes:191280624 (191.2 MB)

eth1      Link encap:Ethernet  HWaddr 78:2b:cb:51:a6:83  
          inet addr:10.10.5.104  Bcast:10.10.5.255  Mask:255.255.255.0            <<<<== Data Network Interface
          inet6 addr: fe80::7a2b:cbff:fe51:a683/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:10463 errors:0 dropped:106 overruns:0 frame:0
          TX packets:396775 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:1729203 (1.7 MB)  TX bytes:35743239 (35.7 MB)

eth2      Link encap:Ethernet  HWaddr 00:1b:21:bc:22:b8  
          inet addr:128.89.91.174  Bcast:128.89.91.191  Mask:255.255.255.224      <<<<== External Network Interface
          inet6 addr: fe80::21b:21ff:febc:22b8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:439246 errors:0 dropped:0 overruns:0 frame:0
          TX packets:168175 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:577114949 (577.1 MB)  TX bytes:13394855 (13.3 MB)
          Memory:dc7c0000-dc7e0000 

eth3      Link encap:Ethernet  HWaddr 00:1b:21:bc:22:b9  
          inet addr:10.10.6.104  Bcast:10.10.6.255  Mask:255.255.255.0     <<<<== Management Network Interface
          inet6 addr: fe80::21b:21ff:febc:22b9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:488460 errors:0 dropped:0 overruns:0 frame:0
          TX packets:11424 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:37649896 (37.6 MB)  TX bytes:2529244 (2.5 MB)
          Memory:dc7e0000-dc800000 

Step 3. Review MAC address

A site administrator reviews the MAC address table on the control plane switch, and verifies that all entries are identifiable and expected.

On the control node listed all MAC addresses on the control network interface (eth0):

lnevers@boscontroller:/var$ arp -i eth0
Address                  HWtype  HWaddress           Flags Mask            Iface
boscompute4.local        ether   78:2b:cb:51:a6:82   C                     eth0
boscompute1.local        ether   78:2b:cb:4f:62:6c   C                     eth0
boscompute2.local        ether   78:2b:cb:50:5c:00   C                     eth0

On compute node 1 listed all MAC addresses on the control network interface (eth0):

lnevers@boscompute1:~$ arp -i eth0
Address                  HWtype  HWaddress           Flags Mask            Iface
boscontroller.local      ether   78:2b:cb:50:68:01   C                     eth0

On compute node 2 listed all MAC addresses on the control network interface (eth0):

lnevers@boscompute2:~$ arp -i eth0
Address                  HWtype  HWaddress           Flags Mask            Iface
boscontroller.local      ether   78:2b:cb:50:68:01   C                     eth0

On compute node 4 listed all MAC addresses on the control network interface (eth0):

lnevers@boscompute4:~$ arp -i eth0
Address                  HWtype  HWaddress           Flags Mask            Iface
boscontroller.local      ether   78:2b:cb:50:68:01   C                     eth0

The above ARP entries show that only expected MAC addresses are in use:

  • control node has MAC 78:2b:cb:50:68:01 on eth0.
  • compute node 1 has MAC 78:2b:cb:4f:62:6c on eth0
  • compute node 2 has MAC 78:2b:cb:50:5c:00 on eth0
  • compute node 4 has MAC 78:2b:cb:51:a6:82 on eth0