= OG-MON-1: Control Network Software and VLAN Inspection Test = This page captures status for the test case OG-MON-1. For additional information see the [wiki:GENIRacksHome/OpenGENIRacks/AcceptanceTestStatusMay2013 Acceptance Test Status - May 2013] page overall status, or the [wiki:GENIRacksHome/AcceptanceTests/OpenGENIAcceptanceTestsPlan OpenGENI Acceptance Test Plan] for details about the planned evaluation. ''Last Update: 2013/05/15" || '''Step''' || '''State'''||''' Notes ''' || '''Tickets''' || || Step 1 ||[[span(Pass: most criteria, style=background-color: #98FB98)]]||boscompute4 not working, git internal || #62 #66 || || Step 2 ||[[span(Pass, style=background-color: green )]]|| || || || Step 3 ||[[span(Pass, style=background-color: green )]]|| || || [[BR]] || '''State Legend''' || '''Description''' || ||[[span(Pass, style=background-color: green )]] || Test completed and met all criteria || ||[[span(Pass: most criteria, style=background-color: #98FB98)]]|| Test completed and met most criteria. Exceptions documented || ||[[span(Fail, style=background-color: red)]] || Test completed and failed to meet criteria. || ||[[span(Complete, style=background-color: yellow)]] || Test completed but will require re-execution due to expected changes || ||[[span(Blocked, style=background-color: orange)]] || Blocked by ticketed issue(s). || ||[[span(In Progress, style=background-color: #63B8FF)]]|| Currently under test. || = Test Plan Steps = == Step 1. Identify processes and versions == A site administrator enumerates processes on each of the server host, the Control VM, the VMOC VM, etc. and an experimental node configured for OpenStack, which listen for network connections from other nodes, identifies what version of what software package is in use for each, and verifies that we know the source of each piece of software and could get access to its source code. First get supported versions for the AM API and verify access to resources: {{{ $ omni.py getversion -a gram -V3 INFO:omni:Loading config file /home/lnevers/.gcf/omni_config INFO:omni:Using control framework gram INFO:omni:Substituting AM nickname gram with URL https://128.89.91.170:5001, URN unspecified_AM_URN INFO:omni:Substituting AM nickname gram with URL https://128.89.91.170:5001, URN unspecified_AM_URN INFO:omni:AM URN: unspecified_AM_URN (url: https://128.89.91.170:5001) has version: INFO:omni:{ 'geni_ad_rspec_versions': [ { 'extensions': [], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd', 'type': 'GENI', 'version': '3'}], 'geni_api': 3, 'geni_api_versions': { '2': 'https://boscontroller.gpolab.bbn.com:5002', '3': 'https://boscontroller.gpolab.bbn.com:5001'}, 'geni_credential_types': [{ 'geni_type': 'geni_sfa', 'geni_version': '3'}], 'geni_request_rspec_versions': [ { 'extensions': [], 'namespace': 'http://www.geni.net/resources/rspec/3', 'schema': 'http://www.geni.net/resources/rspec/3/request.xsd', 'type': 'GENI', 'version': '3'}]} INFO:omni: ------------------------------------------------------------ INFO:omni: Completed getversion: Options as run: aggregate: ['gram'] api_version: 3 framework: gram noGetVersionCache: True Args: getversion Result Summary: Got version for https://128.89.91.170:5001 INFO:omni: ============================================================ }}} Then verify that resources are available for experimenters: {{{ $ omni.py listresources -a gram -o }}} '''Note: Fails with ticket #53 Based on the [http://groups.geni.net/geni/wiki/GENIRacksHome/OpenGENIRacks/ArchitectureDescription OpenGENI Network and Software Architecture Description] document, the following are major services for OpenGENI: - ''gram-ch'' - a test clearinghouse to be replaced by GENI clearinghouse. - ''gram-am'' - the aggregate manager that handles AM API calls for resources - ''gram-vnoc'' - VMOC controller that manages OpenFlow traffic for switch(es) and experimenter controller(s). - ''gram-ctrl''- Default controller that performs layer 2 MAC to port learning. The OpenGENI software is installed in /home/gram/gram on the control node (128.89.91.170) and the OpenGENI services listed above are started by scripts in the /home/gram/gram/src/services directory. All services were verified to be running. The gram-vmoc processes: {{{ gram 18694 1 0 May10 ? 00:09:55 python2.7 -u -O /opt/pox/pox.py log.level --DEBUG vmoc.VMOC --management_port=7001 --default_controller_url=https://localhost:9000 }}} The gram-ctl processes default controller: {{{ gram 18680 1 0 May10 ? 00:05:31 python2.7 -u -O /opt/pox/pox.py log.level --DEBUG openflow.of_01 --port=9000 vmoc.l2_simple_learning }}} The gram-am processes goth V3 (default) and V2: {{{ gram 17475 1 0 07:00 ? 00:00:00 su -s /bin/sh -c exec python /home/gram/gram/src/gram-am.py gram gram 17482 17475 0 07:00 ? 00:00:00 python /home/gram/gram/src/gram-am.py gram 17526 1 0 07:00 ? 00:00:00 su -s /bin/sh -c exec python /home/gram/gram/src/gram-am.py -V 2 gram gram 17533 17526 0 07:00 ? 00:00:00 python /home/gram/gram/src/gram-am.py -V 2 }}} The clearinghouse process: {{{ gram 17360 1 0 07:00 ? 00:00:00 su -s /bin/sh -c exec python /opt/gcf/src/gcf-ch.py gram }}} To check for all processes that are listening: {{{ lnevers@boscontroller:~$ sudo netstat -anp | egrep ':\*.+LISTEN' | egrep -v '127.0.0.1' >netstat-raw.txt lnevers@boscontroller:~$ cat netstat-raw.txt tcp 0 0 0.0.0.0:9292 0.0.0.0:* LISTEN 8982/python tcp 0 0 0.0.0.0:4369 0.0.0.0:* LISTEN 7530/epmd tcp 0 0 10.0.36.2:53 0.0.0.0:* LISTEN 17692/dnsmasq tcp 0 0 192.168.10.2:53 0.0.0.0:* LISTEN 9444/dnsmasq tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 18090/sshd tcp 0 0 0.0.0.0:7001 0.0.0.0:* LISTEN 18694/python2.7 tcp 0 0 0.0.0.0:35357 0.0.0.0:* LISTEN 8631/python tcp 0 0 128.89.91.170:8000 0.0.0.0:* LISTEN 25968/python tcp 0 0 0.0.0.0:9696 0.0.0.0:* LISTEN 18143/python tcp 0 0 0.0.0.0:6080 0.0.0.0:* LISTEN 4428/python tcp 0 0 0.0.0.0:36356 0.0.0.0:* LISTEN 7562/beam.smp tcp 0 0 0.0.0.0:8773 0.0.0.0:* LISTEN 9058/python tcp 0 0 0.0.0.0:8774 0.0.0.0:* LISTEN 9058/python tcp 0 0 0.0.0.0:8775 0.0.0.0:* LISTEN 9058/python tcp 0 0 0.0.0.0:9191 0.0.0.0:* LISTEN 8990/python tcp 0 0 0.0.0.0:9000 0.0.0.0:* LISTEN 18680/python2.7 tcp 0 0 0.0.0.0:5000 0.0.0.0:* LISTEN 8631/python tcp 0 0 128.89.91.170:5001 0.0.0.0:* LISTEN 25822/python tcp 0 0 0.0.0.0:6633 0.0.0.0:* LISTEN 18694/python2.7 tcp 0 0 128.89.91.170:5002 0.0.0.0:* LISTEN 25887/python tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 8427/mysqld tcp6 0 0 fe80::f816:3eff:feed:53 :::* LISTEN 17692/dnsmasq tcp6 0 0 fe80::f816:3eff:fe72:53 :::* LISTEN 9444/dnsmasq tcp6 0 0 :::22 :::* LISTEN 18090/sshd tcp6 0 0 ::1:631 :::* LISTEN 1034/cupsd tcp6 0 0 ::1:6010 :::* LISTEN 30396/22 tcp6 0 0 :::5672 :::* LISTEN 7562/beam.smp }}} Stripping list of processes to get a list of PIDs, and determine what the processes that are listening: {{{ lnevers@boscontroller:~$ sudo netstat -anp | egrep ':\*.+LISTEN' | egrep -v '127.0.0.1' >netstat-raw.txt lnevers@boscontroller:~$ awk '{print $NF}' netstat-raw.txt | sed -r -e 's|/.*||' | sort -n | uniq > netstat-pids.txt lnevers@boscontroller:~$ for pid in $(cat netstat-pids.txt) ; do ps -efwww | egrep "^[^ ]+ +$pid" ; done >& netstat-commands.txt lnevers@boscontroller:~$ cat netstat-commands.txt root 1034 1 0 May10 ? 00:00:00 /usr/sbin/cupsd -F nova 4428 4420 0 May10 ? 00:00:21 /usr/bin/python /usr/bin/nova-novncproxy --config-file=/etc/nova/nova.conf rabbitmq 7530 1 0 May10 ? 00:00:02 /usr/lib/erlang/erts-5.8.5/bin/epmd -daemon rabbitmq 7562 1 0 May10 ? 00:06:00 /usr/lib/erlang/erts-5.8.5/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -noshell -noinput -sname rabbit@boscontroller -boot /var/lib/rabbitmq/mnesia/rabbit@boscontroller-plugins-expand/rabbit -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@boscontroller.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@boscontroller-sasl.log"} -os_mon start_cpu_sup true -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@boscontroller" mysql 8427 1 1 May10 ? 01:52:51 /usr/sbin/mysqld keystone 8631 8623 5 May10 ? 06:13:28 /usr/bin/python /usr/bin/keystone-all glance 8982 8963 0 May10 ? 00:00:00 /usr/bin/python /usr/bin/glance-api glance 8990 8981 0 May10 ? 00:00:00 /usr/bin/python /usr/bin/glance-registry nova 9058 9039 0 May10 ? 00:00:01 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf nobody 9444 1 0 May10 ? 00:00:01 /usr/sbin/dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap350960c0-bf --except-interface=lo --domain=openstacklocal --pid-file=/var/lib/quantum/dhcp/f80c6090-4321-4ffa-8f8b-eb9f897bd4ca/pid --dhcp-hostsfile=/var/lib/quantum/dhcp/f80c6090-4321-4ffa-8f8b-eb9f897bd4ca/host --dhcp-optsfile=/var/lib/quantum/dhcp/f80c6090-4321-4ffa-8f8b-eb9f897bd4ca/opts --dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update --leasefile-ro --dhcp-range=set:tag0,192.168.10.0,static,120s --conf-file= --server=8.8.8.8 nobody 17692 1 0 May14 ? 00:00:01 /usr/sbin/dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tapb48adcd2-64 --except-interface=lo --domain=openstacklocal --pid-file=/var/lib/quantum/dhcp/1072ae37-d6e2-46a6-be76-234db45cc907/pid --dhcp-hostsfile=/var/lib/quantum/dhcp/1072ae37-d6e2-46a6-be76-234db45cc907/host --dhcp-optsfile=/var/lib/quantum/dhcp/1072ae37-d6e2-46a6-be76-234db45cc907/opts --dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update --leasefile-ro --dhcp-range=set:tag0,10.0.36.0,static,120s --conf-file= --server=8.8.8.8 root 18090 1 0 May10 ? 00:00:00 /usr/sbin/sshd -D quantum 18143 1 1 May10 ? 01:42:41 python /usr/bin/quantum-server --config-file /etc/quantum/quantum.conf --log-file /var/log/quantum/server.log --config-file /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini gram 18680 1 0 May10 ? 00:05:37 python2.7 -u -O /opt/pox/pox.py log.level --DEBUG openflow.of_01 --port=9000 vmoc.l2_simple_learning gram 18694 1 0 May10 ? 00:10:07 python2.7 -u -O /opt/pox/pox.py log.level --DEBUG vmoc.VMOC --management_port=7001 --default_controller_url=https://localhost:9000 gram 25822 25815 0 08:46 ? 00:00:01 python /home/gram/gram/src/gram-am.py gram 25887 25880 0 08:46 ? 00:00:01 python /home/gram/gram/src/gram-am.py -V 2 gram 25968 25961 0 08:46 ? 00:00:01 python /opt/gcf/src/gcf-ch.py lnevers 30396 30235 0 09:41 ? 00:00:00 sshd: lnevers@pts/22 lnevers@boscontroller:~$ }}} Additionally OpenGENI software configuration is captured on the control node: {{{ lnevers@boscontroller:~$ more /etc/gram/config.json { "__001": "This file contains OpenGENI installation-specific", "__002": "parameters and structures. The values contained here", "__003": "override values in the gram.am.gram.config module.", "__004": "That is, a key/value pair K, V in this file", "__005": "sets gram.am.config.K = V", "__006": "An error is given if the field (K) does not exist, ", "__007": "Or the type of the value cannot be coerced", "__008": "Further any parameter whose value is None in config.py", "__009": "and set as null here must be set in this file:", "__010": "no default is provided", "disk_image_metadata": { "ubuntu-2nic": { "os": "Linux", "version": "12.0", "description":"Ubuntu image with 2 NICs configured" }, "cirros-2nic-x86_64": { "os": "Linux", "version": "12.0", "description":"Cirros image with 2 NICs configured" } }, "external_interface": "eth2", "external_address": "128.89.91.170", "external_netmask": "255.255.255.224", "control_address": "10.10.8.100", "control_interface": "eth0", "control_netmask": "255.255.255.0", "data_interface": "eth1", "management_interface": "eth3", "management_address": "10.10.6.100", "management_netmask": "255.255.255.0", "mysql_user": "admin", "mysql_password": "XXX", "rabbit_password": "XXX", "nova_password": "XXX", "glance_password": "XXXX", "keystone_password": "XXXX", "quantum_password": "XXXX", "os_tenant_name": "admin", "os_username": "admin", "os_password": "XXXX", "os_auth_url": "http://localhost:5000/v2.0/", "os_region_name" : "RegionOne", "os_no_cache" : "1", "service_token" : "bos", "service_endpoint" : "http://localhost:35357/v2.0", "public_gateway_ip" : "128.89.91.162", "public_subnet_cidr" : "128.89.91.160/27", "public_subnet_start_ip" : "128.89.91.163", "public_subnet_end_ip" : "128.89.91.169", "metadata_port" : "8775", "backup_directory" : "/home/gram/.backup", "control_host" : "boscontroller", "compute_hosts": { "boscompute1": "10.10.8.101", "boscompute2": "10.10.8.102", "boscompute4": "10.10.8.104"}, "glance_images" : [] } }}} '''Note:''' There are 3 configured compute nodes, but only two are being used. boscompute4 not working and needs to be restored. Access to code is possible as follows: - on the OpenGENI nodes, source is available in /home/gram/gram/src. - on the git repository git@superior.bbn.com:gram.git and credentials can be requested from gram-dev@bbn.com. ''Note:'' GIT repository is currently internal only OpenGENI Software version: {{{ lnevers@boscontroller:~$ dpkg -p gram Package: gram Priority: optional Maintainer: OpenGENI Developers Architecture: all Version: 05102013B Depends: git, cloud-utils, kvm, xmlsec1, python-software-properties, ubuntu-cloud-keyring, ntp, mysql-server, python-mysqldb, rabbitmq-server, keystone, python-keystone, p Size: 1921726 Description: OpenGENI installation package lnevers@boscontroller:~$ }}} == Step 2. Review control network configuration == A site administrator reviews the configuration of the rack control plane switch and verifies that each experimental node's control and console access interfaces are on the expected VLANs. According to [http://groups.geni.net/geni/wiki/GENIRacksHome/OpenGENIRacks/ArchitectureDescription OpenGENI Network and Software Architecture Description] document interfaces are configured as follows on the OpenGENI nodes: - eth0: Control network configured for 10.10.8.0/24 - eth1 : Data network configured for 10.10.5.0/24 - eth2 : External network configured for 128.89.91.0/27 - eth3 : Management network configured for 10.10.6.0/24 === Control Node === The following interfaces are configured on the control node (boscontroller/128.89.91.170): {{{ eth0 Link encap:Ethernet HWaddr 78:2b:cb:50:68:01 inet addr:10.10.8.100 Bcast:10.10.8.255 Mask:255.255.255.0 <<<<== Control Network Interface inet6 addr: fe80::7a2b:cbff:fe50:6801/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:2558290 errors:0 dropped:188657 overruns:0 frame:0 TX packets:2770380 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1005762310 (1.0 GB) TX bytes:2334137468 (2.3 GB) eth1 Link encap:Ethernet HWaddr 78:2b:cb:50:68:02 inet addr:10.10.5.100 Bcast:10.10.5.255 Mask:255.255.255.0 <<<<== Data Network Interface inet6 addr: fe80::7a2b:cbff:fe50:6802/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:73830 errors:0 dropped:137 overruns:0 frame:0 TX packets:3228 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:8326147 (8.3 MB) TX bytes:562478 (562.4 KB) eth3 Link encap:Ethernet HWaddr 00:1b:21:bc:27:11 inet addr:10.10.6.100 Bcast:10.10.6.255 Mask:255.255.255.0 <<<<== Management Network Interface inet6 addr: fe80::21b:21ff:febc:2711/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1141956 errors:0 dropped:0 overruns:0 frame:0 TX packets:6392 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:91674129 (91.6 MB) TX bytes:814927 (814.9 KB) Memory:dc7e0000-dc800000 }}} === Compute Nodes === The following interfaces are configured on the compute node 1 (boscompute1/128.89.91.171): {{{ eth0 Link encap:Ethernet HWaddr 78:2b:cb:4f:62:6c inet addr:10.10.8.101 Bcast:10.10.8.255 Mask:255.255.255.0 <<<<== Control Network Interface inet6 addr: fe80::7a2b:cbff:fe4f:626c/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1415472 errors:0 dropped:61316 overruns:0 frame:0 TX packets:728707 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:732279308 (732.2 MB) TX bytes:325014011 (325.0 MB) eth1 Link encap:Ethernet HWaddr 78:2b:cb:4f:62:6d inet addr:10.10.5.101 Bcast:10.10.5.255 Mask:255.255.255.0 <<<<== Data Network Interface inet6 addr: fe80::7a2b:cbff:fe4f:626d/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:93698 errors:0 dropped:4435 overruns:0 frame:0 TX packets:1397 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:12247041 (12.2 MB) TX bytes:254376 (254.3 KB) eth2 Link encap:Ethernet HWaddr 00:1b:21:bc:23:f0 inet addr:128.89.91.171 Bcast:128.89.91.191 Mask:255.255.255.224 <<<<== External Network Interface inet6 addr: fe80::21b:21ff:febc:23f0/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:753699 errors:0 dropped:0 overruns:0 frame:0 TX packets:161176 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:583999876 (583.9 MB) TX bytes:13922812 (13.9 MB) Memory:dc7c0000-dc7e0000 eth3 Link encap:Ethernet HWaddr 00:1b:21:bc:23:f1 inet addr:10.10.6.101 Bcast:10.10.6.255 Mask:255.255.255.0 <<<<== Management Network Interface inet6 addr: fe80::21b:21ff:febc:23f1/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1202254 errors:0 dropped:0 overruns:0 frame:0 TX packets:4896 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:97559060 (97.5 MB) TX bytes:1043945 (1.0 MB) Memory:dc7e0000-dc800000 }}} The following interfaces are configured on the compute node 2 (boscompute2/128.89.91.172): {{{ eth0 Link encap:Ethernet HWaddr 78:2b:cb:50:5c:00 inet addr:10.10.8.102 Bcast:10.10.8.255 Mask:255.255.255.0 <<<<== Control Network Interface inet6 addr: fe80::7a2b:cbff:fe50:5c00/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1934002 errors:0 dropped:64759 overruns:0 frame:0 TX packets:1051218 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1171016212 (1.1 GB) TX bytes:438688993 (438.6 MB) eth1 Link encap:Ethernet HWaddr 78:2b:cb:50:5c:01 inet addr:10.10.5.102 Bcast:10.10.5.255 Mask:255.255.255.0 <<<<== Data Network Interface inet6 addr: fe80::7a2b:cbff:fe50:5c01/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:143088 errors:0 dropped:4494 overruns:0 frame:0 TX packets:6321 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:17382441 (17.3 MB) TX bytes:1056197 (1.0 MB) eth2 Link encap:Ethernet HWaddr 00:1b:21:4b:3e:b4 inet addr:128.89.91.172 Bcast:128.89.91.191 Mask:255.255.255.224 <<<<== External Network Interface inet6 addr: fe80::21b:21ff:fe4b:3eb4/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:766841 errors:0 dropped:0 overruns:0 frame:0 TX packets:165589 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:566156761 (566.1 MB) TX bytes:16091017 (16.0 MB) Memory:df3c0000-df3e0000 eth3 Link encap:Ethernet HWaddr 00:1b:21:4b:3e:b5 inet addr:10.10.6.102 Bcast:0.0.0.0 Mask:255.255.255.255 <<<<== Management Network Interface inet6 addr: fe80::21b:21ff:fe4b:3eb5/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1438112 errors:0 dropped:0 overruns:0 frame:0 TX packets:14301 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:128257526 (128.2 MB) TX bytes:2862471 (2.8 MB) Memory:df3e0000-df400000 }}} The following interfaces are configured on the compute node 4 (boscompute4/128.89.91.174): {{{ eth0 Link encap:Ethernet HWaddr 78:2b:cb:51:a6:82 inet addr:10.10.8.104 Bcast:10.10.8.255 Mask:255.255.255.0 <<<<== Control Network Interface inet6 addr: fe80::7a2b:cbff:fe51:a682/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:772020 errors:0 dropped:7466 overruns:0 frame:0 TX packets:516318 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:550568324 (550.5 MB) TX bytes:191280624 (191.2 MB) eth1 Link encap:Ethernet HWaddr 78:2b:cb:51:a6:83 inet addr:10.10.5.104 Bcast:10.10.5.255 Mask:255.255.255.0 <<<<== Data Network Interface inet6 addr: fe80::7a2b:cbff:fe51:a683/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:10463 errors:0 dropped:106 overruns:0 frame:0 TX packets:396775 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1729203 (1.7 MB) TX bytes:35743239 (35.7 MB) eth2 Link encap:Ethernet HWaddr 00:1b:21:bc:22:b8 inet addr:128.89.91.174 Bcast:128.89.91.191 Mask:255.255.255.224 <<<<== External Network Interface inet6 addr: fe80::21b:21ff:febc:22b8/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:439246 errors:0 dropped:0 overruns:0 frame:0 TX packets:168175 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:577114949 (577.1 MB) TX bytes:13394855 (13.3 MB) Memory:dc7c0000-dc7e0000 eth3 Link encap:Ethernet HWaddr 00:1b:21:bc:22:b9 inet addr:10.10.6.104 Bcast:10.10.6.255 Mask:255.255.255.0 <<<<== Management Network Interface inet6 addr: fe80::21b:21ff:febc:22b9/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:488460 errors:0 dropped:0 overruns:0 frame:0 TX packets:11424 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:37649896 (37.6 MB) TX bytes:2529244 (2.5 MB) Memory:dc7e0000-dc800000 }}} == Step 3. Review MAC address == A site administrator reviews the MAC address table on the control plane switch, and verifies that all entries are identifiable and expected. On the control node listed all MAC addresses on the control network interface (eth0): {{{ lnevers@boscontroller:/var$ arp -i eth0 Address HWtype HWaddress Flags Mask Iface boscompute4.local ether 78:2b:cb:51:a6:82 C eth0 boscompute1.local ether 78:2b:cb:4f:62:6c C eth0 boscompute2.local ether 78:2b:cb:50:5c:00 C eth0 }}} On compute node 1 listed all MAC addresses on the control network interface (eth0): {{{ lnevers@boscompute1:~$ arp -i eth0 Address HWtype HWaddress Flags Mask Iface boscontroller.local ether 78:2b:cb:50:68:01 C eth0 }}} On compute node 2 listed all MAC addresses on the control network interface (eth0): {{{ lnevers@boscompute2:~$ arp -i eth0 Address HWtype HWaddress Flags Mask Iface boscontroller.local ether 78:2b:cb:50:68:01 C eth0 }}} On compute node 4 listed all MAC addresses on the control network interface (eth0): {{{ lnevers@boscompute4:~$ arp -i eth0 Address HWtype HWaddress Flags Mask Iface boscontroller.local ether 78:2b:cb:50:68:01 C eth0 }}} The above ARP entries show that only expected MAC addresses are in use: - control node has MAC 78:2b:cb:50:68:01 on eth0. - compute node 1 has MAC 78:2b:cb:4f:62:6c on eth0 - compute node 2 has MAC 78:2b:cb:50:5c:00 on eth0 - compute node 4 has MAC 78:2b:cb:51:a6:82 on eth0