Changes between Initial Version and Version 1 of GENIRacksHome/OpenGENIRacks/AcceptanceTestStatusMay2013/OG-MON-1


Ignore:
Timestamp:
05/23/14 11:35:49 (10 years ago)
Author:
lnevers@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • GENIRacksHome/OpenGENIRacks/AcceptanceTestStatusMay2013/OG-MON-1

    v1 v1  
     1= GR-MON-1: Control Network Software and VLAN Inspection Test =
     2
     3This page captures status for the test case GR-MON-1. For additional information see the [wiki:GENIRacksHome/GRAMRacks/AcceptanceTestStatusMay2013 Acceptance Test Status - May 2013] page overall status, or the [wiki:GENIRacksHome/AcceptanceTests/GRAMAcceptanceTestsPlan GRAM Acceptance Test Plan] for details about the planned evaluation.
     4
     5
     6''Last Update: 2013/05/15"
     7
     8|| '''Step''' || '''State'''||''' Notes '''            || '''Tickets'''  ||
     9|| Step 1     ||[[span(Pass: most criteria, style=background-color: #98FB98)]]||boscompute4 not working, git internal  || #62 #66           ||
     10|| Step 2     ||[[span(Pass, style=background-color: green )]]||                         ||                ||
     11|| Step 3     ||[[span(Pass, style=background-color: green )]]||                         ||                ||
     12       
     13
     14[[BR]]
     15|| '''State Legend'''                                   || '''Description'''                                                    ||
     16||[[span(Pass, style=background-color: green )]]        || Test completed and met all criteria                                  ||
     17||[[span(Pass: most criteria, style=background-color: #98FB98)]]|| Test completed and met most criteria. Exceptions documented  ||
     18||[[span(Fail, style=background-color: red)]]           || Test completed and failed to meet criteria.                          ||
     19||[[span(Complete, style=background-color: yellow)]]    || Test completed but will require re-execution due to expected changes ||
     20||[[span(Blocked, style=background-color: orange)]]     || Blocked by ticketed issue(s).                                        ||
     21||[[span(In Progress, style=background-color: #63B8FF)]]|| Currently under test.                                                ||
     22
     23= Test Plan Steps =
     24
     25== Step 1.  Identify processes and versions ==
     26
     27A site administrator enumerates processes on each of the server host, the Control VM, the VMOC VM, etc. and an experimental node configured for OpenStack, which listen for network connections from other nodes, identifies what version of what software package is in use for each, and verifies that we know the source of each piece of software and could get access to its source code.
     28
     29First get supported versions for the AM API and verify access to resources:
     30{{{
     31$ omni.py getversion -a gram -V3
     32INFO:omni:Loading config file /home/lnevers/.gcf/omni_config
     33INFO:omni:Using control framework gram
     34INFO:omni:Substituting AM nickname gram with URL https://128.89.91.170:5001, URN unspecified_AM_URN
     35INFO:omni:Substituting AM nickname gram with URL https://128.89.91.170:5001, URN unspecified_AM_URN
     36INFO:omni:AM URN: unspecified_AM_URN (url: https://128.89.91.170:5001) has version:
     37INFO:omni:{   'geni_ad_rspec_versions': [   {   'extensions': [],
     38                                      'namespace': 'http://www.geni.net/resources/rspec/3',
     39                                      'schema': 'http://www.geni.net/resources/rspec/3/ad.xsd',
     40                                      'type': 'GENI',
     41                                      'version': '3'}],
     42    'geni_api': 3,
     43    'geni_api_versions': {   '2': 'https://boscontroller.gpolab.bbn.com:5002',
     44                             '3': 'https://boscontroller.gpolab.bbn.com:5001'},
     45    'geni_credential_types': [{   'geni_type': 'geni_sfa',
     46                                     'geni_version': '3'}],
     47    'geni_request_rspec_versions': [   {   'extensions': [],
     48                                           'namespace': 'http://www.geni.net/resources/rspec/3',
     49                                           'schema': 'http://www.geni.net/resources/rspec/3/request.xsd',
     50                                           'type': 'GENI',
     51                                           'version': '3'}]}
     52INFO:omni: ------------------------------------------------------------
     53INFO:omni: Completed getversion:
     54
     55  Options as run:
     56                aggregate: ['gram']
     57                api_version: 3
     58                framework: gram
     59                noGetVersionCache: True
     60
     61  Args: getversion
     62
     63  Result Summary:
     64Got version for https://128.89.91.170:5001
     65 
     66INFO:omni: ============================================================
     67}}}
     68Then verify that resources are available for experimenters:
     69{{{
     70$ omni.py listresources -a gram  -o
     71}}}
     72'''Note: Fails with ticket #53
     73
     74
     75Based on the [https://superior.bbn.com/trac/bbn-rack/wiki/ArchitectureDescription GRAM Network and Software Architecture Description] document, the following are major services for GRAM:
     76 - ''gram-ch'' - a test clearinghouse to be replaced by GENI clearinghouse.
     77 - ''gram-am'' - the aggregate manager that handles AM API calls for resources
     78 - ''gram-vnoc'' - VMOC controller that manages OpenFlow traffic for switch(es) and experimenter controller(s). 
     79 - ''gram-ctrl''- Default controller that performs layer 2 MAC to port learning.
     80
     81The GRAM software is installed in /home/gram/gram on the control node (128.89.91.170) and the GRAM services listed above are started by scripts in the /home/gram/gram/src/services directory.  All services were verified to be running.
     82
     83The gram-vmoc processes:
     84{{{
     85gram     18694     1  0 May10 ?        00:09:55 python2.7 -u -O /opt/pox/pox.py log.level --DEBUG vmoc.VMOC --management_port=7001 --default_controller_url=https://localhost:9000
     86}}}
     87
     88The gram-ctl processes default controller:
     89{{{
     90gram     18680     1  0 May10 ?        00:05:31 python2.7 -u -O /opt/pox/pox.py log.level --DEBUG openflow.of_01 --port=9000 vmoc.l2_simple_learning
     91}}}
     92
     93The gram-am processes goth V3 (default) and V2:
     94{{{
     95gram     17475     1  0 07:00 ?        00:00:00 su -s /bin/sh -c exec python /home/gram/gram/src/gram-am.py gram
     96gram     17482 17475  0 07:00 ?        00:00:00 python /home/gram/gram/src/gram-am.py
     97gram     17526     1  0 07:00 ?        00:00:00 su -s /bin/sh -c exec python /home/gram/gram/src/gram-am.py -V 2 gram
     98gram     17533 17526  0 07:00 ?        00:00:00 python /home/gram/gram/src/gram-am.py -V 2
     99}}}
     100
     101The clearinghouse process:
     102{{{
     103gram     17360     1  0 07:00 ?        00:00:00 su -s /bin/sh -c exec python /opt/gcf/src/gcf-ch.py gram
     104}}}
     105
     106To check for all processes that are listening:
     107{{{
     108lnevers@boscontroller:~$ sudo netstat -anp | egrep ':\*.+LISTEN' | egrep -v '127.0.0.1' >netstat-raw.txt
     109lnevers@boscontroller:~$ cat netstat-raw.txt
     110tcp        0      0 0.0.0.0:9292            0.0.0.0:*               LISTEN      8982/python     
     111tcp        0      0 0.0.0.0:4369            0.0.0.0:*               LISTEN      7530/epmd       
     112tcp        0      0 10.0.36.2:53            0.0.0.0:*               LISTEN      17692/dnsmasq   
     113tcp        0      0 192.168.10.2:53         0.0.0.0:*               LISTEN      9444/dnsmasq   
     114tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      18090/sshd     
     115tcp        0      0 0.0.0.0:7001            0.0.0.0:*               LISTEN      18694/python2.7
     116tcp        0      0 0.0.0.0:35357           0.0.0.0:*               LISTEN      8631/python     
     117tcp        0      0 128.89.91.170:8000      0.0.0.0:*               LISTEN      25968/python   
     118tcp        0      0 0.0.0.0:9696            0.0.0.0:*               LISTEN      18143/python   
     119tcp        0      0 0.0.0.0:6080            0.0.0.0:*               LISTEN      4428/python     
     120tcp        0      0 0.0.0.0:36356           0.0.0.0:*               LISTEN      7562/beam.smp   
     121tcp        0      0 0.0.0.0:8773            0.0.0.0:*               LISTEN      9058/python     
     122tcp        0      0 0.0.0.0:8774            0.0.0.0:*               LISTEN      9058/python     
     123tcp        0      0 0.0.0.0:8775            0.0.0.0:*               LISTEN      9058/python     
     124tcp        0      0 0.0.0.0:9191            0.0.0.0:*               LISTEN      8990/python     
     125tcp        0      0 0.0.0.0:9000            0.0.0.0:*               LISTEN      18680/python2.7
     126tcp        0      0 0.0.0.0:5000            0.0.0.0:*               LISTEN      8631/python     
     127tcp        0      0 128.89.91.170:5001      0.0.0.0:*               LISTEN      25822/python   
     128tcp        0      0 0.0.0.0:6633            0.0.0.0:*               LISTEN      18694/python2.7
     129tcp        0      0 128.89.91.170:5002      0.0.0.0:*               LISTEN      25887/python   
     130tcp        0      0 0.0.0.0:3306            0.0.0.0:*               LISTEN      8427/mysqld     
     131tcp6       0      0 fe80::f816:3eff:feed:53 :::*                    LISTEN      17692/dnsmasq   
     132tcp6       0      0 fe80::f816:3eff:fe72:53 :::*                    LISTEN      9444/dnsmasq   
     133tcp6       0      0 :::22                   :::*                    LISTEN      18090/sshd     
     134tcp6       0      0 ::1:631                 :::*                    LISTEN      1034/cupsd     
     135tcp6       0      0 ::1:6010                :::*                    LISTEN      30396/22       
     136tcp6       0      0 :::5672                 :::*                    LISTEN      7562/beam.smp   
     137}}}
     138
     139Stripping list of processes to get a list of PIDs, and determine what the processes that are listening:
     140{{{
     141lnevers@boscontroller:~$ sudo netstat -anp | egrep ':\*.+LISTEN' | egrep -v '127.0.0.1' >netstat-raw.txt
     142lnevers@boscontroller:~$ awk '{print $NF}' netstat-raw.txt | sed -r -e 's|/.*||' | sort -n | uniq > netstat-pids.txt
     143lnevers@boscontroller:~$ for pid in $(cat netstat-pids.txt) ; do ps -efwww | egrep "^[^ ]+ +$pid" ; done >& netstat-commands.txt
     144lnevers@boscontroller:~$ cat netstat-commands.txt
     145root      1034     1  0 May10 ?        00:00:00 /usr/sbin/cupsd -F
     146nova      4428  4420  0 May10 ?        00:00:21 /usr/bin/python /usr/bin/nova-novncproxy --config-file=/etc/nova/nova.conf
     147rabbitmq  7530     1  0 May10 ?        00:00:02 /usr/lib/erlang/erts-5.8.5/bin/epmd -daemon
     148rabbitmq  7562     1  0 May10 ?        00:06:00 /usr/lib/erlang/erts-5.8.5/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -noshell -noinput -sname rabbit@boscontroller -boot /var/lib/rabbitmq/mnesia/rabbit@boscontroller-plugins-expand/rabbit -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@boscontroller.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@boscontroller-sasl.log"} -os_mon start_cpu_sup true -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@boscontroller"
     149mysql     8427     1  1 May10 ?        01:52:51 /usr/sbin/mysqld
     150keystone  8631  8623  5 May10 ?        06:13:28 /usr/bin/python /usr/bin/keystone-all
     151glance    8982  8963  0 May10 ?        00:00:00 /usr/bin/python /usr/bin/glance-api
     152glance    8990  8981  0 May10 ?        00:00:00 /usr/bin/python /usr/bin/glance-registry
     153nova      9058  9039  0 May10 ?        00:00:01 /usr/bin/python /usr/bin/nova-api --config-file=/etc/nova/nova.conf
     154nobody    9444     1  0 May10 ?        00:00:01 /usr/sbin/dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tap350960c0-bf --except-interface=lo --domain=openstacklocal --pid-file=/var/lib/quantum/dhcp/f80c6090-4321-4ffa-8f8b-eb9f897bd4ca/pid --dhcp-hostsfile=/var/lib/quantum/dhcp/f80c6090-4321-4ffa-8f8b-eb9f897bd4ca/host --dhcp-optsfile=/var/lib/quantum/dhcp/f80c6090-4321-4ffa-8f8b-eb9f897bd4ca/opts --dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update --leasefile-ro --dhcp-range=set:tag0,192.168.10.0,static,120s --conf-file= --server=8.8.8.8
     155nobody   17692     1  0 May14 ?        00:00:01 /usr/sbin/dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --interface=tapb48adcd2-64 --except-interface=lo --domain=openstacklocal --pid-file=/var/lib/quantum/dhcp/1072ae37-d6e2-46a6-be76-234db45cc907/pid --dhcp-hostsfile=/var/lib/quantum/dhcp/1072ae37-d6e2-46a6-be76-234db45cc907/host --dhcp-optsfile=/var/lib/quantum/dhcp/1072ae37-d6e2-46a6-be76-234db45cc907/opts --dhcp-script=/usr/bin/quantum-dhcp-agent-dnsmasq-lease-update --leasefile-ro --dhcp-range=set:tag0,10.0.36.0,static,120s --conf-file= --server=8.8.8.8
     156root     18090     1  0 May10 ?        00:00:00 /usr/sbin/sshd -D
     157quantum  18143     1  1 May10 ?        01:42:41 python /usr/bin/quantum-server --config-file /etc/quantum/quantum.conf --log-file /var/log/quantum/server.log --config-file /etc/quantum/plugins/openvswitch/ovs_quantum_plugin.ini
     158gram     18680     1  0 May10 ?        00:05:37 python2.7 -u -O /opt/pox/pox.py log.level --DEBUG openflow.of_01 --port=9000 vmoc.l2_simple_learning
     159gram     18694     1  0 May10 ?        00:10:07 python2.7 -u -O /opt/pox/pox.py log.level --DEBUG vmoc.VMOC --management_port=7001 --default_controller_url=https://localhost:9000
     160gram     25822 25815  0 08:46 ?        00:00:01 python /home/gram/gram/src/gram-am.py
     161gram     25887 25880  0 08:46 ?        00:00:01 python /home/gram/gram/src/gram-am.py -V 2
     162gram     25968 25961  0 08:46 ?        00:00:01 python /opt/gcf/src/gcf-ch.py
     163lnevers  30396 30235  0 09:41 ?        00:00:00 sshd: lnevers@pts/22
     164lnevers@boscontroller:~$
     165}}}
     166
     167
     168Additionally GRAM software configuration is captured on the control node:
     169{{{
     170lnevers@boscontroller:~$ more  /etc/gram/config.json
     171{
     172   "__001": "This file contains GRAM installation-specific",
     173   "__002": "parameters and structures. The values contained here",
     174   "__003": "override values in the gram.am.gram.config module.",
     175   "__004": "That is, a key/value pair  K, V  in this file",
     176   "__005": "sets gram.am.config.K = V",
     177   "__006": "An error is given if the field (K) does not exist, ",
     178   "__007": "Or the type of the value cannot be coerced",
     179   "__008": "Further any parameter whose value is None in config.py",
     180   "__009": "and set as null here must be set in this file:",
     181   "__010": "no default is provided",
     182
     183
     184   "disk_image_metadata":
     185   {
     186   "ubuntu-2nic":
     187       {
     188        "os": "Linux",
     189        "version": "12.0",
     190        "description":"Ubuntu image with 2 NICs configured"
     191        },
     192   "cirros-2nic-x86_64":
     193       {
     194        "os": "Linux",
     195        "version": "12.0",
     196        "description":"Cirros image with 2 NICs configured"
     197        }
     198   },
     199
     200   "external_interface": "eth2",
     201   "external_address": "128.89.91.170",
     202   "external_netmask": "255.255.255.224",
     203   "control_address": "10.10.8.100",
     204   "control_interface": "eth0",
     205   "control_netmask": "255.255.255.0",
     206   "data_interface": "eth1",
     207   "management_interface": "eth3",
     208   "management_address": "10.10.6.100",
     209   "management_netmask": "255.255.255.0",
     210   "mysql_user": "admin",
     211   "mysql_password": "XXX",
     212   "rabbit_password": "XXX",
     213   "nova_password": "XXX",
     214   "glance_password": "XXXX",
     215   "keystone_password": "XXXX",
     216   "quantum_password": "XXXX",
     217   "os_tenant_name": "admin",
     218   "os_username": "admin",
     219   "os_password": "XXXX",
     220   "os_auth_url": "http://localhost:5000/v2.0/",
     221   "os_region_name" : "RegionOne",
     222   "os_no_cache" : "1",
     223   "service_token" : "bos",
     224   "service_endpoint" : "http://localhost:35357/v2.0",
     225
     226   "public_gateway_ip" : "128.89.91.162",
     227   "public_subnet_cidr" : "128.89.91.160/27",
     228   "public_subnet_start_ip" : "128.89.91.163",
     229   "public_subnet_end_ip" : "128.89.91.169",
     230
     231   "metadata_port" : "8775",
     232
     233   "backup_directory" : "/home/gram/.backup",
     234   "control_host" : "boscontroller",
     235   "compute_hosts": {
     236            "boscompute1": "10.10.8.101",
     237            "boscompute2": "10.10.8.102",
     238            "boscompute4": "10.10.8.104"},
     239   "glance_images" : []
     240
     241}
     242}}}
     243
     244'''Note:''' There are 3 configured compute nodes, but only two are being used. boscompute4 not working and needs to be restored.
     245
     246Access to code is possible as follows:
     247
     248 - on the GRAM nodes, source is available in /home/gram/gram/src.
     249 - on the git repository  git@superior.bbn.com:gram.git and credentials can be requested from gram-dev@bbn.com.
     250
     251''Note:'' GIT repository is currently internal only
     252
     253GRAM Software version:
     254{{{
     255lnevers@boscontroller:~$ dpkg -p gram
     256Package: gram
     257Priority: optional
     258Maintainer: GRAM Developers <gram-dev@bbn.com>
     259Architecture: all
     260Version: 05102013B
     261Depends: git, cloud-utils, kvm, xmlsec1, python-software-properties, ubuntu-cloud-keyring, ntp, mysql-server, python-mysqldb, rabbitmq-server, keystone, python-keystone, p
     262Size: 1921726
     263Description: GRAM installation package
     264lnevers@boscontroller:~$
     265}}}
     266
     267
     268
     269== Step 2. Review control network configuration ==
     270
     271A site administrator reviews the configuration of the rack control plane switch and verifies that each experimental node's control and console access interfaces are on the expected VLANs.  According to [https://superior.bbn.com/trac/bbn-rack/wiki/ArchitectureDescription GRAM Network and Software Architecture Description] document interfaces are configured as follows on the GRAM nodes:
     272    - eth0: Control network configured for 10.10.8.0/24
     273    - eth1 : Data network configured for 10.10.5.0/24
     274    - eth2 : External network configured for 128.89.91.0/27
     275    - eth3 : Management network configured for 10.10.6.0/24
     276
     277=== Control Node ===
     278The following interfaces are configured on the control node (boscontroller/128.89.91.170):
     279{{{
     280eth0      Link encap:Ethernet  HWaddr 78:2b:cb:50:68:01 
     281          inet addr:10.10.8.100  Bcast:10.10.8.255  Mask:255.255.255.0   <<<<== Control Network Interface
     282          inet6 addr: fe80::7a2b:cbff:fe50:6801/64 Scope:Link
     283          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     284          RX packets:2558290 errors:0 dropped:188657 overruns:0 frame:0
     285          TX packets:2770380 errors:0 dropped:0 overruns:0 carrier:0
     286          collisions:0 txqueuelen:1000
     287          RX bytes:1005762310 (1.0 GB)  TX bytes:2334137468 (2.3 GB)
     288
     289eth1      Link encap:Ethernet  HWaddr 78:2b:cb:50:68:02 
     290          inet addr:10.10.5.100  Bcast:10.10.5.255  Mask:255.255.255.0   <<<<== Data Network Interface
     291          inet6 addr: fe80::7a2b:cbff:fe50:6802/64 Scope:Link
     292          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     293          RX packets:73830 errors:0 dropped:137 overruns:0 frame:0
     294          TX packets:3228 errors:0 dropped:0 overruns:0 carrier:0
     295          collisions:0 txqueuelen:1000
     296          RX bytes:8326147 (8.3 MB)  TX bytes:562478 (562.4 KB)
     297
     298eth3      Link encap:Ethernet  HWaddr 00:1b:21:bc:27:11 
     299          inet addr:10.10.6.100  Bcast:10.10.6.255  Mask:255.255.255.0  <<<<== Management Network Interface
     300          inet6 addr: fe80::21b:21ff:febc:2711/64 Scope:Link
     301          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     302          RX packets:1141956 errors:0 dropped:0 overruns:0 frame:0
     303          TX packets:6392 errors:0 dropped:0 overruns:0 carrier:0
     304          collisions:0 txqueuelen:1000
     305          RX bytes:91674129 (91.6 MB)  TX bytes:814927 (814.9 KB)
     306          Memory:dc7e0000-dc800000
     307
     308}}}
     309
     310 
     311=== Compute Nodes ===
     312The following interfaces are configured on the compute node 1 (boscompute1/128.89.91.171):
     313{{{
     314eth0      Link encap:Ethernet  HWaddr 78:2b:cb:4f:62:6c   
     315          inet addr:10.10.8.101  Bcast:10.10.8.255  Mask:255.255.255.0       <<<<== Control Network Interface
     316          inet6 addr: fe80::7a2b:cbff:fe4f:626c/64 Scope:Link
     317          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     318          RX packets:1415472 errors:0 dropped:61316 overruns:0 frame:0
     319          TX packets:728707 errors:0 dropped:0 overruns:0 carrier:0
     320          collisions:0 txqueuelen:1000
     321          RX bytes:732279308 (732.2 MB)  TX bytes:325014011 (325.0 MB)
     322
     323eth1      Link encap:Ethernet  HWaddr 78:2b:cb:4f:62:6d 
     324          inet addr:10.10.5.101  Bcast:10.10.5.255  Mask:255.255.255.0       <<<<== Data Network Interface
     325          inet6 addr: fe80::7a2b:cbff:fe4f:626d/64 Scope:Link
     326          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     327          RX packets:93698 errors:0 dropped:4435 overruns:0 frame:0
     328          TX packets:1397 errors:0 dropped:0 overruns:0 carrier:0
     329          collisions:0 txqueuelen:1000
     330          RX bytes:12247041 (12.2 MB)  TX bytes:254376 (254.3 KB)
     331
     332eth2      Link encap:Ethernet  HWaddr 00:1b:21:bc:23:f0 
     333          inet addr:128.89.91.171  Bcast:128.89.91.191  Mask:255.255.255.224    <<<<== External Network Interface
     334          inet6 addr: fe80::21b:21ff:febc:23f0/64 Scope:Link
     335          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     336          RX packets:753699 errors:0 dropped:0 overruns:0 frame:0
     337          TX packets:161176 errors:0 dropped:0 overruns:0 carrier:0
     338          collisions:0 txqueuelen:1000
     339          RX bytes:583999876 (583.9 MB)  TX bytes:13922812 (13.9 MB)
     340          Memory:dc7c0000-dc7e0000
     341
     342eth3      Link encap:Ethernet  HWaddr 00:1b:21:bc:23:f1 
     343          inet addr:10.10.6.101  Bcast:10.10.6.255  Mask:255.255.255.0       <<<<== Management Network Interface
     344          inet6 addr: fe80::21b:21ff:febc:23f1/64 Scope:Link
     345          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     346          RX packets:1202254 errors:0 dropped:0 overruns:0 frame:0
     347          TX packets:4896 errors:0 dropped:0 overruns:0 carrier:0
     348          collisions:0 txqueuelen:1000
     349          RX bytes:97559060 (97.5 MB)  TX bytes:1043945 (1.0 MB)
     350          Memory:dc7e0000-dc800000
     351}}}
     352
     353The following interfaces are configured on the compute node 2 (boscompute2/128.89.91.172):
     354{{{
     355eth0      Link encap:Ethernet  HWaddr 78:2b:cb:50:5c:00 
     356          inet addr:10.10.8.102  Bcast:10.10.8.255  Mask:255.255.255.0      <<<<== Control Network Interface
     357          inet6 addr: fe80::7a2b:cbff:fe50:5c00/64 Scope:Link
     358          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     359          RX packets:1934002 errors:0 dropped:64759 overruns:0 frame:0
     360          TX packets:1051218 errors:0 dropped:0 overruns:0 carrier:0
     361          collisions:0 txqueuelen:1000
     362          RX bytes:1171016212 (1.1 GB)  TX bytes:438688993 (438.6 MB)
     363
     364eth1      Link encap:Ethernet  HWaddr 78:2b:cb:50:5c:01 
     365          inet addr:10.10.5.102  Bcast:10.10.5.255  Mask:255.255.255.0      <<<<== Data Network Interface
     366          inet6 addr: fe80::7a2b:cbff:fe50:5c01/64 Scope:Link
     367          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     368          RX packets:143088 errors:0 dropped:4494 overruns:0 frame:0
     369          TX packets:6321 errors:0 dropped:0 overruns:0 carrier:0
     370          collisions:0 txqueuelen:1000
     371          RX bytes:17382441 (17.3 MB)  TX bytes:1056197 (1.0 MB)
     372
     373eth2      Link encap:Ethernet  HWaddr 00:1b:21:4b:3e:b4 
     374          inet addr:128.89.91.172  Bcast:128.89.91.191  Mask:255.255.255.224   <<<<== External Network Interface
     375          inet6 addr: fe80::21b:21ff:fe4b:3eb4/64 Scope:Link
     376          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     377          RX packets:766841 errors:0 dropped:0 overruns:0 frame:0
     378          TX packets:165589 errors:0 dropped:0 overruns:0 carrier:0
     379          collisions:0 txqueuelen:1000
     380          RX bytes:566156761 (566.1 MB)  TX bytes:16091017 (16.0 MB)
     381          Memory:df3c0000-df3e0000
     382
     383eth3      Link encap:Ethernet  HWaddr 00:1b:21:4b:3e:b5 
     384          inet addr:10.10.6.102  Bcast:0.0.0.0  Mask:255.255.255.255          <<<<== Management Network Interface
     385          inet6 addr: fe80::21b:21ff:fe4b:3eb5/64 Scope:Link
     386          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     387          RX packets:1438112 errors:0 dropped:0 overruns:0 frame:0
     388          TX packets:14301 errors:0 dropped:0 overruns:0 carrier:0
     389          collisions:0 txqueuelen:1000
     390          RX bytes:128257526 (128.2 MB)  TX bytes:2862471 (2.8 MB)
     391          Memory:df3e0000-df400000
     392
     393}}}
     394The following interfaces are configured on the compute node 4 (boscompute4/128.89.91.174):
     395{{{
     396eth0      Link encap:Ethernet  HWaddr 78:2b:cb:51:a6:82 
     397          inet addr:10.10.8.104  Bcast:10.10.8.255  Mask:255.255.255.0           <<<<== Control Network Interface
     398          inet6 addr: fe80::7a2b:cbff:fe51:a682/64 Scope:Link
     399          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     400          RX packets:772020 errors:0 dropped:7466 overruns:0 frame:0
     401          TX packets:516318 errors:0 dropped:0 overruns:0 carrier:0
     402          collisions:0 txqueuelen:1000
     403          RX bytes:550568324 (550.5 MB)  TX bytes:191280624 (191.2 MB)
     404
     405eth1      Link encap:Ethernet  HWaddr 78:2b:cb:51:a6:83 
     406          inet addr:10.10.5.104  Bcast:10.10.5.255  Mask:255.255.255.0            <<<<== Data Network Interface
     407          inet6 addr: fe80::7a2b:cbff:fe51:a683/64 Scope:Link
     408          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     409          RX packets:10463 errors:0 dropped:106 overruns:0 frame:0
     410          TX packets:396775 errors:0 dropped:0 overruns:0 carrier:0
     411          collisions:0 txqueuelen:1000
     412          RX bytes:1729203 (1.7 MB)  TX bytes:35743239 (35.7 MB)
     413
     414eth2      Link encap:Ethernet  HWaddr 00:1b:21:bc:22:b8 
     415          inet addr:128.89.91.174  Bcast:128.89.91.191  Mask:255.255.255.224      <<<<== External Network Interface
     416          inet6 addr: fe80::21b:21ff:febc:22b8/64 Scope:Link
     417          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     418          RX packets:439246 errors:0 dropped:0 overruns:0 frame:0
     419          TX packets:168175 errors:0 dropped:0 overruns:0 carrier:0
     420          collisions:0 txqueuelen:1000
     421          RX bytes:577114949 (577.1 MB)  TX bytes:13394855 (13.3 MB)
     422          Memory:dc7c0000-dc7e0000
     423
     424eth3      Link encap:Ethernet  HWaddr 00:1b:21:bc:22:b9 
     425          inet addr:10.10.6.104  Bcast:10.10.6.255  Mask:255.255.255.0     <<<<== Management Network Interface
     426          inet6 addr: fe80::21b:21ff:febc:22b9/64 Scope:Link
     427          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
     428          RX packets:488460 errors:0 dropped:0 overruns:0 frame:0
     429          TX packets:11424 errors:0 dropped:0 overruns:0 carrier:0
     430          collisions:0 txqueuelen:1000
     431          RX bytes:37649896 (37.6 MB)  TX bytes:2529244 (2.5 MB)
     432          Memory:dc7e0000-dc800000
     433
     434}}}
     435
     436
     437== Step 3. Review MAC address ==
     438
     439A site administrator reviews the MAC address table on the control plane switch, and verifies that all entries are identifiable and expected.
     440
     441On the control node listed all MAC addresses on the control network interface (eth0):
     442{{{
     443lnevers@boscontroller:/var$ arp -i eth0
     444Address                  HWtype  HWaddress           Flags Mask            Iface
     445boscompute4.local        ether   78:2b:cb:51:a6:82   C                     eth0
     446boscompute1.local        ether   78:2b:cb:4f:62:6c   C                     eth0
     447boscompute2.local        ether   78:2b:cb:50:5c:00   C                     eth0
     448}}}
     449
     450On compute node 1 listed all MAC addresses on the control network interface (eth0):
     451{{{
     452lnevers@boscompute1:~$ arp -i eth0
     453Address                  HWtype  HWaddress           Flags Mask            Iface
     454boscontroller.local      ether   78:2b:cb:50:68:01   C                     eth0
     455}}}
     456On compute node 2 listed all MAC addresses on the control network interface (eth0):
     457{{{
     458lnevers@boscompute2:~$ arp -i eth0
     459Address                  HWtype  HWaddress           Flags Mask            Iface
     460boscontroller.local      ether   78:2b:cb:50:68:01   C                     eth0
     461}}}
     462
     463On compute node 4 listed all MAC addresses on the control network interface (eth0):
     464{{{
     465lnevers@boscompute4:~$ arp -i eth0
     466Address                  HWtype  HWaddress           Flags Mask            Iface
     467boscontroller.local      ether   78:2b:cb:50:68:01   C                     eth0
     468}}}
     469
     470The above ARP entries show that only expected MAC addresses are in use:
     471   - control node has MAC 78:2b:cb:50:68:01 on eth0.
     472   - compute node 1 has MAC 78:2b:cb:4f:62:6c on eth0
     473   - compute node 2 has MAC 78:2b:cb:50:5c:00 on eth0
     474   - compute node 4 has MAC 78:2b:cb:51:a6:82 on eth0