incorrect MAC addresses are reported for dataplane interfaces on OpenVZ containers

[chaos@bbn.com]

I am comparing three sources of information about MAC addresses assigned to dataplane interfaces on experimental nodes:

• The output of /sbin/ifconfig on the node itself (which should be authoritative)
• The output of sliverstatus run by the experimenter on the slice to which the node belongs
• The experiment details page as viewed by an admin, e.g. ​https://boss.utah.geniracks.net/showexp.php3?experiment=363#details

These three data sources agree given a physical node. However, given an OpenVZ container, the second data source is incorrect, and the third is missing.

I've only tried to verify this once, using the sliver pgeni-gpolab-bbn-com/ecgtest, which contains the mapping:

Physical Node Mapping:
ID              Type         OS              Physical
--------------- ------------ --------------- ------------
phys1           dl360        FEDORA15-STD    pc3
virt1           pcvm         OPENVZ-STD      pcvm5-1 (pc5)

For this experiment, i see:

• For the physical node interface, everything is correct:

  MAC addrs reported for phys1:0 == 10.10.1.1
    E8:39:35:B1:4E:8A: from /sbin/ifconfig eth1 run on phys1 (authoritative)
    e83935b14e8a:      from sliverstatus as experimenter (correct)
    e8:39:35:b1:4e:8a: from: https://boss.utah.geniracks.net/showexp.php3?experiment=363#details (correct)
  

• For the virtual node interface, sliverstatus reports an incorrect MAC, and showexp.php3 reports no MAC at all:

  MAC addrs reported for virt1:0 == 10.10.1.2
    82:01:0A:0A:01:02: from /sbin/ifconfig mv1.1 run on virt1 (authoritative)
    00000a0a0102:      from sliverstatus as experimenter (incorrect: first four digits are wrong)
    -                : from https://boss.utah.geniracks.net/showexp.php3?experiment=363#details (not reported)
  

[chaos@bbn.com]

On Friday, Leigh said:

> * The output of sliverstatus run by the experimenter on the slice to
> which the node belongs

This was fixed by Jon last week, but it is possible that I have not             
done an install on the rack since he pushed it up. I just did an                
install, so give it a try.

Hmm. I tried this out today incidentally while starting up and shutting down experiments on physical and OpenVZ nodes. It looks to me like:

• Physical nodes still work fine: here's a relevant attributes block, excerpted from ecgtest3-sliverstatus-www-utah-geniracks-net-protogeni.json:

    {'attributes': {'client_id': 'phys1:if0',
                    'component_id': 'urn:publicid:IDN+utah.geniracks.net+interface+pc5:eth1',
                    'mac_address': 'e4115bed1cb6',
                    'sliver_id': 'urn:publicid:IDN+utah.geniracks.net+sliver+782'},
                    'children': [{'attributes': {'address': '10.10.1.1',
                                                 'type': 'ipv4'},
                                                 'children': [],
                                                 'name': 'ip'}],
                    'name': 'interface'},
  

• OpenVZ nodes now don't report a MAC address at all: here's a relevant attributes blocked, excerpted from ecgtest2-sliverstatus-www-utah-geniracks-net-protogeni.json:

    {'attributes': {'client_id': 'virt1:if0',
                    'component_id': 'urn:publicid:IDN+utah.geniracks.net+interface+pc3:lo0',
                    'sliver_id': 'urn:publicid:IDN+utah.geniracks.net+sliver+776'},
                    'children': [{'attributes': {'address': '10.10.1.1',
                                                 'type': 'ipv4'},
                                                 'children': [],
                                                 'name': 'ip'}],
                    'name': 'interface'},
  

I'm attaching those two files to this ticket. Here are the timestamps (in EDT) at which each was captured via omni sliverstatus using the -o option:

-rw-r--r-- 1 chaos chaos 16160 May 21 14:32 ecgtest2-sliverstatus-www-utah-geniracks-net-protogeni.json
-rw-r--r-- 1 chaos chaos 16410 May 21 14:53 ecgtest3-sliverstatus-www-utah-geniracks-net-protogeni.json

As a reminder: on Friday, OpenVZ nodes did report a MAC address, but one which was slightly incorrect.

[chaos@bbn.com]

sliverstatus on an experiment with one physical and one virtual node: mac address on virtual is still wrong, but both MACs are reported

[chaos@bbn.com]

I don't think this is entirely fixed yet:

• It is now the case that the MAC address is reported both by sliverstatus and by the admin UI, so that part seems to be fixed.
• However, both of those reports still show the dataplane MAC address for virt1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net as 00000a0a0102, while that VM itself sees its dataplane MAC address as:

  [chaos@virt1 ~]$ /sbin/ifconfig mv1.1
  mv1.1     Link encap:Ethernet  HWaddr 82:01:0A:0A:01:02
            inet addr:10.10.1.2  Bcast:10.10.1.255  Mask:255.255.255.0
            inet6 addr: fe80::8001:aff:fe0a:102/64 Scope:Link
            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
            RX packets:10215105 errors:0 dropped:0 overruns:0 frame:0
            TX packets:7722087 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:0 
            RX bytes:126832638471 (118.1 GiB)  TX bytes:558860501 (532.9 MiB)
  

The first four hex digits of the address are still different on the VM itself vs. in reporting.

This seems likely to have something to do with how Emulab gets information from OpenVZ (or, for that matter, with how OpenVZ fabricates or translates MAC addresses), rather than with the UI, since it is consistent between the two methods of getting the information.

[chaos@bbn.com]

Leigh said on 30 May that this is fixed. I need to retest it.

[chaos@bbn.com]

I'm sorry i've been so slow to retest this, but i am still seeing weird behavior.

I created a sliver just now, slice ecgtest2, and was able to replicate a behavior i have seen sometimes before:

• I create an experiment containing two OpenVZ containers and no physical nodes, and sliverstatus does not report MAC addresses at all.
• The log of what i did is at ​http://groups.geni.net/geni/wiki/GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-MON-3#Resultsoftestingstep1:2012-06-07, and the experiment is still running.
• If i look in ​https://boss.utah.geniracks.net/showexp.php3?experiment=1069#details, the MAC addresses are blank there too

Some things that are interesting about this test:

• Both containers were created on the same host (pc5)
• There is no physical host

I'll create some different experiments now, and see if i can nail down further when this happens.

[chaos@bbn.com]

Leigh said:

> * If i look in
>  [https://boss.utah.geniracks.net/showexp.php3?experiment=1069#details],      
>  the MAC addresses are blank there too

Well, this part is fixed; wrong case on a variable.

Regarding sliverstatus. So the manifest does not have the MAC addresses?

Verified: ​https://boss.utah.geniracks.net/showexp.php3?experiment=1069#details now shows MAC addresses, indeed, correct MAC addresses:

• virt1 has:

  mv3.3     Link encap:Ethernet  HWaddr 02:00:70:B7:95:54  
            inet addr:10.10.1.1  Bcast:10.10.1.255  Mask:255.255.255.0
  

• virt2 has:

  mv4.4     Link encap:Ethernet  HWaddr 02:00:83:CF:E6:09  
            inet addr:10.10.1.2  Bcast:10.10.1.255  Mask:255.255.255.0
  

and the page contains those.

Sliverstatus still doesn't have this. I'll attach my full sliverstatus output for ecgtest2 to the ticket.

[chaos@bbn.com]

sliverstatus on an experiment with two openvz containers, no mac_address fields

[ricci@cs.utah.edu]

We believe this is fixed, but it should probably be re-tested.

[chaos@bbn.com]

I still need to test this, and will try to do so by the end of the week. Sorry for the slow turnaround as usual.

[chaos@bbn.com]

Ugh. I'm really sorry to be so slow retesting this, especially since sliverstatus output still doesn't seem to contain mac addresses for my VMs. I'll attach sliverstatus for my new ecgtest2 sliver, which exists right now, and you can see it at ​https://boss.utah.geniracks.net/showexp.php3?pid=pgeni-gpolab-bbn-com&eid=ecgtest2#details.

I have to run to a meeting, but i will try to more clearly summarize what i have and haven't seen in sliverstatus at various points after that meeting, since i think "there's no mac address" is probably somewhat underspecified.

[chaos@bbn.com]

Okay. A long time ago, Leigh said:

Note that mac addresses are in the *manifest* which is different then           
what sliverstatus returns.

I didn't understand that, but have had it explained to me that, if sliverstatus doesn't give me information that is useful, i should consider trying listresources on the slice name and see if that differs.

So i'm going to try to reproduce this once again and wrap it up. I will do:

• Create a slice, ecgtest, and a sliver using this rspec:

  <?xml version="1.0" encoding="UTF-8"?>
  <!-- This rspec will reserve one physical node and one openvz node, each
       with no OS specified, and create a single dataplane link between
       them.  It should work on any Emulab which has nodes available and
       supports OpenVZ.  -->
  <rspec xmlns="http://www.geni.net/resources/rspec/3"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://www.geni.net/resources/rspec/3
                             http://www.geni.net/resources/rspec/3/request.xsd"
         type="request">
  
    <node client_id="phys1" exclusive="true">
      <sliver_type name="raw" />
      <interface client_id="phys1:if0" />
    </node>
    <node client_id="virt1" exclusive="false">
      <sliver_type name="emulab-openvz" />
      <interface client_id="virt1:if0" />
    </node>
  
    <link client_id="phys1-virt1-0">
      <interface_ref client_id="phys1:if0"/>
      <interface_ref client_id="virt1:if0"/>
      <property source_id="phys1:if0" dest_id="virt1:if0"/>
      <property source_id="virt1:if0" dest_id="phys1:if0"/>
    </link>
  </rspec>
  

• Login to phys1 and virt1 and run ifconfig to get the mac addresses of each dataplane interface
• Look at the experiment details page and make sure that contains both mac addresses
• Run sliverstatus on the slice and see if that output contains either or both mac addresses
• Run listresources on the slice and see if that output contains either or both mac addresses

Trying all that now.

[chaos@bbn.com]

Hmm, i got a createsliver error i didn't expect. Pasting it here in case it rings a bell to anyone who has been paying more attention than i have:

$ omniam u
tah.geniracks.net pgeni createsliver ecgtest ./IG-MON-nodes-B.rspec 
INFO:omni:Loading config file /home/chaos/omni/omni_pgeni
INFO:omni:Using control framework pg
INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest expires on 2012-09-15 18:58:19 UTC
INFO:omni:Creating sliver(s) from rspec file ./IG-MON-nodes-B.rspec for slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest
ERROR:omni.protogeni:Call for Create Sliver urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest at https://utah.geniracks.net/protogeni/xmlrpc/am failed. Server says: <Fault 2: '*** WARNING: mapper:***   phys1-virt1-0: changing phys1:0 to non-encapsulating since physical***   host does not support veth-ne *** ERROR: mapper:***   Can't call method "_vlanname" on an undefined value at***   /usr/testbed/lib/libvtop_test.pm line 5937, <DATA> line 595.
    \nseed = 1347743430\nPhysical Graph: 7\nCalculating shortest paths on switch fabric.\nVirtual Graph: 2\nGenerating physical equivalence classes:7\nType precheck:\nType precheck passed.\nNode mapping precheck:\nNode mapping precheck succeeded\nPolicy precheck:\nPolicy precheck succeeded\nAnnealing.\nDoing melting run\nReverting: forced\nReverting to best solution\nDone\n   BEST SCORE:  3.74 in 17000 iters and 0.210396 seconds\nWith 0 violations\nIters to find best score:  31\nViolations: 0\n  unassigned:  0\n  pnode_load:  0\n  no_connect:  0\n  link_users:  0\n  bandwidth:   0\n  desires:     0\n  vclass:      0\n  delay:       0\n  trivial mix: 0\n  subnodes:    0\n  max_types:   0\n  endpoints:   0\nNodes:\nphys1 pc4\nvirt1 pc3\nEnd Nodes\nEdges:\nlinksimple/phys1-virt1-0/phys1:0,virt1:0 intraswitch link-pc4:eth1-procurve2:(null) (pc4/eth1,(null)) link-pc3:eth1-procurve2:(null) (pc3/eth1,(null))\nEnd Edges\nEnd solution\nSummary:\nprocurve2 0 vnodes, 200000 nontrivial BW, 0 trivial BW, type=(null)\npc3 1 vnodes, 100000 nontrivial BW, 0 trivial BW, type=pcvm\n    100000 link-pc3:eth1-procurve2:(null)\npc4 1 vnodes, 100000 nontrivial BW, 0 trivial BW, type=pc\n    100000 link-pc4:eth1-procurve2:(null)\n    ?+virtpercent: used=0 total=100\n    ?+cpu: used=0 total=2666\n    ?+ram: used=0 total=3574\n    ?+cpupercent: used=0 total=92\n    ?+rampercent: used=0 total=80\nTotal physical nodes used: 2\nEnd summary\n'>
INFO:omni:Asked https://utah.geniracks.net/protogeni/xmlrpc/am to reserve resources. Result:
INFO:omni:<!-- Reserved resources for:
        Slice: ecgtest
        At AM:
        URL: https://utah.geniracks.net/protogeni/xmlrpc/am
 -->
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed createsliver:

  Options as run:
                aggregate: https://utah.geniracks.net/protogeni/xmlrpc/am
                configfile: /home/chaos/omni/omni_pgeni
                framework: pg
                native: True

  Args: createsliver ecgtest ./IG-MON-nodes-B.rspec

  Result Summary: Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest expires on 2012-09-15 18:58:19 UTC
Asked https://utah.geniracks.net/protogeni/xmlrpc/am to reserve resources. No manifest Rspec returned. <Fault 2: '*** WARNING: mapper:***   phys1-virt1-0: changing phys1:0 to non-encapsulating since physical***   host does not support veth-ne *** ERROR: mapper:***   Can't call method "_vlanname" on an undefined value at***   /usr/testbed/lib/libvtop_test.pm line 5937, <DATA> line 595.
    \nseed = 1347743430\nPhysical Graph: 7\nCalculating shortest paths on switch fabric.\nVirtual Graph: 2\nGenerating physical equivalence classes:7\nType precheck:\nType precheck passed.\nNode mapping precheck:\nNode mapping precheck succeeded\nPolicy precheck:\nPolicy precheck succeeded\nAnnealing.\nDoing melting run\nReverting: forced\nReverting to best solution\nDone\n   BEST SCORE:  3.74 in 17000 iters and 0.210396 seconds\nWith 0 violations\nIters to find best score:  31\nViolations: 0\n  unassigned:  0\n  pnode_load:  0\n  no_connect:  0\n  link_users:  0\n  bandwidth:   0\n  desires:     0\n  vclass:      0\n  delay:       0\n  trivial mix: 0\n  subnodes:    0\n  max_types:   0\n  endpoints:   0\nNodes:\nphys1 pc4\nvirt1 pc3\nEnd Nodes\nEdges:\nlinksimple/phys1-virt1-0/phys1:0,virt1:0 intraswitch link-pc4:eth1-procurve2:(null) (pc4/eth1,(null)) link-pc3:eth1-procurve2:(null) (pc3/eth1,(null))\nEnd Edges\nEnd solution\nSummary:\nprocurve2 0 vnodes, 200000 nontrivial BW, 0 trivial BW, type=(null)\npc3 1 vnodes, 100000 nontrivial BW, 0 trivial BW, type=pcvm\n    100000 link-pc3:eth1-procurve2:(null)\npc4 1 vnodes, 100000 nontrivial BW, 0 trivial BW, type=pc\n    100000 link-pc4:eth1-procurve2:(null)\n    ?+virtpercent: used=0 total=100\n    ?+cpu: used=0 total=2666\n    ?+ram: used=0 total=3574\n    ?+cpupercent: used=0 total=92\n    ?+rampercent: used=0 total=80\nTotal physical nodes used: 2\nEnd summary\n'> 
INFO:omni: ============================================================

[chaos@bbn.com]

Okay, sliver creation succeeded, thanks to a bugfix from Leigh!

• Login to phys1 and virt1 and run ifconfig to get the mac addresses of each dataplane interface:

  [chaos@virt1 ~]$ /sbin/ifconfig 
  eth999    Link encap:Ethernet  HWaddr 02:94:E7:91:EC:D1  
            inet addr:172.17.3.4  Bcast:172.31.255.255  Mask:255.240.0.0
            inet6 addr: fe80::94:e7ff:fe91:ecd1/64 Scope:Link
            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
            RX packets:331 errors:0 dropped:0 overruns:0 frame:0
            TX packets:109 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:0 
            RX bytes:28843 (28.1 KiB)  TX bytes:9747 (9.5 KiB)
  
  lo        Link encap:Local Loopback  
            inet addr:127.0.0.1  Mask:255.0.0.0
            inet6 addr: ::1/128 Scope:Host
            UP LOOPBACK RUNNING  MTU:16436  Metric:1
            RX packets:3 errors:0 dropped:0 overruns:0 frame:0
            TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:0 
            RX bytes:294 (294.0 b)  TX bytes:294 (294.0 b)
  
  mv4.18    Link encap:Ethernet  HWaddr 02:9A:DD:C8:00:C9  
            inet addr:10.10.1.2  Bcast:10.10.1.255  Mask:255.255.255.0
            inet6 addr: fe80::9a:ddff:fec8:c9/64 Scope:Link
            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
            RX packets:0 errors:0 dropped:0 overruns:0 frame:0
            TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:0 
            RX bytes:0 (0.0 b)  TX bytes:468 (468.0 b)
  
  [chaos@phys1 ~]$ /sbin/ifconfig 
  eth0      Link encap:Ethernet  HWaddr E8:39:35:B1:EC:9C  
            inet addr:155.98.34.14  Bcast:155.98.34.255  Mask:255.255.255.0
            inet6 addr: fe80::ea39:35ff:feb1:ec9c/64 Scope:Link
            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
            RX packets:415 errors:0 dropped:0 overruns:0 frame:0
            TX packets:264 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1000 
            RX bytes:43909 (42.8 KiB)  TX bytes:27206 (26.5 KiB)
            Interrupt:30 Memory:f4000000-f4012800 
  
  eth1      Link encap:Ethernet  HWaddr E8:39:35:B1:EC:9E  
            inet addr:10.10.1.1  Bcast:10.10.1.255  Mask:255.255.255.0
            inet6 addr: fe80::ea39:35ff:feb1:ec9e/64 Scope:Link
            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
            RX packets:0 errors:0 dropped:0 overruns:0 frame:0
            TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:1000 
            RX bytes:0 (0.0 b)  TX bytes:492 (492.0 b)
            Interrupt:37 Memory:f2000000-f2012800 
  
  lo        Link encap:Local Loopback  
            inet addr:127.0.0.1  Mask:255.0.0.0
            inet6 addr: ::1/128 Scope:Host
            UP LOOPBACK RUNNING  MTU:16436  Metric:1
            RX packets:41 errors:0 dropped:0 overruns:0 frame:0
            TX packets:41 errors:0 dropped:0 overruns:0 carrier:0
            collisions:0 txqueuelen:0 
            RX bytes:3619 (3.5 KiB)  TX bytes:3619 (3.5 KiB)
  

• Look at the experiment details page and make sure that contains both mac addresses: ​https://boss.utah.geniracks.net/showexp.php3?experiment=2243#details says:

  ID              Member          IP              MAC                  NodeID
  --------------- --------------- --------------- -------------------- ---------
  phys1-virt1-0   phys1:0         10.10.1.1       e8:39:35:b1:ec:9e    pc4
                                                  1/1 <-> 1/13         procurve2
  phys1-virt1-0   virt1:0         10.10.1.2       02:9a:dd:c8:00:c9    pcvm3-4
  

  So that's present and correct on both counts.
• Run sliverstatus on the slice and see if that output contains either or both mac addresses

  ...
                                       'children': [{'attributes': {'client_id': 'phys1',
                                                                    'component_id': 'urn:publicid:IDN+utah.geniracks.net+node+pc4',
                                                                    'component_manager_id': 'urn:publicid:IDN+utah.geniracks.net+authority+cm',
                                                                    'exclusive': 'true',
                                                                    'sliver_id': 'urn:publicid:IDN+utah.geniracks.net+sliver+9449'}, 
                                                     'children': [{'attributes': {'name': 'raw-pc'},
                                                                   'children': [],
                                                                   'name': 'sliver_type'},
                                                                  {'attributes': {'client_id': 'phys1:if0',
  
  'component_id': 'urn:publicid:IDN+utah.geniracks.net+interface+pc4:eth1',
  
  'mac_address': 'e83935b1ec9e',
  
  'sliver_id': 'urn:publicid:IDN+utah.geniracks.net+sliver+9452'}, 
                                                                   'children': [{'attributes': {'address': '10.10.1.1',
  
                'type': 'ipv4'},
                                                                                 'children': [],
                                                                                 'name': 'ip'}],
  ...
                                                    {'attributes': {'client_id': 'virt1',
                                                                    'component_id': 'urn:publicid:IDN+utah.geniracks.net+node+pc3',
                                                                    'component_manager_id': 'urn:publicid:IDN+utah.geniracks.net+authority+cm',
                                                                    'exclusive': 'false',
                                                                    'sliver_id': 'urn:publicid:IDN+utah.geniracks.net+sliver+9450'}, 
                                                     'children': [{'attributes': {'name': 'emulab-openvz'},
                                                                   'children': [],
                                                                   'name': 'sliver_type'},
                                                                  {'attributes': {'client_id': 'virt1:if0',
  
  'component_id': 'urn:publicid:IDN+utah.geniracks.net+interface+pc3:eth1',
  
  'mac_address': '029addc800c9',
  
  'sliver_id': 'urn:publicid:IDN+utah.geniracks.net+sliver+9453'}, 
  ...
  

• Run listresources on the slice and see if that output contains either or both mac addresses:

  ...
          <interface client_id="phys1:if0" component_id="urn:publicid:IDN+utah.geniracks.net+interface+pc4:eth1" mac_address="e83935b1ec9e" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+9452">      <ip address="10.10.1.1" type="ipv4"/>    </interface>    
  ...
          <interface client_id="virt1:if0" component_id="urn:publicid:IDN+utah.geniracks.net+interface+pc3:eth1" mac_address="029addc800c9" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+9453">      <ip address="10.10.1.2" type="ipv4"/
  ...
  

So, it looks to me like all four of those things now match. Great! Closing this out.