Creating sliver with 2 VMs on shared vlan "mesoscale-openflow" fails at GATech

[lnevers@bbn.com]

Requesting 2 VMs on the shared VLAN "mesoscale-openflow" results in both nodes in geni_status "failed". Attaching the request RSpec, and below is the geni_error reported:

"geni_error": "Running os dependent initialization script rc.linux\n*** WARNING: could not identify root disk, no swap enabled
Informing Emulab Control that we have rebooted
Checking Testbed reservation status
Cleaning node; removing configuration files
Allocated! pgeni-gpolab-bbn-com/EG-CT-4/gatech-ig2
Setting up for SFS ... 
Setting up Ping of Death
Authenticated IPOD enabled from 143.215.216.132/255.255.255.255
Starting node health monitoring daemon
FATAL: Module i2c_piix4 not found.
FATAL: Module w83781d not found.
Failed running rc.healthd (256)! at /usr/local/etc/emulab/libsetup.pm line 1152, <TD> line 2.
Starting slothd usage detector
Starting testbed update watchdog
Running config scripts
Checking manifest...
Checking Testbed localization configuration ... 
Checking Testbed keys ...
Checking Testbed Blobs configuration ... 
Checking Testbed user accounts configuration ... 
Adding group: pgeni-gpolab-bbn/503
Adding: lnevers/20001/503/1/lnevers//users/lnevers/
updating local home directory
Adding: geniuser/502/503/1/Geni User//users/geniuser/
updating local home directory
Checking Testbed route configuration ... 
/usr/local/etc/emulab/dijkstra: Invalid source argument: source=gatech-ig2
*** WARNING: dijkstra exited with status 256!
*** WARNING: Could not get routes from ddijkstra!
*** /usr/local/etc/emulab/rc/rc.route:
    Could not get router configuration from libsetup!
 Failed running rc.route (256)! at /usr/local/etc/emulab/libsetup.pm line 1152.
Checking Testbed tunnel configuration ... 
Checking Testbed interface configuration ... 
Use of uninitialized value $speed in pattern match (m//) at /usr/local/etc/emulab/liblocsetup.pm line 645.
*** Bad speed units  in ifconfig, default to 100Mbps
Use of uninitialized value $duplex in string eq at /usr/local/etc/emulab/liblocsetup.pm line 678.
Use of uninitialized value $duplex in string eq at /usr/local/etc/emulab/liblocsetup.pm line 681.
Use of uninitialized value $duplex in concatenation (.) or string at /usr/local/etc/emulab/liblocsetup.pm line 685.
*** Bad duplex  in ifconfig, default to full
Cannot set new settings: Operation not supported
  not setting speed
  not setting duplex
  not setting autoneg
/var/emulab/boot/rc.ifc: line 13: /var/emulab/boot/rc.route: No such file or directory
*** /usr/local/etc/emulab/rc/rc.ifconfig:
    Error running /var/emulab/boot/rc.ifc
  Failed running rc.ifconfig (6400)! at /usr/local/etc/emulab/libsetup.pm line 1152.
Using Linux netem instead of custom qdiscs.
Checking Testbed hostnames configuration ... 
Checking Testbed trace configuration ... 
Checking Testbed trafgen configuration ... 
Checking Testbed Tarball configuration ... 
Checking Testbed RPM configuration ... 
Starting linktest daemon
*** ERROR: linktest daemon failed to start. Status: 15
*** /usr/local/etc/emulab/rc/rc.bootsetup:
   Error running /usr/local/etc/emulab/rc/rc.linktest ", 
      "geni_status": "failed"

[lnevers@bbn.com]

Capturing exchange outside of this ticket, which provided two workaround while waiting for fix to route calculator

On 5/6/13 3:30 PM, Leigh Stoller wrote:

Okay, I think the route calculator is confused by it. The calculator should ignore these, but it will take me a bit to figure that out.

In the meantime, try combining into a single link instead of two different links. I think that is fixed now. If not, another thing to try is changing the netmask to 255.255.255.255 since the way it is specified now, you have two independent links with the same subnet, which ultimately is (I believe) the cause of the route calculator error.

Based on the above, modified the link in the RSpec from 2 entries:

  <link client_id="openflow-mesoscale-0">
    <interface_ref client_id="gatech-ig:if0" />
    <sharedvlan:link_shared_vlan name="mesoscale-openflow" />
  </link>
  <link client_id="openflow-mesoscale-1">
    <interface_ref client_id="gatech-ig2:if0" />
    <sharedvlan:link_shared_vlan name="mesoscale-openflow" />
  </link>

Into one link definition:

  <link client_id="openflow-mesoscale-0">
    <interface_ref client_id="gatech-ig:if0" />
    <interface_ref client_id="gatech-ig2:if0" />
    <sharedvlan:link_shared_vlan name="mesoscale-openflow" />
  </link>

The above showed another issue:

On 5/6/13 4:29 PM, Leigh Stoller wrote:

On May 6, 2013, at 1:23 PM, Luisa Nevers wrote:

Ok, re-ran with the combined link and this failure was reported:

And yet another error!

event_register_withkeydata_withretry: could not get hostname: File name too long could not register with event system

[root@gatech-ig logs]# hostname gatech-ig.IG-CT-4.pgeni-gpolab-bbn-com.instageni.rnoc.gatech.edu

Sadly, it is exactly 64 chars, which means there is a bug in the event library we never noticed.

We have talked about this before, and we still have the fixit token, but in the meantime, try shorter client_id names.

Tried shorter hostname (gt, gt2) with the combined shared mesoscale link and can now run. Will continue testing with workaround.

[lnevers@bbn.com]

Verified that this is no longer an issue either work at GA Tech:

  <link client_id="openflow-mesoscale">
    <interface_ref client_id="gt:if0" />
    <interface_ref client_id="gt2:if0" />
    <sharedvlan:link_shared_vlan name="mesoscale-openflow" />
  </link>

or

  <link client_id="openflow-mesoscale-0">
    <interface_ref client_id="gt:if0" />
    <sharedvlan:link_shared_vlan name="mesoscale-openflow" />
  </link>
  <link client_id="openflow-mesoscale-1">
    <interface_ref client_id="gt:if0" />
    <sharedvlan:link_shared_vlan name="mesoscale-openflow" />
  </link>

Closing ticket.