Sliver failures while requesting mulitple VLANs for PG Utah

[lnevers@bbn.com]

Writing ticket to track to resolution:

Not able to exchange traffic over the second link when a stitched two link scenario is run between a GPO Xen host and a Utah Xen host. The same scenario works if the 2 Xen hosts are in the same rack.

Email exchanged about this issue:

On 8/16/13 9:35 AM, Luisa Nevers wrote:
> Hi Jonathan,
>
> I tried this 2 link topology again this morning and it finally works.  I have
> successfully created a stitched sliver between the GPO and Utah racks,
> but only one link works.  There is no network connectivity over one of the
> links.  The slice is named xen-2link and the hosts allocated are:
>
>    - pc5.instageni.gpolab.bbn.com port 30266
>   - pc3.utah.geniracks.net port port 30010
>
> Luisa
>
>
> On 8/15/13 6:11 PM, Jonathon Duerig wrote:
>> Luisa, I didn't get a chance to look at this today but will look tomorrow morning when I get in.
>>
>> ---
>> Broad audience or deep message: Pick one.
>>
>>
>> On Thu, 15 Aug 2013, Leigh Stoller wrote:
>>>> Hi Leigh, 
>>>
>>> Hi Luisa. I'll defer to Jon on this ...
>>>
>>>> I have been trying to set up stitched sliver from IG GPO to IG Utah with 2 links
>>>> for the past two days, but each attempt always fails with Emulab reporting not 
>>>> having a VLAN for one of the links. I am wondering if there is really only one 
>>>>VLAN available at Emulab? I just ran the test and here are some of the failure:
>>>> https://www.emulab.net/spewlogfile.php3?logfile=ac1fc8f53ba852a957f8d7a54fd8691f
>>>> https://www.emulab.net/spewlogfile.php3?logfile=49e1c5c2040262428f40bd01dc29ab10
>>>> https://www.emulab.net/spewlogfile.php3?logfile=ac1fc8f53ba852a957f8d7a54fd8691f
>>>>
>>>> Luisa
>>>>
>>>
>>>

[lnevers@bbn.com]

The problem captured in this ticket was probably due to a sliver expiring. But, the original email exchange that resulted in this ticket was reporting concerns about VLAN availability at PG Utah when requesting multiple VLAN (about 1 out of 5 requests was successful). VLAN issue found and captured for tracking below. Also modifying ticket summary to capture current issue:

On 8/21/13 8:29 AM, Luisa Nevers wrote:

Since the sliver had expired at all aggregates, I tried to set up the stitched sliver again.
The new slice named "xen-2l" was created after several failed attempts.  Once
created there is  connectivity over each of the two links! So Rob's theory may
be right, and there is no problem once the topology is set up.

But, I have run into the original problem of VLAN availability. If I request 2 links, most
requests  just cannot get 2 VLANs at the emulab  aggregate, which seems strange, there
must be 2 VLANs at emulab?!?.

I have captured the failures for the unsuccessful  tries while trying to to set up the sliver "xen-2l":

https://www.emulab.net/spewlogfile.php3?logfile=025a02b5d0393dc8750c3d4163c9ca07 (vlan tag for link1 not available)
https://www.emulab.net/spewlogfile.php3?logfile=b59187a696e9d719fb7fc85fb2d6546c (Error reserving vlan tag for link0)
https://www.emulab.net/spewlogfile.php3?logfile=4ee69032ab9f04df77167d6bce0de33d (Error reserving vlan tag for link1) 

On 8/21/13 9:50 AM, Leigh Stoller wrote:

> But, I have run into the original problem of VLAN availability. If I
> request 2 links, most requests just cannot get 2 VLANs at the emulab
> aggregate, which seems strange, there must be 2 VLANs at emulab?!?.
>
> I have captured the failures for the unsuccessful tries while trying to
> to set up the sliver "xen-2l":

Ug, the problem is that we are leaking vlans like crazy on the Emulab
switch infrastructure.

Rob, if you do a wap snmpit -l -l -O -w, you will see hundreds of vlans on
the switches that should have been deleted.

I will poke around to see if I can figure out what happened (is happening).