No VLANs available on link (PCE_CREATE_FAILED)

[lnevers@bbn.com]

While creating a sliver (lnstitchc) between PG Utah and PG KY, there was an error reporting that no VLAN are available. This problem has been seem in the past and explained as a SCS issue with randomizing VLAN tags. Reporting in case the issue was not fully resolved with the last fix. Here is the failure reported:

07:26:58 INFO     stitch.Aggregate: Writing to '/tmp/lnstitchb-createsliver-request-11-ion-internet2-edu.xml'
07:26:58 INFO     stitch.Aggregate: 
	Stitcher doing createsliver at http://geni-am.net.internet2.edu:12346
07:27:45 INFO     stitch.Aggregate: DCN AM <Aggregate urn:publicid:IDN+ion.internet2.edu+authority+cm>: must wait for status ready....
07:27:45 INFO     stitch.Aggregate: Pause to let circuit become ready...
07:28:27 INFO     stitch.Aggregate: Pause to let circuit become ready...
07:29:05 WARNING  stitch.Aggregate: sliverstatus 21421 is (still) failed at 
<Aggregate urn:publicid:IDN+ion.internet2.edu+authority+cm>. Delete and retry.
07:29:05 WARNING  stitch.Aggregate:   Status had error message: VLAN PCE(PCE_CREATE_FAILED): 
'There are no VLANs available on link ion.internet2.edu:rtr.atla:ge-10/3/2:protogeni  
on reservation ion.internet2.edu-21421 in VLAN PCE'
07:29:11 INFO     stitch.launcher: Will put <Aggregate urn:publicid:IDN+ion.internet2.edu+authority+cm> 
back in the pool to allocate. 
Got Sliver status for circuit 21421 was (still): failed: VLAN PCE(PCE_CREATE_FAILED): 
'There are no VLANs available on link ion.internet2.edu:rtr.atla:ge-10/3/2:protogeni 
 on reservation ion.internet2.edu-21421 in VLAN PCE'

}}}

[ckotil@grnoc.iu.edu]

In the OSCARS UI I was able to recreate (clone) this circuit without any problem. The cloned circuit went to active. We have seen this error , PCE Error, sporadically appear. I actually have a new release of the oscars api and and coordinator packages in hand that are supposed to resolve this issue. The upgrade is scheduled for Friday June 21 at 1200 UTC.

Since the error appears to be random, and the vlan IS actually available, if you try the test again it should work. According to the oscars developers after the upgrade, this issue should be resolved for good.

[ckotil@grnoc.iu.edu]

Luisa, Oscars has been updated, as you're going through your testing please let me know if this error resurfaces.

Thanks, --Chad

[lnevers@bbn.com]

Will re-run the test that showed this problem now.

[lnevers@bbn.com]

Just saw the error again:

10:48:03 WARNING  stitch.Aggregate: sliverstatus 21681 is (still) failed at <Aggregate urn:publicid:IDN+ion.internet2.edu+authority+cm>. Delete and retry.
10:48:03 WARNING  stitch.Aggregate:   Status had error message: VLAN PCE(PCE_CREATE_FAILED): 'There are no VLANs available on link ion.internet2.edu:rtr.atla:ge-10/3/2:protogeni  on reservation ion.internet2.edu-21681 in VLAN PCE'
10:48:09 INFO     stitch.launcher: Will put <Aggregate urn:publicid:IDN+ion.internet2.edu+authority+cm> back in the pool to allocate. Got Sliver status for circuit 21681 was (still): failed: VLAN PCE(PCE_CREATE_FAILED): 'There are no VLANs available on link ion.internet2.edu:rtr.atla:ge-10/3/2:protogeni  on reservation ion.internet2.edu-21681 in VLAN PCE'

[ckotil@grnoc.iu.edu]

What happened here is that the circuit ion.internet2.edu-21671 was being configured, when your client attempted to create ion.internet2.edu-21681 exactly 1 minute later. This caused Oscars to throw the PCE error because the vlan was already in use by ion.internet2.edu-21671. I'm not sure how you want to proceed, but this doesn't appear to be an issue with Oscars, rather an issue with the client attempting to submit too quickly to oscars.

[lnevers@bbn.com]

Looking at the logs at the time, there were two slivers being set up concurrently:

• Sliver "lnstitcha" requesting a sliver from GPO IG to Utah PG (circuit 21681)
• Sliver "lnstitchb" requesting a sliver from GPO IG to Utah PG (circuit 21671)

The VLAN PCE failure for "lnsticha" attempting to get circuit 21681 is reported as failing because circuit 21671 was configured for "lnstitchb".

I am attaching the log for the two slivers creations.

[lnevers@bbn.com]

Incorrectly categorized as task, modifying to defect category

[xyang@maxgigapop.net]

This is expected behavior not a bug. VLAN contention is inevitable before we have stitching negotiation. Randomization only reduces the chance of contention but cannot eliminate it.