'Path Setup failed with error from PSSSetup' reported for I2/ION aggregate

[lnevers@bbn.com]

Creating a sliver from IG GPO to IG Utah. Createsliver reports a PATH_SETUP_FAILED error for Internet2/ION aggregate. The request RSpec is attached, the output from stitcher:

08:54:37 INFO     stitch.Aggregate: Writing to '/tmp/demo-createsliver-request-11-ion-internet2-edu.xml'
08:54:37 INFO     stitch.Aggregate: 
	Stitcher doing createsliver at http://geni-am.net.internet2.edu:12346
08:55:22 INFO     stitch.Aggregate: DCN AM <Aggregate urn:publicid:IDN+ion.internet2.edu+authority+cm>: must wait for status ready....
08:55:22 INFO     stitch.Aggregate: Pause to let circuit become ready...

08:56:29 WARNING  stitch.Aggregate: sliverstatus 22201 is (still) failed at <Aggregate urn:publicid:IDN+ion.internet2.edu+authority+cm>. Delete and retry.
08:56:29 WARNING  stitch.Aggregate:   Status had error message: Coordinator(PATH_SETUP_FAILED): 'Path Setup failed with error from PSSSetup'
08:56:35 INFO     stitch.launcher: Will put <Aggregate urn:publicid:IDN+ion.internet2.edu+authority+cm> back in the pool to allocate. Got Sliver status for circuit 22201 was (still): failed: Coordinator(PATH_SETUP_FAILED): 'Path Setup failed with error from PSSSetup'
08:56:35 INFO     stitch.launcher: Pausing for 120 seconds for Aggregates to free up resources...

08:58:35 INFO     stitch.Aggregate: Writing to '/tmp/demo-createsliver-request-12-ion-internet2-edu.xml'
08:58:35 INFO     stitch.Aggregate: 
	Stitcher doing createsliver at http://geni-am.net.internet2.edu:12346
08:59:20 INFO     stitch.Aggregate: DCN AM <Aggregate urn:publicid:IDN+ion.internet2.edu+authority+cm>: must wait for status ready....
08:59:20 INFO     stitch.Aggregate: Pause to let circuit become ready...
09:00:27 WARNING  stitch.Aggregate: sliverstatus 22211 is (still) failed at <Aggregate urn:publicid:IDN+ion.internet2.edu+authority+cm>. Delete and retry.
09:00:27 WARNING  stitch.Aggregate:   Status had error message: Coordinator(PATH_SETUP_FAILED): 'Path Setup failed with error from PSSSetup'
09:00:33 INFO     stitch.launcher: Will put <Aggregate urn:publicid:IDN+ion.internet2.edu+authority+cm> back in the pool to allocate. Got Sliver status for circuit 22211 was (still): failed: Coordinator(PATH_SETUP_FAILED): 'Path Setup failed with error from PSSSetup'

[xyang@maxgigapop.net]

Reassign to Chad to check with NOC.

[lnevers@bbn.com]

Will re-test when the ION update is applied. Email from Chad explaining about the update:

On 7/3/13 11:16 AM, Chad Kotil wrote:

Luisa, We have identified the problem. There was an interface change at atla and wash which has not been applied to ION yet. Essentially anything taking the path rtr.atla ae8.10 or rtr.wash ae8.10 will fail until the fix is applied. There is a systems ticket tracking the issue. I have asked that an update be sent to you and Xi when it has been fixed. Should be some time later today. Im out of the office this week so I will be slower than usual to respond. If you need immediate help, please contact the NOC. Thanks, --Chad

[lnevers@bbn.com]

Tried after getting an update about the fix being installed. No longer see the path error, but there are no VLANs available at rtr.newy:ae0:bbn, so will try again later. Following is the email form the GlobalNOC:

On 7/3/13 11:33 AM, Jason Al-Mansor wrote:

> This should be fixed now. I created a test circuit between our test
> points in ATLA and WASH successfully. Please verify that this is working
> correctly for you now.
>
>
> Jason Al-Mansor
> Systems Engineer
> GlobalNOC at Indiana University
> jalmanso@grnoc.iu.edu
>
> On 7/3/13 11:15 AM, Footprints Systems Support Project wrote:
>> When replying, type your text above this line.
>> ----------------------------------------------
>>
>> System Support ticket 19118 was updated on 07/03/2013 at 15:15:12 UTC.
>>
>> TITLE:        Internet2 ION Topology File Correction Request
>> SUBMITTER:    noc@net.internet2.edu
>> CREATED:      07/02/2013 20:09:14
>>
>> CI:           4-Normal
>> TICKET TYPE:  Problem/Request
>> STATUS:       Engineering
>> ASSIGNEES:    System Engineering, Jason Al-Mansor 
>> LAST CHANGE:  Appended a Description, Incoming mail: From: ckotil@grnoc.iu.edu; To: systems-support@tick.globalnoc.iu.edu.
>>
>> SUMMARY OF CURRENT STATUS
>>
>> I2 NetEng has requested for the Topology File for
>> rtr.wash and rtr.atla to be updated to use the proper
>> ae8.30.
>>
>> DESCRIPTION
>>
>> Geni folks are reporting that this is causing problems for their testing.
>> I tracked the issue down to this 'corruption'. Change ae8.10 to ae8.30 for rtr.atla , and probably wash too, and fix up the IPs.
>> When this is done please send a note to:
>>
>> lnevers@bbn.com
>> xyang@maxgigapop.net
>>
>> Thanks,
>> --Chad
>>
>> On Jul 2, 2013, at 4:15 PM, Footprints Systems Support Project <systems-support@tick.globalnoc.iu.edu> wrote:
>>
>>> [Duplicate message snipped]
>>
>>
>> Ticket URL:  https://tick.globalnoc.iu.edu/MRcgi/MRlogin.pl?DL=19118DA101
>>
>

[lnevers@bbn.com]

Still unable to verify the fix to the path. Had several attempts and all fail with:

VLAN PCE(PCE_CREATE_FAILED): 'There are no VLANs available on link 
ion.internet2.edu:rtr.newy:ae0:bbn  on reservation ion.internet2.edu-22331 
in VLAN PCE'

[xyang@maxgigapop.net]

Two slices had taken 3748 (ig-gpo-ig-utah ) and 3749 (pg-utah-ig-gpo). Only 3747 was left. It may take SCS several times to hit that left one.

[hdempsey@bbn.com]

ION AM has been decomissioned.