Stitching from MAX to PG Utah fails with Internal API error at the I2 ION aggregate

[lnevers@bbn.com]

While setting up a sliver from MAX to PG Utah, an internal error occurred:

$ stitcher.py createsliver max-pg-utah stitch-max-ig-utah.rspec --ionRetryIntervalSecs 180
10:00:26 INFO     stitcher: Loading config file /home/lnevers/.gcf/omni_config
10:00:26 INFO     stitcher: Using control framework portal
10:00:26 INFO     stitcher: Checking that slice max-pg-utah is valid...
10:00:28 INFO     stitcher: Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+max-pg-utah expires within 1 day on 2013-06-18 22:00:19 UTC
10:00:30 INFO     stitcher: <Aggregate urn:publicid:IDN+emulab.net+authority+cm> speaks AM API v3, but sticking with v2
10:00:30 INFO     stitcher: <Aggregate urn:publicid:IDN+utah.geniracks.net+authority+cm> speaks AM API v3, but sticking with v2
10:00:30 ERROR    stitcher: RSpec requires AM urn:publicid:IDN+ion.internet2.edu+authority+am which is not in workflow and URL is unknown!
10:00:30 INFO     stitch.Aggregate: Writing to '/tmp/max-pg-utah-createsliver-request-11-dragon-maxgigapop-net.xml'
10:00:30 INFO     stitch.Aggregate: 
	Stitcher doing createsliver at http://max-myplc.dragon.maxgigapop.net:12346
10:01:16 INFO     stitch.Aggregate: DCN AM <Aggregate urn:publicid:IDN+dragon.maxgigapop.net+authority+am>: must wait for status ready....
10:01:16 INFO     stitch.Aggregate: Pause to let circuit become ready...
10:01:55 INFO     stitch.Aggregate: Pause to let circuit become ready...
10:02:41 INFO     stitch.Aggregate: Allocation at <Aggregate urn:publicid:IDN+dragon.maxgigapop.net+authority+am> complete.
10:02:41 INFO     stitch.Aggregate: Writing to '/tmp/max-pg-utah-createsliver-request-11-emulab-net.xml'
10:02:41 INFO     stitch.Aggregate: 
	Stitcher doing createsliver at https://www.emulab.net:12369/protogeni/xmlrpc/am
10:03:19 INFO     stitch.Aggregate: Allocation at <Aggregate urn:publicid:IDN+emulab.net+authority+cm> complete.
10:03:19 INFO     stitch.Aggregate: Writing to '/tmp/max-pg-utah-createsliver-request-11-ion-internet2-edu.xml'
10:03:19 INFO     stitch.Aggregate: 
	Stitcher doing createsliver at http://geni-am.net.internet2.edu:12346
10:04:04 INFO     stitch.Aggregate: DCN AM <Aggregate urn:publicid:IDN+ion.internet2.edu+authority+cm>: must wait for status ready....
10:04:04 INFO     stitch.Aggregate: Pause to let circuit become ready...
10:04:42 ERROR    omni:  {'output': "Internal API error: 'NoneType' object has no attribute 'group'", 'geni_api': 2, 'code': {'am_type': 'sfa', 'geni_code': 5, 'am_code': 5}, 'value': ''}
10:04:42 WARNING  stitcher: Stitching failed with an error: sliverstatus max-pg-utah failed at <Aggregate urn:publicid:IDN+ion.internet2.edu+authority+cm>: AMAPIError: Error from Aggregate: code 5. sfa AM code: 5: Internal API error: 'NoneType' object has no attribute 'group'.
10:04:42 WARNING  stitcher: Had reservation at http://max-myplc.dragon.maxgigapop.net:12346
10:04:48 WARNING  stitcher: Deleted reservation at http://max-myplc.dragon.maxgigapop.net:12346
10:04:48 WARNING  stitcher: Had reservation at https://www.emulab.net:12369/protogeni/xmlrpc/am
10:04:48 INFO     stitch.Aggregate: Doing deletesliver at https://www.emulab.net:12369/protogeni/xmlrpc/am
10:05:25 WARNING  stitcher: Deleted reservation at https://www.emulab.net:12369/protogeni/xmlrpc/am
sliverstatus max-pg-utah failed at <Aggregate urn:publicid:IDN+ion.internet2.edu+authority+cm>: AMAPIError: Error from Aggregate:
 code 5. sfa AM code: 5: Internal API error: 'NoneType' object has no attribute 'group'.

This topology has been set up successfully several times in the past.

[xyang@maxgigapop.net]

Made a patch to guard against the exception. Chad, please apply it to SFA. Also make sure you have the latest Java code.

Lusia, please post the rspec if you still see problem.

[ckotil@grnoc.iu.edu]

I've updated the ION AM code, and patched sfa. Everything has been restarted loading the new code.

[lnevers@bbn.com]

Replying to xyang@maxgigapop.net:

Lusia, please post the rspec if you still see problem.

Will do!

[lnevers@bbn.com]

Re-ran test and the problem seems resolved. Closing ticket.