"global name 'extract_exception' is not defined" error reported for create sliver

[lnevers@bbn.com]

Creating a sliver between PG Utah and PG UKY failed with the following error:

$ stitcher.py createsliver lnstitchc ./stitch-pg-utah-pg-uky.rspec 
07:25:06 INFO     stitcher: Loading config file /home/lnevers/.gcf/omni_config
07:25:06 INFO     stitcher: Using control framework portal
07:25:07 INFO     stitcher: Checking that slice lnstitchc is valid...
07:25:09 INFO     stitcher: Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+lnstitchc expires within 1 day on 2013-06-20 19:24:41 UTC
07:25:09 INFO     stitcher: <Aggregate urn:publicid:IDN+uky.emulab.net+authority+cm> speaks AM API v3, but sticking with v2
07:25:09 INFO     stitcher: <Aggregate urn:publicid:IDN+emulab.net+authority+cm> speaks AM API v3, but sticking with v2
07:25:09 INFO     stitch.Aggregate: Writing to '/tmp/lnstitchc-createsliver-request-11-uky-emulab-net.xml'
07:25:09 INFO     stitch.Aggregate: 
	Stitcher doing createsliver at https://www.uky.emulab.net:12369/protogeni/xmlrpc/am
07:25:33 INFO     stitch.Aggregate: Allocation at <Aggregate urn:publicid:IDN+uky.emulab.net+authority+cm> complete.
07:25:33 INFO     stitch.Aggregate: Writing to '/tmp/lnstitchc-createsliver-request-11-emulab-net.xml'
07:25:33 INFO     stitch.Aggregate: 
	Stitcher doing createsliver at https://www.emulab.net:12369/protogeni/xmlrpc/am
07:26:58 INFO     stitch.Aggregate: Allocation at <Aggregate urn:publicid:IDN+emulab.net+authority+cm> complete.
07:26:58 INFO     stitch.Aggregate: Writing to '/tmp/lnstitchc-createsliver-request-11-ion-internet2-edu.xml'
07:26:58 INFO     stitch.Aggregate: 
	Stitcher doing createsliver at http://geni-am.net.internet2.edu:12346
07:27:44 INFO     stitch.Aggregate: DCN AM <Aggregate urn:publicid:IDN+ion.internet2.edu+authority+cm>: must wait for status ready....
07:27:44 INFO     stitch.Aggregate: Pause to let circuit become ready...
07:28:26 INFO     stitch.Aggregate: Pause to let circuit become ready...
07:29:04 INFO     stitch.Aggregate: DCN circuit 21401 is ready
07:29:12 WARNING  omni: Didn't get a valid RSpec!
07:29:12 ERROR    omni: No resources from AM http://geni-am.net.internet2.edu:12346:  
{'output': "Internal API error: global name 'extract_exception' is not defined", 
'geni_api': 2, 'code': {'am_type': 'sfa', 'geni_code': 5, 'am_code': 5}, 'value': ''}
07:29:12 WARNING  stitcher: Stitching failed with an error: Stitching failed in 
handleDcn trying listresources at <Aggregate urn:publicid:IDN+ion.internet2.edu+authority+cm>: 
AMAPIError: No resources from AM http://geni-am.net.internet2.edu:12346: 
Error from Aggregate: code 5. sfa AM code: 5: Internal API error: global name 'extract_exception' is not defined.
07:29:12 WARNING  stitcher: Had reservation at https://www.uky.emulab.net:12369/protogeni/xmlrpc/am
07:30:10 WARNING  stitcher: Deleted reservation at https://www.uky.emulab.net:12369/protogeni/xmlrpc/am
07:30:10 WARNING  stitcher: Had reservation at https://www.emulab.net:12369/protogeni/xmlrpc/am
07:30:10 INFO     stitch.Aggregate: Doing deletesliver at https://www.emulab.net:12369/protogeni/xmlrpc/am
07:31:45 WARNING  stitcher: Deleted reservation at https://www.emulab.net:12369/protogeni/xmlrpc/am
Stitching failed in handleDcn trying listresources at <Aggregate urn:publicid:IDN+ion.internet2.edu+authority+cm>: 
AMAPIError: No resources from AM http://geni-am.net.internet2.edu:12346: 
Error from Aggregate: code 5. sfa AM code: 5: Internal API error: global name 'extract_exception' is not defined.

[xyang@maxgigapop.net]

Which aggregate is this? Can you also post the rspec?

[lnevers@bbn.com]

This problem was seen while stitching a sliver from Utah PG to UKY PG. Subsequent attempts have worked. Attaching RSpec.

[xyang@maxgigapop.net]

I was not able to reproduce the problem on my testing ION AM. This should not happen unless the python code got corrupt at one point.

Chad, can search the sfa.log for this error message (or simply "extract_exception") and post the exception stack trace?

[xyang@maxgigapop.net]

There was a patch for this problem.

Chad, can you check if you have applied the attached patch: sfa-2.0-9-patch-13.diff?

[hdempsey@bbn.com]

ION AM decomissioned