Opened 6 years ago

Last modified 6 years ago

#1197 new

MAX and ION aggregates report Internal API error for out-of-sync credentials sometimes

Reported by: lnevers@bbn.com Owned by: xyang@maxgigapop.net
Priority: major Milestone:
Component: STITCHING Version: SPIRAL6
Keywords: Network Stitching Cc: tmack@CS.Princeton.EDU
Dependencies:

Description

Both ION and MAX aggregates have reported the following type of exception:

14:24:26 INFO     stitch.Aggregate: Writing to '/tmp/ig-nyse-ig-miss-createsliver-request-11-ion-internet2-edu.xml'
14:24:26 INFO     stitch.Aggregate:    Stitcher doing createsliver at http://geni-am.net.internet2.edu:12346
14:24:26 ERROR    omni:  {'output': "Internal API error: 'NoneType' object is not iterable", 'geni_api': 2, 'code': {'am_type': 'sfa', 'geni_code': 5, 'am_code': 5}, 'value': ''}
14:24:26 INFO     stitch.Aggregate: Got AMAPIError doing createsliver ig-nyse-ig-miss at <Aggregate urn:publicid:IDN+ion.internet2.edu+authority+am>: AMAPIError: Error from Aggregate: code 5. sfa AM code: 5: Internal API error: 'NoneType' object is not iterable.

There is no manifest files for the above failure. The problem does not occur on the next try.

On 2/10/14 4:53 PM, Xi Yang wrote:

> I looked back into the sfa.log at MAX AM. The trace stack is below:
> 2014-02-06 13:25:17,573 - ERROR - XmlrpcApi.handle has caught Exception BEG TRACEBACK
> Traceback (most recent call last):
>   File "/usr/lib/python2.5/site-packages/sfa/server/xmlrpcapi.py", line 153, in handle
>     result = self.call(source, method, *args)
>   File "/usr/lib/python2.5/site-packages/sfa/server/xmlrpcapi.py", line 125, in call
>     return function(*args)
>   File "/usr/lib/python2.5/site-packages/sfa/util/method.py", line 79, in __call__
>     result = self.call(*args, **kwds)
>   File "/usr/lib/python2.5/site-packages/sfa/methods/CreateSliver.py", line 58, in call
>     result = self.api.manager.CreateSliver(self.api, slice_xrn, creds, rspec, users, options)
>   File "/usr/lib/python2.5/site-packages/sfa/managers/aggregate_manager_max.py", line 380, in CreateSliver
>     ret = self.create_slice(api, slice_xrn, creds, rspec_string, users)
>   File "/usr/lib/python2.5/site-packages/sfa/managers/aggregate_manager_max.py", line 247, in create_slice
>     self.prepare_slice(api, xrn, cred, users)
>   File "/usr/lib/python2.5/site-packages/sfa/managers/aggregate_manager_max.py", line 175, in prepare_slice
>     site = slices.verify_site(hrn, slice_record, peer, sfa_peer)
>   File "/usr/lib/python2.5/site-packages/sfa/plc/plslices.py", line 288, in verify_site
>     sites = self.driver.shell.GetSites(login_base)
>   File "/usr/lib/python2.5/site-packages/sfa/plc/plshell.py", line 83, in func
>     result=getattr(self.proxy, actual_name)(self.plauth, *args, **kwds)
>   File "/usr/share/plc_api/PLC/Shell.py", line 52, in __call__
>     return self.func(*args, **kwds)
>   File "/usr/share/plc_api/PLC/Method.py", line 97, in __call__
>     result = self.call(*args, **kwds)
>   File "/usr/share/plc_api/PLC/Methods/GetSites.py", line 33, in call
>     return Sites(self.api, site_filter, return_fields)
>   File "/usr/share/plc_api/PLC/Sites.py", line 274, in __init__
>     self.selectall(sql)
>   File "/usr/share/plc_api/PLC/Table.py", line 426, in selectall
>     for row in self.api.db.selectall(sql, params):
>   File "/usr/share/plc_api/PLC/PostgreSQL.py", line 208, in selectall
>     labels = [column[0] for column in self.description]
> TypeError: 'NoneType' object is not iterable
>
> This problem is similar to what I have seen with first-timer project. It is an SFA-MyPLC problem that I have no idea how it happens.
> Looks like SFA has some issue when out-of-sync site credential between the Portal clearing house and MyPLC in some rare cases.
> I have no clue on either the cause or the pattern.

Change History (1)

comment:1 Changed 6 years ago by lnevers@bbn.com

Cc: tmack@CS.Princeton.EDU added

Adding Tony Mack (tmack@CS.Princeton.EDU) to the CC: as requested by Xi.

Note: See TracTickets for help on using tickets.