Opened 6 years ago

Closed 6 years ago

#1040 closed (fixed)

Shutdown of stitched slice fails with Internal Error at PG and IG aggregates

Reported by: lnevers@bbn.com Owned by: somebody
Priority: major Milestone:
Component: STITCHING Version: SPIRAL5
Keywords: Network Stitching Cc:
Dependencies:

Description

Stitched slice shutdown fails:

$ stitcher.py shutdown ln-ext1 --debug
INFO:stitcher:Loading config file /home/lnevers/.gcf/omni_config
INFO:stitcher:Using control framework portal
INFO:stitcher:Python: CPython 2.6.5
OS: Linux-2.6.32-47-generic-pae-i686-with-Ubuntu-10.04-lucid
DEBUG:stitcher:Using framework type pgch
DEBUG:omni.protogeni:Configured with key file /home/lnevers/.ssl/geni_cert_portal-1.pem
DEBUG:omni.protogeni:Using clearinghouse https://ch.geni.net:8443/
DEBUG:omni.protogeni:Using slice authority https://ch.geni.net:8443/
DEBUG:stitcher:Passing call to Omni
INFO:stitcher:Reading stitching slice ln-ext1 aggregates from file ch\.geni\.net.ln-prj.ln-ext1-amlist.txt
INFO:stitcher:Adding aggregate option https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am (urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm)
INFO:stitcher:Adding aggregate option http://geni-am.net.internet2.edu:12346 (urn:publicid:IDN+ion.internet2.edu+authority+cm)
INFO:stitcher:Adding aggregate option https://www.emulab.net:12369/protogeni/xmlrpc/am (urn:publicid:IDN+emulab.net+authority+cm)
INFO:stitcher:Adding aggregate option https://boss.utah.geniracks.net:12369/protogeni/xmlrpc/am (urn:publicid:IDN+utah.geniracks.net+authority+cm)
INFO:omni:Loading config file /home/lnevers/.gcf/omni_config
INFO:omni:Using control framework portal
DEBUG:omni:Using framework type pgch
DEBUG:omni.protogeni:Configured with key file /home/lnevers/.ssl/geni_cert_portal-1.pem
DEBUG:omni.protogeni:Using clearinghouse https://ch.geni.net:8443/
DEBUG:omni.protogeni:Using slice authority https://ch.geni.net:8443/
DEBUG:omni:User Cert File: /home/lnevers/.ssl/geni_cert_portal-1.pem
INFO:omni:Python: CPython 2.6.5
OS: Linux-2.6.32-47-generic-pae-i686-with-Ubuntu-10.04-lucid
Omni: GENI Omni Command Line Aggregate Manager Tool Version 2.3.1
Copyright (c) 2013 Raytheon BBN Technologies
DEBUG:omni:Read GetVersionCache from /home/lnevers/.gcf/get_version_cache.json
DEBUG:omni:Checking cache for https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am
DEBUG:omni:Pulling GetVersion from cache
DEBUG:omni:Checking cache for https://www.emulab.net:12369/protogeni/xmlrpc/am
DEBUG:omni:Pulling GetVersion from cache
DEBUG:omni:Checking cache for https://boss.utah.geniracks.net:12369/protogeni/xmlrpc/am
DEBUG:omni:Pulling GetVersion from cache
DEBUG:omni:Checking cache for http://geni-am.net.internet2.edu:12346
DEBUG:omni:Pulling GetVersion from cache
DEBUG:omni:Config version spoken here by all AMs
DEBUG:omni:No filename provided for credential
DEBUG:omni.pgch:Getting user credential from GENI Clearinghouse SA https://ch.geni.net:8443/
DEBUG:omni.pgch:Resolving urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln-ext1 at slice authority
 ....

DEBUG:omni.pgch:Resolved slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln-ext1, getting credential
INFO:omni:Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln-ext1 expires within 1 day on 2013-06-07 00:50:54 UTC
DEBUG:omni:Doing shutdown with urn urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln-ext1, 1 creds, options {}
DEBUG:omni:Doing SSL/XMLRPC call to https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am invoking Shutdown
ERROR:omni.pgch:Call for Shutdown urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln-ext1 on https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am failed. Server says: <Fault -32600: 'Internal Error executing Shutdown'>
DEBUG:omni:Raw result from AM API call was None?!
WARNING:omni:Failed to shutdown sliver urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln-ext1 on AM unspecified_AM_URN at https://boss.instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am. <Fault -32600: 'Internal Error executing Shutdown'> (missing result)
DEBUG:omni:Doing SSL/XMLRPC call to https://www.emulab.net:12369/protogeni/xmlrpc/am invoking Shutdown
ERROR:omni.pgch:Call for Shutdown urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln-ext1 on https://www.emulab.net:12369/protogeni/xmlrpc/am failed. Server says: <Fault -32600: 'Internal Error executing Shutdown'>
DEBUG:omni:Raw result from AM API call was None?!
WARNING:omni:Failed to shutdown sliver urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln-ext1 on AM unspecified_AM_URN++ at https://www.emulab.net:12369/protogeni/xmlrpc/am. <Fault -32600: 'Internal Error executing Shutdown'> (missing result)
DEBUG:omni:Doing SSL/XMLRPC call to https://boss.utah.geniracks.net:12369/protogeni/xmlrpc/am invoking Shutdown
ERROR:omni.pgch:Call for Shutdown urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln-ext1 on https://boss.utah.geniracks.net:12369/protogeni/xmlrpc/am failed. Server says: <Fault -32600: 'Internal Error executing Shutdown'>
DEBUG:omni:Raw result from AM API call was None?!
WARNING:omni:Failed to shutdown sliver urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln-ext1 on AM unspecified_AM_URN+++ at https://boss.utah.geniracks.net:12369/protogeni/xmlrpc/am. <Fault -32600: 'Internal Error executing Shutdown'> (missing result)
DEBUG:omni:Doing SSL/XMLRPC call to http://geni-am.net.internet2.edu:12346 invoking Shutdown
WARNING:omni:Failed to shutdown sliver urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln-ext1 on AM unspecified_AM_URN+ at http://geni-am.net.internet2.edu:12346. Error from Aggregate: code 12. sfa AM code: 12: : Shutdown: Non existing record urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln-ext1, .
Shutdown slivers of slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln-ext1 on 0 of 4 possible aggregates

Change History (3)

comment:1 Changed 6 years ago by lnevers@bbn.com

Summary: Shutdown of stitched slice fails with Internal ErrorShutdown of stitched slice fails with Internal Error at PG and IG aggregates

comment:2 Changed 6 years ago by lnevers@bbn.com

Capturing email exchange for fix:

On 6/6/13 3:32 PM, Jonathon Duerig wrote:

Ok. This was an incorrect identifier. A fix is installed at Utah. Try again. 

On 6/6/13 3:43 PM, Luisa Nevers wrote:

Tried again and here is the response from emulab to the shutdown command:

  DEBUG:omni:Doing SSL/XMLRPC call to https://www.emulab.net:12369/protogeni/xmlrpc/am 
  invoking Shutdown
  INFO:omni: (PG log url - look here for details on any failures: https://www.emulab.net
  /spewlogfile.php3?logfile=b1e0f95ec259e38fd530c44fee24f555)

and to a subsequent sliverstatus:

  DEBUG:omni:Doing SSL/XMLRPC call to https://www.emulab.net:12369/protogeni/xmlrpc/am/2.0 
  invoking SliverStatus
  Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln-ext1 expires on 2013-06-08 00:00:00 UTC

  Failed to get SliverStatus on ln-ext1 at AM https://www.emulab.net:12369/protogeni/xmlrpc/am/2.0: 
  Error from
   Aggregate: code 12. protogeni AM code: 12: No slice or aggregate here (PG log url - look here for 
  details on any
  failures: https://www.emulab.net/spewlogfile.php3?logfile=0a3be44877d4b82a15b0912e67248aa0).


The sliver is deleted after the shutdown, as expected. 

comment:3 Changed 6 years ago by lnevers@bbn.com

Resolution: fixed
Status: newclosed

The shutdown fix was verified at PG Utah using 'code_tag': '0a814e9fa25f2335fce9122911946353e70090b6', which will propagate to InstaGENI rack in future updates. Problem is resolved. Closing ticket.

Note: See TracTickets for help on using tickets.