Opened 5 years ago

Closed 5 years ago

#1350 closed (fixed)

Deleting slivers at OESS aggregate causes Gateway Time-out failure

Reported by: lnevers@bbn.com Owned by: luke@grnoc.iu.edu
Priority: major Milestone:
Component: STITCHING Version: SPIRAL7
Keywords: GENI Network Stitching Cc:
Dependencies:

Description

Deleting complex topologies at the OESS aggregate reports a "Gateway timeout error" error:

10/14 09:07:42 ERROR    dossl.py:206 Call for DeleteSliver urn:publicid:IDN+ch.geni.net:ln-test+slice+4linear at
http://foam-oess-stage.grnoc.iu.edu:3626/foam/gapi/2 failed.: ProtocolError: <ProtocolError for
foam-oess-stage.grnoc.iu.edu:3626/foam/gapi/2: 504 Gateway Time-out>
10/14 09:07:42 DEBUG    dossl.py:209 Traceback (most recent call last):
  File "/home/lnevers/gcf-2.7-rc/src/gcf/omnilib/util/dossl.py", line
70, in _do_ssl
    result = fn(*args)
  File "/usr/lib/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib/python2.6/xmlrpclib.py", line 1243, in request
    headers
ProtocolError: <ProtocolError for
foam-oess-stage.grnoc.iu.edu:3626/foam/gapi/2: 504 Gateway Time-out>

10/14 09:07:42 DEBUG    amhandler.py:5310 Raw result from AM API call was None?!
10/14 09:07:42 WARNING  amhandler.py:4073 Failed to delete sliver urn:publicid:IDN+ch.geni.net:ln-test+slice+4linear at oess (got result
'None'). ProtocolError: <ProtocolError for foam-oess-stage.grnoc.iu.edu:3626/foam/gapi/2: 504 Gateway Time-out>
(missing result)

The deletsliver is actually successful, and this timeout is not the SSL timeout in omni (360 seconds).

The two topologies that show the above failure on every deletesliver command are:

  • 4 node linear
  • 5 node star

Change History (6)

comment:1 Changed 5 years ago by lnevers@bbn.com

Summary: Deleting complex slivers at OESS aggregate causes failureDeleting slivers at OESS aggregate causes Gateway Time-out failure

This problem is also happening for 3 nodes linear topologies and intermittently for 2 nodes point to point topologies (al2s-only).

comment:2 Changed 5 years ago by luke@grnoc.iu.edu

I'll take a look at this today and try to either determine if the delete operation can be sped up or if not how to increase the timeout.

comment:3 Changed 5 years ago by luke@grnoc.iu.edu

Increased timeout in nginx to 360 seconds, which should mitigate the issue for all but very complex requests.

Will ask OESS team to investigate improving performance of delete call. Currently taking ~20 seconds per circuit.

comment:4 Changed 5 years ago by lnevers@bbn.com

Running some tests now.

comment:5 Changed 5 years ago by lnevers@bbn.com

Problem is solved, closing ticket.

comment:6 Changed 5 years ago by lnevers@bbn.com

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.