Opened 9 years ago
Closed 9 years ago
#1350 closed (fixed)
Deleting slivers at OESS aggregate causes Gateway Time-out failure
Reported by: | lnevers@bbn.com | Owned by: | luke@grnoc.iu.edu |
---|---|---|---|
Priority: | major | Milestone: | |
Component: | STITCHING | Version: | SPIRAL7 |
Keywords: | GENI Network Stitching | Cc: | |
Dependencies: |
Description
Deleting complex topologies at the OESS aggregate reports a "Gateway timeout error" error:
10/14 09:07:42 ERROR dossl.py:206 Call for DeleteSliver urn:publicid:IDN+ch.geni.net:ln-test+slice+4linear at http://foam-oess-stage.grnoc.iu.edu:3626/foam/gapi/2 failed.: ProtocolError: <ProtocolError for foam-oess-stage.grnoc.iu.edu:3626/foam/gapi/2: 504 Gateway Time-out> 10/14 09:07:42 DEBUG dossl.py:209 Traceback (most recent call last): File "/home/lnevers/gcf-2.7-rc/src/gcf/omnilib/util/dossl.py", line 70, in _do_ssl result = fn(*args) File "/usr/lib/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/lib/python2.6/xmlrpclib.py", line 1489, in __request verbose=self.__verbose File "/usr/lib/python2.6/xmlrpclib.py", line 1243, in request headers ProtocolError: <ProtocolError for foam-oess-stage.grnoc.iu.edu:3626/foam/gapi/2: 504 Gateway Time-out> 10/14 09:07:42 DEBUG amhandler.py:5310 Raw result from AM API call was None?! 10/14 09:07:42 WARNING amhandler.py:4073 Failed to delete sliver urn:publicid:IDN+ch.geni.net:ln-test+slice+4linear at oess (got result 'None'). ProtocolError: <ProtocolError for foam-oess-stage.grnoc.iu.edu:3626/foam/gapi/2: 504 Gateway Time-out> (missing result)
The deletsliver is actually successful, and this timeout is not the SSL timeout in omni (360 seconds).
The two topologies that show the above failure on every deletesliver command are:
- 4 node linear
- 5 node star
Change History (6)
comment:1 Changed 9 years ago by
Summary: | Deleting complex slivers at OESS aggregate causes failure → Deleting slivers at OESS aggregate causes Gateway Time-out failure |
---|
comment:2 Changed 9 years ago by
I'll take a look at this today and try to either determine if the delete operation can be sped up or if not how to increase the timeout.
comment:3 Changed 9 years ago by
Increased timeout in nginx to 360 seconds, which should mitigate the issue for all but very complex requests.
Will ask OESS team to investigate improving performance of delete call. Currently taking ~20 seconds per circuit.
comment:6 Changed 9 years ago by
Resolution: | → fixed |
---|---|
Status: | new → closed |
This problem is also happening for 3 nodes linear topologies and intermittently for 2 nodes point to point topologies (al2s-only).