Opened 5 years ago

Last modified 4 years ago

#1156 new

Unable to stitch multiple link from GPO ExoGENI rack

Reported by: lnevers@bbn.com Owned by: lnevers@bbn.com
Priority: major Milestone:
Component: STITCHING Version: SPIRAL6
Keywords: Network Stitching Cc: exogeni-design@geni.net, divyashri.bhat@gmail.com
Dependencies:

Description

When creating a stitched sliver from EG GPO to IG Utah with 2 links between the 2 nodes, the following error is reported:

10:07:24 INFO stitch.Aggregate: Stitcher doing createsliver at https://bbn-hn.exogeni.net:11443/orca/xmlrpc
10:07:28 ERROR    omni:  {'output': 'ERROR: Exception encountered: 
orca.ndl.NdlException: Interface bbnNet.rdf#BbnNet/IBM/G8052/GigabitEthernet
/1/2/ethernet should be owned by exactly two entities (a node and a connection/vlan),
 instead: 4', 'code': {'geni_code': 2}}
10:07:28 INFO     stitch.Aggregate: Got AMAPIError doing createsliver lndivy at 
<Aggregate urn:publicid:IDN+exogeni.net:bbnvmsite+authority+am>: AMAPIError: Error 
from Aggregate: code 2: ERROR: Exception encountered: orca.ndl.NdlException: Interface 
bbnNet.rdf#BbnNet/IBM/G8052/GigabitEthernet/1/2/ethernet should be owned by exactly 
two entities (a node and a connection/vlan), instead: 4. 

The original RSpec (stitch-eg-gpo-ig-gpo-2l.rspec) and the request generated by the stitcher and submitted to EG aggregate (ndivy-createsliver-request-11-exogeni-net-bbnvmsite.xml) are attached to this ticket.

Attachments (3)

lndivy-createsliver-request-11-exogeni-net-bbnvmsite.xml (13.3 KB) - added by lnevers@bbn.com 5 years ago.
stitch-eg-gpo-ig-utah-2l.rspec (1.7 KB) - added by lnevers@bbn.com 5 years ago.
Stitch5nodes-ln2.rspec (7.5 KB) - added by lnevers@bbn.com 5 years ago.

Download all attachments as: .zip

Change History (9)

Changed 5 years ago by lnevers@bbn.com

Changed 5 years ago by lnevers@bbn.com

comment:1 Changed 5 years ago by lnevers@bbn.com

This issue was discovered while investigating the errors reported by Divy for a 5 node stitched topology with 1 node at IG Utah and 4 nodes at ExoGENI GPO. Error differed, and failures changed with each attempt, will verify the support of multiple links by the ExoGENI GPO rack by verifying solution with Divy's RSpec, attached (Stitch5nodes-ln2.rspec).

Changed 5 years ago by lnevers@bbn.com

Attachment: Stitch5nodes-ln2.rspec added

comment:2 Changed 5 years ago by divyashri.bhat@gmail.com

When I tried this again after the recent maintenance on ExoGENI racks, this is what I get. It seems the link between IG-Utah and EG-GPO is being rejected.

dbhat@emmy9:~/Omni/gcf/src$ python stitcher.py createsliver testexostitch Stitch5nodes-ln2.rspec 
14:04:43 INFO     stitcher: Loading agg_nick_cache file '/home/dbhat/.gcf/agg_nick_cache'
14:04:43 INFO     stitcher: Loading config file /home/dbhat/.gcf/omni_config
14:04:43 INFO     stitcher: Using control framework portal
14:04:43 INFO     stitcher: Checking that slice testexostitch is valid...
14:04:43 INFO     stitcher: Slice urn:publicid:IDN+ch.geni.net:GIMITesting+slice+testexostitch expires on 2014-01-24 19:00:08 UTC
14:04:45 INFO     stitcher: <Aggregate urn:publicid:IDN+emulab.net+authority+cm> speaks AM API v3, but sticking with v2
14:04:45 INFO     stitcher: <Aggregate urn:publicid:IDN+utah.geniracks.net+authority+cm> speaks AM API v3, but sticking with v2
14:04:45 INFO     stitcher: Stitched reservation will include resources from these aggregates:
14:04:45 INFO     stitcher: 	<Aggregate urn:publicid:IDN+ion.internet2.edu+authority+am>
14:04:45 INFO     stitcher: 	<Aggregate urn:publicid:IDN+exogeni.net:bbnvmsite+authority+am>
14:04:45 INFO     stitcher: 	<Aggregate urn:publicid:IDN+emulab.net+authority+cm>
14:04:45 INFO     stitcher: 	<Aggregate urn:publicid:IDN+utah.geniracks.net+authority+cm>
14:04:45 INFO     stitch.Aggregate: Writing to '/tmp/testexostitch-createsliver-request-11-exogeni-net-bbnvmsite.xml'
14:04:45 INFO     stitch.Aggregate: 
	Stitcher doing createsliver at https://bbn-hn.exogeni.net:11443/orca/xmlrpc
14:04:49 ERROR    omni:  {'output': 'ERROR: Exception encountered: orca.ndl.NdlException: Interface bbnNet.rdf#BbnNet/IBM/G8052/GigabitEthernet/1/2/ethernet is not really owned by this resource ab7ed642-f739-4116-b2eb-4b6efdb7b028#link1', 'code': {'geni_code': 2}}
14:04:49 INFO     stitch.Aggregate: Got AMAPIError doing createsliver testexostitch at <Aggregate urn:publicid:IDN+exogeni.net:bbnvmsite+authority+am>: AMAPIError: Error from Aggregate: code 2: ERROR: Exception encountered: orca.ndl.NdlException: Interface bbnNet.rdf#BbnNet/IBM/G8052/GigabitEthernet/1/2/ethernet is not really owned by this resource ab7ed642-f739-4116-b2eb-4b6efdb7b028#link1.
14:04:50 WARNING  stitcher: Stitching failed but will retry: Circuit reservation failed at <Aggregate urn:publicid:IDN+exogeni.net:bbnvmsite+authority+am> (AMAPIError: Error from Aggregate: code 2: ERROR: Exception encountered: orca.ndl.NdlException: Interface bbnNet.rdf#BbnNet/IBM/G8052/GigabitEthernet/1/2/ethernet is not really owned by this resource ab7ed642-f739-4116-b2eb-4b6efdb7b028#link1.). Try again from the SCS
14:04:50 INFO     stitcher: Calling SCS for the 2rd time...
14:04:51 INFO     stitcher: Pausing for 600 seconds for Aggregates to free up resources...

comment:3 Changed 5 years ago by lnevers@bbn.com

Owner: changed from somebody to lnevers@bbn.com

comment:4 Changed 5 years ago by Ilia Baldine

This one will not get fixed until after recovery code is working (may be even after GEC19, depending on how things go). The fix is in the controller code and we don't have the cycles to do it right now, as we are focusing on recovery.

comment:5 Changed 4 years ago by lnevers@bbn.com

Support multiple GENI stitched links is available at the ucdavis-eg site.

First attempt to test fix failed due to stitcher bug (http://trac.gpolab.bbn.com/gcf/ticket/805) which is solved in gcf-2.9.

Using an early version of GCF 2.9, I was able to create a sliver with 2 links and exchange traffic. Also created a 5 node stitched star topology (4 stitched links) with ucdavis-eg as the center node. Was able to exchange traffic on all links.

Keeping ticket open until fix is installed at all EG racks and GCF 2.9 is released.

comment:6 in reply to:  5 Changed 4 years ago by lnevers@bbn.com

Replying to lnevers@…:

Keeping ticket open until fix is installed at all EG racks and GCF 2.9 is released.

The support for multiple GENI stitched links is on all ExoGENI racks. All that remains to do is to make GCF 2.9 available to experimenters.

Note: See TracTickets for help on using tickets.