wiki:GENIRacksHome/CiscoGENIRacks/AcceptanceTestStatus/CG-EXP-4

Version 16 (modified by lnevers@bbn.com, 5 years ago) (diff)

--

CG-EXP-4: GENI Stitching Support

This test allocated resources in both NCSU2 and NCSU Cisco racks. Note the following Omni nick names are used:

ncsu2-eg=urn:publicid:IDN+exogeni.net:ncsu2vmsite+authority+am,https://ncsu2-hn.exogeni.net:11443/orca/xmlrpc
ncsu-eg=urn:publicid:IDN+exogeni.net:ncsuvmsite+authority+am,https://ncsu-hn.exogeni.net:11443/orca/xmlrpc

Test Status

This section captures the status for each step in the acceptance test plan.

Stitching topologies State Tickets/Comments
NCSU2 to NCSU2 N/A Not planned at this time
NCSU2 to GPO EG N/A This scenario is ExoGENI stitching; not available at this time and covered in CG-EXP-2
NCSU2 to IG ION site Color(green,Pass)?Stitching setup; remoteLinkId definition
NCSU2 to IG AL2S site Color(yellow,Complete)? Throughput low
NYSERNet IG<ION-AL2S->NCSU2<-AL2S->Stanford-IGColor(red,Fail)? Know ExoGENI issue http://groups.geni.net/exogeni/ticket/193


State Legend Description
Color(green,Pass)? Test completed and met all criteria
Color(#98FB98,Pass: most criteria)? Test completed and met most criteria. Exceptions documented
Color(red,Fail)? Test completed and failed to meet criteria.
Color(yellow,Complete)? Test completed but will require re-execution due to expected changes
Color(orange,Blocked)? Blocked by ticketed issue(s).
Color(#63B8FF,In Progress)? Currently under test.


Test Details

Testing scenario cover both ExoGENI and GENI stitching. A mix of test topologies are run with the GENI Stitching feature.

NCSU2 to GPO EG

This is ExoGENI Stitching, which is covered in CG-EXP-2. Using the RSpec cisco-ncsu2-eg-gpo.rspec tried to create an ExoGENI stitched sliver via the ExoSM:

$ omni createsliver -a eg-sm CG-CT-4a ./cisco-ncsu2-eg-gpo.rspec 
14:25:14 INFO    : Loading agg_nick_cache file '/home/lnevers/.gcf/agg_nick_cache'
14:25:14 INFO    : Loading config file '/home/lnevers/.gcf/omni_config'
14:25:14 INFO    : Using control framework portal
14:25:14 INFO    : Member Authority is https://ch.geni.net/MA (from config)
14:25:14 INFO    : Slice Authority is https://ch.geni.net/SA (from config)
14:25:14 INFO    : Substituting AM nickname eg-sm with URL https://geni.renci.org:11443/orca/xmlrpc, URN urn:publicid:IDN+exogeni.net+authority+am
14:25:14 INFO    : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+CG-CT-4a expires on 2015-01-16 19:24:40 UTC
14:25:14 INFO    : Creating sliver(s) from rspec file ./cisco-ncsu2-eg-gpo.rspec for slice urn:publicid:IDN+ch.geni.net:ln-test+slice+CG-CT-4a
14:25:20 WARNING : Failed CreateSliver for slice CG-CT-4a at eg-sm.  Error from Aggregate: code 2: Embedding workflow ERROR: 4:No Edge Domain Exist:http://geni-orca.renci.org/owl/ec743afb-fa45-4031-ab4c-23b0f5df7806#eg-gpo:http://geni-orca.renci.org/owl/ec743afb-fa45-4031-ab4c-23b0f5df7806#cg-ncsu2.
 Please see https://geni-orca.renci.org/trac/wiki/orca-errors for possible solutions.
14:25:20 INFO    :  ------------------------------------------------------
14:25:20 INFO    :  Completed createsliver:

  Options as run:
		aggregate: ['eg-sm']
		framework: portal
		project: ln-test

  Args: createsliver CG-CT-4a ./cisco-ncsu2-eg-gpo.rspec

  Result Summary: Failed CreateSliver for slice CG-CT-4a at eg-sm.  Error from Aggregate: code 2: Embedding workflow ERROR: 4:No Edge Domain Exist:http://geni-orca.renci.org/owl/ec743afb-fa45-4031-ab4c-23b0f5df7806#eg-gpo:http://geni-orca.renci.org/owl/ec743afb-fa45-4031-ab4c-23b0f5df7806#cg-ncsu2.
 Please see https://geni-orca.renci.org/trac/wiki/orca-errors for possible solutions. 

NCSU2 to IG ION site

Using the RSpec stitch-eg-ncsu2-ig-gpo.rspec the following topology was created:

GPO IG <-ION-><-AL2S->NCSU2

The stitcher was used to create the sliver:

$ stitcher createsliver CG-CT-4 ./stitch-eg-ncsu2-ig-gpo.rspec --defaultCapacity=100000 -o
09:14:24 INFO    : Configured logging from file /home/lnevers/gcf-2.8-pre1/src/gcf/stitcher_logging.conf
09:14:24 INFO    : Reading slice CG-CT-4 credential...
09:14:24 INFO    : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+CG-CT-4 expires on 2015-01-26 12:56:18 UTC
09:14:24 INFO    : Using SCS at https://nutshell.maxgigapop.net:8443/geni/xmlrpc
09:14:24 INFO    : Calling SCS...
09:15:26 INFO    : Multi-AM reservation will include resources from these aggregates:
09:15:26 INFO    : 	<Aggregate ncsu2-eg>
09:15:26 INFO    : 	<Aggregate gpo-ig>
09:15:26 INFO    : 	<Aggregate oess>
09:15:26 INFO    : 	<Aggregate ion>
09:15:26 INFO    : Stitcher doing createsliver at <Aggregate ncsu2-eg>...
09:15:32 INFO    : ... Allocation at <Aggregate ncsu2-eg> complete.
09:15:32 INFO    : Stitcher doing createsliver at <Aggregate gpo-ig>...
09:15:52 INFO    : ... Allocation at <Aggregate gpo-ig> complete.
09:15:52 INFO    : Stitcher doing createsliver at <Aggregate oess>...
09:15:57 INFO    : ... Allocation at <Aggregate oess> complete.
09:15:57 INFO    : Stitcher doing createsliver at <Aggregate ion>...
09:16:43 INFO    : DCN AM <Aggregate ion>: must wait for status ready....
09:16:43 INFO    : Pausing 30 seconds to let circuit become ready...
09:17:21 INFO    : DCN circuit 149001 is ready at <Aggregate ion>
09:17:29 INFO    : ... Allocation at <Aggregate ion> complete.
09:17:29 INFO    : All aggregates are complete.
09:17:29 INFO    : Your resources expire at 2015-01-25T14:05:26 (UTC). 
09:17:29 INFO    : Saved combined reservation RSpec at 4 AMs to file '/home/lnevers/gcf-test/cisco/CG-CT-4-manifest-rspec-multiam-combined.xml'
Success: Reserved resources in slice CG-CT-4 at 4 Aggregates (including 2 intermediate aggregate(s) not in the original request), creating 1 stitched link(s).

Determine login for NCSU2 host:

$ readyToLogin CG-CT-4 -a ncsu2-eg
...
eg-ncsu2's geni_status is: ready 
User lnevers logs in to eg-ncsu2 using:
	ssh -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@152.48.13.191

Determine login for GPO IG host:

$ readyToLogin CG-CT-4 -a gpo-ig
...
ig-gpo's geni_status is: ready 
User lnevers logs in to ig-gpo using:
	ssh -p 37946  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc5.instageni.gpolab.bbn.com

Measurements

Iperf GPO IG VM to NCSU2 CG (TCP) - TCP window size: 23.5 KByte (default)

Collected: 2015-01-19

One Client_

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec   715 MBytes   100 Mbits/sec

Five Clients

[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-60.0 sec   162 MBytes  22.6 Mbits/sec
[  3]  0.0-60.1 sec   148 MBytes  20.7 Mbits/sec
[  4]  0.0-60.1 sec   150 MBytes  20.9 Mbits/sec
[  7]  0.0-60.1 sec   117 MBytes  16.4 Mbits/sec
[  5]  0.0-60.4 sec   107 MBytes  14.9 Mbits/sec
[SUM]  0.0-60.4 sec   685 MBytes  95.0 Mbits/sec

Ten Clients

[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-60.0 sec  82.2 MBytes  11.5 Mbits/sec
[ 11]  0.0-60.1 sec  53.8 MBytes  7.51 Mbits/sec
[  9]  0.0-60.1 sec  81.9 MBytes  11.4 Mbits/sec
[  5]  0.0-60.1 sec  54.2 MBytes  7.57 Mbits/sec
[  8]  0.0-60.2 sec  73.8 MBytes  10.3 Mbits/sec
[  6]  0.0-60.2 sec  70.0 MBytes  9.75 Mbits/sec
[ 10]  0.0-60.2 sec  61.1 MBytes  8.51 Mbits/sec
[  7]  0.0-60.3 sec  64.5 MBytes  8.97 Mbits/sec
[ 12]  0.0-60.4 sec  66.0 MBytes  9.17 Mbits/sec
[  3]  0.0-60.5 sec   101 MBytes  14.0 Mbits/sec
[SUM]  0.0-60.5 sec   708 MBytes  98.3 Mbits/sec

Iperf GPO IG VM to NCSU2 CG (UDP) - UDP buffer size: 224 KByte (default)

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec   719 MBytes   101 Mbits/sec
[  3] Sent 512809 datagrams
[  3] Server Report:
[  3]  0.0-60.2 sec   693 MBytes  96.6 Mbits/sec  15.508 ms 18139/512808 (3.5%)
[  3]  0.0-60.2 sec  1 datagrams received out-of-order

Ping from GPO IG VM to the NCSU2 CG

60 packets transmitted, 60 received, 0% packet loss, time 59103ms
rtt min/avg/max/mdev = 17.292/17.562/17.902/0.181 ms

Iperf NCSU2 CG to GPO IG (TCP) - TCP window size: 85.5 KByte (default)

Collected: 2015-01-19

One Client_

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.1 sec  95.9 MBytes  13.4 Mbits/sec

Five Clients

[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-60.0 sec   112 MBytes  15.7 Mbits/sec
[  3]  0.0-60.1 sec   107 MBytes  15.0 Mbits/sec
[  7]  0.0-60.1 sec   118 MBytes  16.4 Mbits/sec
[  4]  0.0-60.1 sec   110 MBytes  15.4 Mbits/sec
[  6]  0.0-60.1 sec   105 MBytes  14.7 Mbits/sec
[SUM]  0.0-60.1 sec   552 MBytes  77.1 Mbits/sec

Ten Clients

[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-60.0 sec  69.2 MBytes  9.68 Mbits/sec
[  4]  0.0-60.1 sec  69.6 MBytes  9.72 Mbits/sec
[  7]  0.0-60.1 sec  70.1 MBytes  9.79 Mbits/sec
[  9]  0.0-60.1 sec  75.4 MBytes  10.5 Mbits/sec
[  8]  0.0-60.1 sec  69.9 MBytes  9.75 Mbits/sec
[  5]  0.0-60.2 sec  77.0 MBytes  10.7 Mbits/sec
[  3]  0.0-60.2 sec  64.1 MBytes  8.94 Mbits/sec
[ 11]  0.0-60.2 sec  63.0 MBytes  8.78 Mbits/sec
[ 12]  0.0-60.2 sec  67.9 MBytes  9.46 Mbits/sec
[ 10]  0.0-60.2 sec  66.2 MBytes  9.23 Mbits/sec
[SUM]  0.0-60.2 sec   692 MBytes  96.5 Mbits/sec

Iperf NCSU2 CG to GPO IG (UDP) - UDP buffer size: 208 KByte (default)

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec   719 MBytes   101 Mbits/sec
[  3] Sent 512805 datagrams
[  3] Server Report:
[  3]  0.0-60.1 sec   686 MBytes  95.7 Mbits/sec   0.125 ms 23571/512804 (4.6%)
[  3]  0.0-60.1 sec  31 datagrams received out-of-order

Ping from GPO IG VM to the NCSU2 CG

60 packets transmitted, 60 received, 0% packet loss, time 59095ms
rtt min/avg/max/mdev = 17.263/17.660/19.099/0.300 ms

NCSU2 to IG AL2S site

Using the RSpec stitch-eg-ncsu2-ig-chic.rspec created a stitched sliver to a remote AL2S site:

$ stitcher createsliver CG-CT-4x ./stitch-eg-ncsu2-ig-chic.rspec --defaultCapacity=100000 -o
08:36:29 INFO    : Configured logging from file /home/lnevers/gcf-2.8-pre1/src/gcf/stitcher_logging.conf
08:36:30 INFO    : Reading slice CG-CT-4x credential...
08:36:30 INFO    : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+CG-CT-4x expires on 2015-01-26 13:16:54 UTC
08:36:30 INFO    : Using SCS at https://nutshell.maxgigapop.net:8443/geni/xmlrpc
08:36:30 INFO    : Calling SCS...
08:37:31 INFO    : Multi-AM reservation will include resources from these aggregates:
08:37:31 INFO    : 	<Aggregate ncsu2-eg>
08:37:31 INFO    : 	<Aggregate chicago-ig>
08:37:31 INFO    : 	<Aggregate oess>
08:37:31 INFO    : Stitcher doing createsliver at <Aggregate ncsu2-eg>...
08:37:36 INFO    : ... Allocation at <Aggregate ncsu2-eg> complete.
08:37:36 INFO    : Stitcher doing createsliver at <Aggregate chicago-ig>...
08:37:56 INFO    : ... Allocation at <Aggregate chicago-ig> complete.
08:37:56 INFO    : Stitcher doing createsliver at <Aggregate oess>...
08:38:01 INFO    : ... Allocation at <Aggregate oess> complete.
08:38:01 INFO    : All aggregates are complete.
08:38:02 INFO    : Your resources expire at 2015-01-25T13:27:31 (UTC). 
08:38:02 INFO    : Saved combined reservation RSpec at 3 AMs to file '/home/lnevers/gcf-test/cisco/CG-CT-4x-manifest-rspec-multiam-combined.xml'

Determine login:

$ readyToLogin CG-CT-4x -a chicago-ig 
...
ig-chic's geni_status is: ready 
User lnevers logs in to ig-chic using:
	ssh -p 30266  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc3.geni.uchicago.edu

$ readyToLogin CG-CT-4x -a ncsu2-eg
...
eg-ncsu2's geni_status is: ready 
User lnevers logs in to eg-ncsu2 using:
	ssh -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@152.48.13.194

Measurements

Iperf Chicago IG VM to NCSU2 CG (TCP) - TCP window size: 23.5 KByte (default)

Collected: 2015-01-19

One Client

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec   697 MBytes  97.4 Mbits/sec

Five Clients

[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-60.1 sec   142 MBytes  19.9 Mbits/sec
[  6]  0.0-60.1 sec   113 MBytes  15.8 Mbits/sec
[  3]  0.0-60.1 sec   121 MBytes  16.9 Mbits/sec
[  4]  0.0-60.2 sec   133 MBytes  18.5 Mbits/sec
[  7]  0.0-60.3 sec   188 MBytes  26.1 Mbits/sec
[SUM]  0.0-60.3 sec   698 MBytes  97.1 Mbits/sec

Ten Clients

[ ID] Interval       Transfer     Bandwidth
[ 12]  0.0-60.0 sec  59.0 MBytes  8.25 Mbits/sec
[  7]  0.0-60.1 sec  46.5 MBytes  6.50 Mbits/sec
[ 11]  0.0-60.1 sec  67.9 MBytes  9.48 Mbits/sec
[  6]  0.0-60.1 sec  62.6 MBytes  8.74 Mbits/sec
[  3]  0.0-60.1 sec  68.6 MBytes  9.58 Mbits/sec
[  8]  0.0-60.2 sec  94.4 MBytes  13.1 Mbits/sec
[  5]  0.0-60.3 sec  92.1 MBytes  12.8 Mbits/sec
[ 10]  0.0-60.4 sec  75.9 MBytes  10.5 Mbits/sec
[  4]  0.0-60.5 sec  95.5 MBytes  13.2 Mbits/sec
[  9]  0.0-60.7 sec  52.9 MBytes  7.31 Mbits/sec
[SUM]  0.0-60.7 sec   715 MBytes  98.9 Mbits/sec

Iperf Chicago IG VM to NCSU2 CG (UDP) - UDP buffer size: 224 KByte (default)

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec   719 MBytes   101 Mbits/sec
[  3] Sent 512778 datagrams
[  3] Server Report:
[  3]  0.0-60.0 sec   693 MBytes  96.9 Mbits/sec   0.138 ms 18523/512777 (3.6%)
[  3]  0.0-60.0 sec  1 datagrams received out-of-order

Ping from Chicago IG VM to the NCSU2 CG

60 packets transmitted, 60 received, 0% packet loss, time 59091ms
rtt min/avg/max/mdev = 25.204/25.391/25.798/0.228 ms

Iperf NCSU2 CG to Chicago IG (TCP) - TCP window size: 85.0 KByte (default)

Collected: 2015-01-19

One Client

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.2 sec  66.0 MBytes  9.19 Mbits/sec

Five Clients

[ ID] Interval       Transfer     Bandwidth
[  7]  0.0-60.1 sec  76.5 MBytes  10.7 Mbits/sec
[  4]  0.0-60.1 sec  77.5 MBytes  10.8 Mbits/sec
[  5]  0.0-60.2 sec  81.4 MBytes  11.3 Mbits/sec
[  6]  0.0-60.3 sec  67.4 MBytes  9.38 Mbits/sec
[  3]  0.0-60.3 sec  82.9 MBytes  11.5 Mbits/sec
[SUM]  0.0-60.3 sec   386 MBytes  53.7 Mbits/sec

Ten Clients

[ ID] Interval       Transfer     Bandwidth
[ 12]  0.0-60.2 sec  67.9 MBytes  9.47 Mbits/sec
[ 13]  0.0-60.2 sec  59.0 MBytes  8.23 Mbits/sec
[  7]  0.0-60.2 sec  62.6 MBytes  8.73 Mbits/sec
[  8]  0.0-60.2 sec  46.5 MBytes  6.48 Mbits/sec
[  4]  0.0-60.3 sec  68.6 MBytes  9.54 Mbits/sec
[  9]  0.0-60.5 sec  94.4 MBytes  13.1 Mbits/sec
[  6]  0.0-60.5 sec  92.1 MBytes  12.8 Mbits/sec
[  5]  0.0-60.8 sec  95.5 MBytes  13.2 Mbits/sec
[ 11]  0.0-60.8 sec  75.9 MBytes  10.5 Mbits/sec
[ 10]  0.0-60.8 sec  52.9 MBytes  7.29 Mbits/sec
[SUM]  0.0-60.8 sec   715 MBytes  98.6 Mbits/sec

Iperf NCSU2 CG to Chicago IG (UDP) - UDP buffer size: 208 KByte (default)

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec   719 MBytes   101 Mbits/sec
[  3] Sent 512821 datagrams
[  3] Server Report:
[  3]  0.0-60.1 sec   681 MBytes  95.0 Mbits/sec   0.072 ms 26950/512820 (5.3%)
[  3]  0.0-60.1 sec  1 datagrams received out-of-order

Ping from Chicago IG VM to the NCSU2 CG

60 packets transmitted, 60 received, 0% packet loss, time 59091ms
rtt min/avg/max/mdev = 25.283/25.482/25.966/0.147 ms

Using the RSpec stitch-3linear.rspec requested the following topology:

NYSERNet IG<ION-AL2S->NCSU2<-AL2S->Stanford-IG

The create sliver:

$ stitcher createsliver 3linear ./stitch-3linear.rspec 
15:27:46 INFO    : Configured logging from file /home/lnevers/gcf-2.7/src/gcf/stitcher_logging.conf
15:27:46 INFO    : Reading slice 3linear credential...
15:27:46 INFO    : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+3linear expires on 2015-01-12 20:21:43 UTC
15:27:46 INFO    : Using SCS at http://nutshell.maxgigapop.net:8081/geni/xmlrpc
15:27:46 INFO    : Calling SCS...
15:27:48 INFO    : Multi-AM reservation will include resources from these aggregates:
15:27:48 INFO    : 	<Aggregate ncsu2-eg>
15:27:48 INFO    : 	<Aggregate gpo-ig>
15:27:48 INFO    : 	<Aggregate chicago-ig>
15:27:48 INFO    : 	<Aggregate oess>
15:27:48 INFO    : 	<Aggregate ion>
15:27:48 INFO    : Stitcher doing createsliver at <Aggregate ncsu2-eg>...
15:27:51 ERROR   :  {'output': 'Invalid slice urn urn:publicid:IDN+ch.geni.net:ln-test+slice+3linear', 'code': {'geni_code': 2}}
15:27:51 INFO    : Got an error reserving resources in 3linear at <Aggregate ncsu2-eg>
15:27:52 WARNING : Failed to delete sliver urn:publicid:IDN+ch.geni.net:ln-test+slice+3linear at ncsu2-eg (got result 'False'). (no reason given)
15:27:52 WARNING : Stitching failed but will retry: Circuit reservation failed at <Aggregate ncsu2-eg> (AMAPIError: Error from Aggregate: code 2: Invalid slice urn urn:publicid:IDN+ch.geni.net:ln-test+slice+3linear.). Try again from the SCS