| Version 12 (modified by , 9 years ago) (diff) |
|---|
CG-EXP-4: GENI Stitching Support
This test allocated resources in both NCSU2 and NCSU Cisco racks. Note the following Omni nick names are used:
ncsu2-eg=urn:publicid:IDN+exogeni.net:ncsu2vmsite+authority+am,https://ncsu2-hn.exogeni.net:11443/orca/xmlrpc ncsu-eg=urn:publicid:IDN+exogeni.net:ncsuvmsite+authority+am,https://ncsu-hn.exogeni.net:11443/orca/xmlrpc
Test Status
This section captures the status for each step in the acceptance test plan.
| Stitching topologies | State | Tickets/Comments | |
| NCSU2 to NCSU2 | N/A | Not planned at this time | |
| NCSU2 to GPO EG | N/A | This scenario is ExoGENI stitching; not available at this time and covered in CG-EXP-2 | |
| NCSU2 to IG ION site | Color(green,Pass)? | ||
| NCSU2 to IG AL2S site | Color(yellow,Complete)? | Throughput low?? | |
| NYSERNet IG<ION-AL2S->NCSU2<-AL2S->Stanford-IG | Color(red,Fail)? | Know ExoGENI issue http://groups.geni.net/exogeni/ticket/193 |
| State Legend | Description |
| Color(green,Pass)? | Test completed and met all criteria |
| Color(#98FB98,Pass: most criteria)? | Test completed and met most criteria. Exceptions documented |
| Color(red,Fail)? | Test completed and failed to meet criteria. |
| Color(yellow,Complete)? | Test completed but will require re-execution due to expected changes |
| Color(orange,Blocked)? | Blocked by ticketed issue(s). |
| Color(#63B8FF,In Progress)? | Currently under test. |
Test Details
Testing scenario cover both ExoGENI and GENI stitching. A mix of test topologies are run with the GENI Stitching feature.
NCSU2 to GPO EG
This is ExoGENI Stitching, which is covered in CG-EXP-2. Using the RSpec cisco-ncsu2-eg-gpo.rspec tried to create an ExoGENI stitched sliver via the ExoSM:
$ omni createsliver -a eg-sm CG-CT-4a ./cisco-ncsu2-eg-gpo.rspec 14:25:14 INFO : Loading agg_nick_cache file '/home/lnevers/.gcf/agg_nick_cache' 14:25:14 INFO : Loading config file '/home/lnevers/.gcf/omni_config' 14:25:14 INFO : Using control framework portal 14:25:14 INFO : Member Authority is https://ch.geni.net/MA (from config) 14:25:14 INFO : Slice Authority is https://ch.geni.net/SA (from config) 14:25:14 INFO : Substituting AM nickname eg-sm with URL https://geni.renci.org:11443/orca/xmlrpc, URN urn:publicid:IDN+exogeni.net+authority+am 14:25:14 INFO : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+CG-CT-4a expires on 2015-01-16 19:24:40 UTC 14:25:14 INFO : Creating sliver(s) from rspec file ./cisco-ncsu2-eg-gpo.rspec for slice urn:publicid:IDN+ch.geni.net:ln-test+slice+CG-CT-4a 14:25:20 WARNING : Failed CreateSliver for slice CG-CT-4a at eg-sm. Error from Aggregate: code 2: Embedding workflow ERROR: 4:No Edge Domain Exist:http://geni-orca.renci.org/owl/ec743afb-fa45-4031-ab4c-23b0f5df7806#eg-gpo:http://geni-orca.renci.org/owl/ec743afb-fa45-4031-ab4c-23b0f5df7806#cg-ncsu2. Please see https://geni-orca.renci.org/trac/wiki/orca-errors for possible solutions. 14:25:20 INFO : ------------------------------------------------------ 14:25:20 INFO : Completed createsliver: Options as run: aggregate: ['eg-sm'] framework: portal project: ln-test Args: createsliver CG-CT-4a ./cisco-ncsu2-eg-gpo.rspec Result Summary: Failed CreateSliver for slice CG-CT-4a at eg-sm. Error from Aggregate: code 2: Embedding workflow ERROR: 4:No Edge Domain Exist:http://geni-orca.renci.org/owl/ec743afb-fa45-4031-ab4c-23b0f5df7806#eg-gpo:http://geni-orca.renci.org/owl/ec743afb-fa45-4031-ab4c-23b0f5df7806#cg-ncsu2. Please see https://geni-orca.renci.org/trac/wiki/orca-errors for possible solutions.
NCSU2 to IG ION site
Using the RSpec stitch-eg-ncsu2-ig-gpo.rspec the following topology was created:
GPO IG <-ION-><-AL2S->NCSU2
The stitcher was used to create the sliver:
$ stitcher createsliver CG-CT-4 ./stitch-eg-ncsu2-ig-gpo.rspec 14:16:02 INFO : Configured logging from file /home/lnevers/gcf-2.7/src/gcf/stitcher_logging.conf 14:16:02 INFO : Reading slice CG-CT-4 credential... 14:16:03 INFO : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+CG-CT-4 expires on 2015-01-16 19:15:12 UTC 14:16:03 INFO : Using SCS at http://nutshell.maxgigapop.net:8081/geni/xmlrpc 14:16:03 INFO : Calling SCS... 14:16:04 INFO : Multi-AM reservation will include resources from these aggregates: 14:16:04 INFO : <Aggregate ncsu2-eg> 14:16:04 INFO : <Aggregate gpo-ig> 14:16:04 INFO : <Aggregate oess> 14:16:04 INFO : <Aggregate ion> 14:16:04 INFO : Stitcher doing createsliver at <Aggregate ncsu2-eg>... 14:16:09 INFO : ... Allocation at <Aggregate ncsu2-eg> complete. 14:16:09 INFO : Stitcher doing createsliver at <Aggregate gpo-ig>... 14:16:24 INFO : ... Allocation at <Aggregate gpo-ig> complete. 14:16:24 INFO : Stitcher doing createsliver at <Aggregate oess>... 14:16:33 INFO : ... Allocation at <Aggregate oess> complete. 14:16:33 INFO : Stitcher doing createsliver at <Aggregate ion>... 14:17:19 INFO : DCN AM <Aggregate ion>: must wait for status ready.... 14:17:19 INFO : Pausing 30 seconds to let circuit become ready... 14:17:56 INFO : DCN circuit 146681 is ready at <Aggregate ion> 14:18:04 INFO : ... Allocation at <Aggregate ion> complete. 14:18:04 INFO : All aggregates are complete. 14:18:04 INFO : Your resources expire at 2015-01-15T19:06:04 (UTC). 14:18:04 INFO : Saved combined reservation RSpec at 4 AMs to file '/home/lnevers/gcf-test/cisco/CG-CT-4-manifest-rspec-multiam-combined.xml' Success: Reserved resources in slice CG-CT-4 at 4 Aggregates (including 2 intermediate aggregate(s) not in the original request), creating 1 stitched link(s).
Determine login for NCSU2 host:
$ readyToLogin CG-CT-4 -a ncsu2-eg ... eg-ncsu2's geni_status is: ready User lnevers logs in to eg-ncsu2 using: ssh -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@152.48.13.191
Determine login for GPO IG host:
$ readyToLogin CG-CT-4 -a gpo-ig ... ig-gpo's geni_status is: ready User lnevers logs in to ig-gpo using: ssh -p 33338 -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc5.instageni.gpolab.bbn.com
Measurements
Iperf GPO IG VM to NCSU2 CG (TCP) - TCP window size: 23.5 KByte (default)
Collected: 2015-01-09
One Client_
[ ID] Interval Transfer Bandwidth [ 3] 0.0-60.0 sec 715 MBytes 100 Mbits/sec
Five Clients
[ ID] Interval Transfer Bandwidth [ 6] 0.0-60.0 sec 119 MBytes 16.6 Mbits/sec [ 4] 0.0-60.1 sec 124 MBytes 17.4 Mbits/sec [ 5] 0.0-60.2 sec 161 MBytes 22.4 Mbits/sec [ 7] 0.0-60.2 sec 122 MBytes 16.9 Mbits/sec [ 3] 0.0-60.2 sec 172 MBytes 24.0 Mbits/sec [SUM] 0.0-60.2 sec 698 MBytes 97.3 Mbits/sec
Ten Clients
[ ID] Interval Transfer Bandwidth [ 6] 0.0-60.0 sec 59.2 MBytes 8.28 Mbits/sec [ 10] 0.0-60.0 sec 87.6 MBytes 12.2 Mbits/sec [ 11] 0.0-60.1 sec 75.9 MBytes 10.6 Mbits/sec [ 3] 0.0-60.1 sec 62.5 MBytes 8.73 Mbits/sec [ 12] 0.0-60.1 sec 66.0 MBytes 9.21 Mbits/sec [ 7] 0.0-60.3 sec 71.1 MBytes 9.89 Mbits/sec [ 5] 0.0-60.4 sec 47.2 MBytes 6.56 Mbits/sec [ 4] 0.0-60.5 sec 60.0 MBytes 8.31 Mbits/sec [ 9] 0.0-60.7 sec 85.8 MBytes 11.9 Mbits/sec [ 8] 0.0-61.0 sec 91.2 MBytes 12.6 Mbits/sec [SUM] 0.0-61.0 sec 707 MBytes 97.2 Mbits/sec
Iperf GPO IG VM to NCSU2 CG (UDP) - UDP buffer size: 224 KByte (default)
[ ID] Interval Transfer Bandwidth [ 3] 0.0-60.0 sec 719 MBytes 101 Mbits/sec [ 3] Sent 512809 datagrams [ 3] Server Report: [ 3] 0.0-60.0 sec 694 MBytes 97.0 Mbits/sec 0.121 ms 17641/512808 (3.4%) [ 3] 0.0-60.0 sec 1 datagrams received out-of-order
Ping from GPO IG VM to the NCSU2 CG
60 packets transmitted, 60 received, 0% packet loss, time 59099ms rtt min/avg/max/mdev = 17.163/17.569/18.046/0.234 ms
Iperf NCSU2 CG to GPO IG (TCP) - TCP window size: 85.5 KByte (default)
Collected: 2015-01-09
One Client_
[ ID] Interval Transfer Bandwidth [ 3] 0.0-60.1 sec 35.0 MBytes 4.89 Mbits/sec
Five Clients
[ ID] Interval Transfer Bandwidth [ 6] 0.0-60.0 sec 37.0 MBytes 5.17 Mbits/sec [ 3] 0.0-60.2 sec 38.0 MBytes 5.30 Mbits/sec [ 7] 0.0-60.3 sec 33.6 MBytes 4.68 Mbits/sec [ 4] 0.0-60.4 sec 36.8 MBytes 5.11 Mbits/sec [ 5] 0.0-60.7 sec 36.9 MBytes 5.10 Mbits/sec [SUM] 0.0-60.7 sec 182 MBytes 25.2 Mbits/sec
Ten Clients
[ ID] Interval Transfer Bandwidth [ 3] 0.0-60.2 sec 38.6 MBytes 5.38 Mbits/sec [ 7] 0.0-60.2 sec 37.6 MBytes 5.24 Mbits/sec [ 11] 0.0-60.2 sec 38.2 MBytes 5.33 Mbits/sec [ 8] 0.0-60.2 sec 38.0 MBytes 5.29 Mbits/sec [ 12] 0.0-60.3 sec 37.8 MBytes 5.26 Mbits/sec [ 9] 0.0-60.3 sec 37.6 MBytes 5.24 Mbits/sec [ 4] 0.0-60.3 sec 38.1 MBytes 5.31 Mbits/sec [ 10] 0.0-60.3 sec 40.0 MBytes 5.56 Mbits/sec [ 5] 0.0-60.4 sec 37.6 MBytes 5.23 Mbits/sec [ 6] 0.0-60.6 sec 37.8 MBytes 5.23 Mbits/sec [SUM] 0.0-60.6 sec 381 MBytes 52.8 Mbits/sec
Note: If a larger TCP window size was specified, was able to get much better throughput.
Iperf NCSU2 CG to GPO IG (UDP) - UDP buffer size: 208 KByte (default)
[ ID] Interval Transfer Bandwidth [ 3] 0.0-60.0 sec 719 MBytes 101 Mbits/sec [ 3] Sent 512821 datagrams [ 3] Server Report: [ 3] 0.0-60.1 sec 685 MBytes 95.6 Mbits/sec 0.067 ms 24133/512820 (4.7%) [ 3] 0.0-60.1 sec 9 datagrams received out-of-order
Ping from GPO IG VM to the NCSU2 CG
60 packets transmitted, 60 received, 0% packet loss, time 59097ms rtt min/avg/max/mdev = 17.201/17.547/18.112/0.214 ms
NCSU2 to IG AL2S site
Using the RSpec stitch-eg-ncsu2-ig-chic.rspec created a stitched sliver to a remote AL2S site:
$ stitcher createsliver CG-CT-4x ./stitch-eg-ncsu2-ig-chic.rspec
15:14:21 INFO : Configured logging from file /home/lnevers/gcf-2.7/src/gcf/stitcher_logging.conf
15:14:21 INFO : Reading slice CG-CT-4z credential...
15:14:21 INFO : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+CG-CT-4z expires on 2015-01-16 20:14:20 UTC
15:14:21 INFO : Using SCS at http://nutshell.maxgigapop.net:8081/geni/xmlrpc
15:14:21 INFO : Calling SCS...
15:14:22 INFO : Multi-AM reservation will include resources from these aggregates:
15:14:22 INFO : <Aggregate ncsu2-eg>
15:14:22 INFO : <Aggregate chicago-ig>
15:14:22 INFO : <Aggregate oess>
15:14:22 INFO : Stitcher doing createsliver at <Aggregate ncsu2-eg>...
15:14:27 INFO : ... Allocation at <Aggregate ncsu2-eg> complete.
15:14:27 INFO : Stitcher doing createsliver at <Aggregate chicago-ig>...
15:14:46 INFO : ... Allocation at <Aggregate chicago-ig> complete.
15:14:46 INFO : Stitcher doing createsliver at <Aggregate oess>...
15:14:49 ERROR : {'output': 'Exception: requested VLAN unavailable: sdn-sw.star.net.internet2.edu,eth7/1 VLAN=3076', 'code': {'geni_code': 24}}
15:14:49 WARNING : Stitching failed but will retry: Circuit reservation failed at <Aggregate oess>. Try again from the SCS. (Error: Topology too complex for local redo - ask Stitching Service to find a VLAN tag (AMAPIError: Error from Aggregate: code 24: Exception: requested VLAN unavailable: sdn-sw.star.net.internet2.edu,eth7/1 VLAN=3076.))
15:14:49 INFO : Deleting existing reservations...
15:14:49 INFO : Doing deletesliver at <Aggregate ncsu2-eg>...
15:14:50 INFO : Deleted reservation at <Aggregate ncsu2-eg>.
15:14:50 INFO : Doing deletesliver at <Aggregate chicago-ig>...
15:16:20 ERROR : {'output': 'start/restart in progress; try again later', 'code': {'protogeni_error_log': 'urn:publicid:IDN+geni.uchicago.edu+log+d1498aed6f901cd5bbe792a8c9dde5b6', 'am_type': 'protogeni', 'geni_code': 14, 'am_code': 14, 'protogeni_error_url': 'https://www.geni.uchicago.edu/spewlogfile.php3?logfile=d1498aed6f901cd5bbe792a8c9dde5b6'}, 'value': 0}
15:18:00 INFO : Deleted reservation at <Aggregate chicago-ig>.
15:18:00 INFO : Calling SCS for the 2nd time...
15:18:00 INFO : Pausing for 30 seconds for Aggregates to free up resources...
15:18:30 INFO : Multi-AM reservation will include resources from these aggregates:
15:18:30 INFO : <Aggregate ncsu2-eg>
15:18:30 INFO : <Aggregate chicago-ig>
15:18:30 INFO : <Aggregate oess>
15:18:30 INFO : Stitcher doing createsliver at <Aggregate ncsu2-eg>...
15:18:35 INFO : ... Allocation at <Aggregate ncsu2-eg> complete.
15:18:35 INFO : Stitcher doing createsliver at <Aggregate chicago-ig>...
15:19:14 INFO : ... Allocation at <Aggregate chicago-ig> complete.
15:19:14 INFO : Stitcher doing createsliver at <Aggregate oess>...
15:19:19 INFO : ... Allocation at <Aggregate oess> complete.
15:19:19 INFO : All aggregates are complete.
15:19:19 INFO : Your resources expire at 2015-01-15T20:08:30 (UTC).
15:19:19 INFO : Saved combined reservation RSpec at 3 AMs to file '/home/lnevers/gcf-test/cisco/CG-CT-4z-manifest-rspec-multiam-combined.xml'
Success: Reserved resources in slice CG-CT-4z at 3 Aggregates (including 1 intermediate aggregate(s) not in the original request), creating 1 stitched link(s).
Determine login:
$ readyToLogin CG-CT-4z -a chicago-ig ... ig-chic's geni_status is: ready User lnevers logs in to ig-chic using: ssh -p 30266 -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc3.geni.uchicago.edu $ readyToLogin CG-CT-4z -a ncsu2-eg ... eg-ncsu2's geni_status is: ready User lnevers logs in to eg-ncsu2 using: ssh -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@152.48.13.191
Measurements
Iperf Chicago IG VM to NCSU2 CG (TCP) - TCP window size: 23.5 KByte (default)
Collected: 2015-01-09
One Client
[ ID] Interval Transfer Bandwidth [ 3] 0.0-60.0 sec 680 MBytes 95.1 Mbits/sec
Five Clients
[ ID] Interval Transfer Bandwidth [ 4] 0.0-60.0 sec 158 MBytes 22.1 Mbits/sec [ 6] 0.0-60.1 sec 151 MBytes 21.1 Mbits/sec [ 3] 0.0-60.1 sec 145 MBytes 20.2 Mbits/sec [ 7] 0.0-60.4 sec 133 MBytes 18.5 Mbits/sec [ 5] 0.0-60.5 sec 120 MBytes 16.6 Mbits/sec [SUM] 0.0-60.5 sec 707 MBytes 98.0 Mbits/sec
Ten Clients
[ ID] Interval Transfer Bandwidth [ 4] 0.0-60.1 sec 71.2 MBytes 9.95 Mbits/sec [ 10] 0.0-60.1 sec 62.9 MBytes 8.78 Mbits/sec [ 12] 0.0-60.1 sec 67.2 MBytes 9.39 Mbits/sec [ 6] 0.0-60.1 sec 91.9 MBytes 12.8 Mbits/sec [ 9] 0.0-60.2 sec 85.1 MBytes 11.9 Mbits/sec [ 8] 0.0-60.2 sec 59.4 MBytes 8.28 Mbits/sec [ 11] 0.0-60.2 sec 67.4 MBytes 9.39 Mbits/sec [ 3] 0.0-60.3 sec 65.6 MBytes 9.13 Mbits/sec [ 7] 0.0-60.3 sec 83.5 MBytes 11.6 Mbits/sec [ 5] 0.0-60.4 sec 65.2 MBytes 9.06 Mbits/sec [SUM] 0.0-60.4 sec 720 MBytes 99.9 Mbits/sec
Iperf Chicago IG VM to NCSU2 CG (UDP) - UDP buffer size: 224 KByte (default)
[ ID] Interval Transfer Bandwidth [ 3] 0.0-60.0 sec 719 MBytes 101 Mbits/sec [ 3] Sent 512778 datagrams [ 3] Server Report: [ 3] 0.0-60.0 sec 693 MBytes 96.9 Mbits/sec 0.138 ms 18523/512777 (3.6%) [ 3] 0.0-60.0 sec 1 datagrams received out-of-order
Ping from Chicago IG VM to the NCSU2 CG
60 packets transmitted, 60 received, 0% packet loss, time 59091ms rtt min/avg/max/mdev = 25.204/25.391/25.798/0.228 ms
Iperf NCSU2 CG to Chicago IG (TCP) - TCP window size: 85.0 KByte (default)
Collected: 2015-01-09
One Client_
[ ID] Interval Transfer Bandwidth [ 3] 0.0-60.7 sec 21.9 MBytes 3.02 Mbits/sec
Five Clients
[ ID] Interval Transfer Bandwidth [ 3] 0.0-60.0 sec 27.1 MBytes 3.79 Mbits/sec [ 4] 0.0-60.5 sec 26.2 MBytes 3.64 Mbits/sec [ 6] 0.0-60.5 sec 28.6 MBytes 3.97 Mbits/sec [ 5] 0.0-60.6 sec 27.0 MBytes 3.74 Mbits/sec [ 7] 0.0-60.6 sec 28.8 MBytes 3.98 Mbits/sec [SUM] 0.0-60.6 sec 138 MBytes 19.1 Mbits/sec
Ten Clients
[ ID] Interval Transfer Bandwidth [ 5] 0.0-60.2 sec 27.5 MBytes 3.83 Mbits/sec [ 10] 0.0-60.2 sec 25.1 MBytes 3.50 Mbits/sec [ 3] 0.0-60.2 sec 27.2 MBytes 3.80 Mbits/sec [ 7] 0.0-60.2 sec 28.5 MBytes 3.97 Mbits/sec [ 12] 0.0-60.3 sec 26.2 MBytes 3.65 Mbits/sec [ 6] 0.0-60.4 sec 26.6 MBytes 3.70 Mbits/sec [ 8] 0.0-60.4 sec 25.0 MBytes 3.47 Mbits/sec [ 4] 0.0-60.5 sec 26.6 MBytes 3.69 Mbits/sec [ 9] 0.0-60.5 sec 26.6 MBytes 3.69 Mbits/sec [ 11] 0.0-60.5 sec 25.6 MBytes 3.55 Mbits/sec [SUM] 0.0-60.5 sec 265 MBytes 36.7 Mbits/sec
Iperf NCSU2 CG to Chicago IG (UDP) - UDP buffer size: 208 KByte (default)
[ ID] Interval Transfer Bandwidth [ 3] 0.0-60.0 sec 719 MBytes 101 Mbits/sec [ 3] Sent 512821 datagrams [ 3] Server Report: [ 3] 0.0-60.1 sec 681 MBytes 95.0 Mbits/sec 0.072 ms 26950/512820 (5.3%) [ 3] 0.0-60.1 sec 1 datagrams received out-of-order
Ping from Chicago IG VM to the NCSU2 CG
60 packets transmitted, 60 received, 0% packet loss, time 59091ms rtt min/avg/max/mdev = 25.283/25.482/25.966/0.147 ms
NCSU2 Multiple Stitched links
Using the RSpec stitch-3linear.rspec requested the following topology:
NYSERNet IG<ION-AL2S->NCSU2<-AL2S->Stanford-IG
The create sliver:
$ stitcher createsliver 3linear ./stitch-3linear.rspec
15:27:46 INFO : Configured logging from file /home/lnevers/gcf-2.7/src/gcf/stitcher_logging.conf
15:27:46 INFO : Reading slice 3linear credential...
15:27:46 INFO : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+3linear expires on 2015-01-12 20:21:43 UTC
15:27:46 INFO : Using SCS at http://nutshell.maxgigapop.net:8081/geni/xmlrpc
15:27:46 INFO : Calling SCS...
15:27:48 INFO : Multi-AM reservation will include resources from these aggregates:
15:27:48 INFO : <Aggregate ncsu2-eg>
15:27:48 INFO : <Aggregate gpo-ig>
15:27:48 INFO : <Aggregate chicago-ig>
15:27:48 INFO : <Aggregate oess>
15:27:48 INFO : <Aggregate ion>
15:27:48 INFO : Stitcher doing createsliver at <Aggregate ncsu2-eg>...
15:27:51 ERROR : {'output': 'Invalid slice urn urn:publicid:IDN+ch.geni.net:ln-test+slice+3linear', 'code': {'geni_code': 2}}
15:27:51 INFO : Got an error reserving resources in 3linear at <Aggregate ncsu2-eg>
15:27:52 WARNING : Failed to delete sliver urn:publicid:IDN+ch.geni.net:ln-test+slice+3linear at ncsu2-eg (got result 'False'). (no reason given)
15:27:52 WARNING : Stitching failed but will retry: Circuit reservation failed at <Aggregate ncsu2-eg> (AMAPIError: Error from Aggregate: code 2: Invalid slice urn urn:publicid:IDN+ch.geni.net:ln-test+slice+3linear.). Try again from the SCS
