[[PageOutline]] = CG-EXP-4: GENI Stitching Support = This test allocated resources in both NCSU2 and NCSU Cisco racks. Note the following Omni nick names are used: {{{ ncsu2-eg=urn:publicid:IDN+exogeni.net:ncsu2vmsite+authority+am,https://ncsu2-hn.exogeni.net:11443/orca/xmlrpc ncsu-eg=urn:publicid:IDN+exogeni.net:ncsuvmsite+authority+am,https://ncsu-hn.exogeni.net:11443/orca/xmlrpc }}} == Test Status == This section captures the status for each step in the acceptance test plan. || '''Stitching topologies''' ||'''State''' || ''' !Tickets/Comments''' || ||NCSU2 to NCSU2 || N/A ||Not planned at this time || || ||NCSU2 to GPO EG || N/A || This scenario is ExoGENI stitching; not available at this time and covered in [http://groups.geni.net/geni/wiki/GENIRacksHome/CiscoGENIRacks/AcceptanceTestStatus/CG-EXP-2 CG-EXP-2] || ||NCSU2 to IG ION site ||[[Color(green,Pass)]]||~~Stitching setup; remoteLinkId definition~~|| ||NCSU2 to IG AL2S site || || || ||NYSERNet IGNCSU2<-AL2S->Stanford-IG|| || || [[BR]] || '''State Legend''' || '''Description''' || || [[Color(green,Pass)]] || Test completed and met all criteria || || [[Color(#98FB98,Pass: most criteria)]] || Test completed and met most criteria. Exceptions documented || || [[Color(red,Fail)]] || Test completed and failed to meet criteria. || || [[Color(yellow,Complete)]] || Test completed but will require re-execution due to expected changes || || [[Color(orange,Blocked)]] || Blocked by ticketed issue(s). || || [[Color(#63B8FF,In Progress)]] || Currently under test. || [[BR]] == Test Details == Testing scenario cover both ExoGENI and GENI stitching. A mix of test topologies are run with the GENI Stitching feature. === NCSU2 to GPO EG === This is ExoGENI Stitching, which is covered in [http://groups.geni.net/geni/wiki/GENIRacksHome/CiscoGENIRacks/AcceptanceTestStatus/CG-EXP-2 CG-EXP-2]. Using the RSpec [cisco-ncsu2-eg-gpo.rspec] tried to create an ExoGENI stitched sliver via the ExoSM: {{{ $ omni createsliver -a eg-sm CG-CT-4a ./cisco-ncsu2-eg-gpo.rspec 14:25:14 INFO : Loading agg_nick_cache file '/home/lnevers/.gcf/agg_nick_cache' 14:25:14 INFO : Loading config file '/home/lnevers/.gcf/omni_config' 14:25:14 INFO : Using control framework portal 14:25:14 INFO : Member Authority is https://ch.geni.net/MA (from config) 14:25:14 INFO : Slice Authority is https://ch.geni.net/SA (from config) 14:25:14 INFO : Substituting AM nickname eg-sm with URL https://geni.renci.org:11443/orca/xmlrpc, URN urn:publicid:IDN+exogeni.net+authority+am 14:25:14 INFO : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+CG-CT-4a expires on 2015-01-16 19:24:40 UTC 14:25:14 INFO : Creating sliver(s) from rspec file ./cisco-ncsu2-eg-gpo.rspec for slice urn:publicid:IDN+ch.geni.net:ln-test+slice+CG-CT-4a 14:25:20 WARNING : Failed CreateSliver for slice CG-CT-4a at eg-sm. Error from Aggregate: code 2: Embedding workflow ERROR: 4:No Edge Domain Exist:http://geni-orca.renci.org/owl/ec743afb-fa45-4031-ab4c-23b0f5df7806#eg-gpo:http://geni-orca.renci.org/owl/ec743afb-fa45-4031-ab4c-23b0f5df7806#cg-ncsu2. Please see https://geni-orca.renci.org/trac/wiki/orca-errors for possible solutions. 14:25:20 INFO : ------------------------------------------------------ 14:25:20 INFO : Completed createsliver: Options as run: aggregate: ['eg-sm'] framework: portal project: ln-test Args: createsliver CG-CT-4a ./cisco-ncsu2-eg-gpo.rspec Result Summary: Failed CreateSliver for slice CG-CT-4a at eg-sm. Error from Aggregate: code 2: Embedding workflow ERROR: 4:No Edge Domain Exist:http://geni-orca.renci.org/owl/ec743afb-fa45-4031-ab4c-23b0f5df7806#eg-gpo:http://geni-orca.renci.org/owl/ec743afb-fa45-4031-ab4c-23b0f5df7806#cg-ncsu2. Please see https://geni-orca.renci.org/trac/wiki/orca-errors for possible solutions. }}} === NCSU2 to IG ION site === Using the RSpec [stitch-eg-ncsu2-ig-gpo.rspec] the following topology was created: GPO IG <-ION-><-AL2S->NCSU2 The stitcher was used to create the sliver: {{{ $ stitcher createsliver CG-CT-4 ./stitch-eg-ncsu2-ig-gpo.rspec 14:16:02 INFO : Configured logging from file /home/lnevers/gcf-2.7/src/gcf/stitcher_logging.conf 14:16:02 INFO : Reading slice CG-CT-4 credential... 14:16:03 INFO : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+CG-CT-4 expires on 2015-01-16 19:15:12 UTC 14:16:03 INFO : Using SCS at http://nutshell.maxgigapop.net:8081/geni/xmlrpc 14:16:03 INFO : Calling SCS... 14:16:04 INFO : Multi-AM reservation will include resources from these aggregates: 14:16:04 INFO : 14:16:04 INFO : 14:16:04 INFO : 14:16:04 INFO : 14:16:04 INFO : Stitcher doing createsliver at ... 14:16:09 INFO : ... Allocation at complete. 14:16:09 INFO : Stitcher doing createsliver at ... 14:16:24 INFO : ... Allocation at complete. 14:16:24 INFO : Stitcher doing createsliver at ... 14:16:33 INFO : ... Allocation at complete. 14:16:33 INFO : Stitcher doing createsliver at ... 14:17:19 INFO : DCN AM : must wait for status ready.... 14:17:19 INFO : Pausing 30 seconds to let circuit become ready... 14:17:56 INFO : DCN circuit 146681 is ready at 14:18:04 INFO : ... Allocation at complete. 14:18:04 INFO : All aggregates are complete. 14:18:04 INFO : Your resources expire at 2015-01-15T19:06:04 (UTC). 14:18:04 INFO : Saved combined reservation RSpec at 4 AMs to file '/home/lnevers/gcf-test/cisco/CG-CT-4-manifest-rspec-multiam-combined.xml' Success: Reserved resources in slice CG-CT-4 at 4 Aggregates (including 2 intermediate aggregate(s) not in the original request), creating 1 stitched link(s). }}} Determine login for NCSU2 host: {{{ $ readyToLogin CG-CT-4 -a ncsu2-eg ... eg-ncsu2's geni_status is: ready User lnevers logs in to eg-ncsu2 using: ssh -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@152.48.13.191 }}} Determine login for GPO IG host: {{{ $ readyToLogin CG-CT-4 -a gpo-ig ... ig-gpo's geni_status is: ready User lnevers logs in to ig-gpo using: ssh -p 33338 -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc5.instageni.gpolab.bbn.com }}} ==== Measurements ==== '''Iperf GPO IG VM to NCSU2 CG (TCP) - TCP window size: 23.5 KByte (default)''' ''Collected: 2015-01-09'' __One Client_ {{{ [ ID] Interval Transfer Bandwidth [ 3] 0.0-60.0 sec 715 MBytes 100 Mbits/sec }}} __Five Clients__ {{{ [ ID] Interval Transfer Bandwidth [ 6] 0.0-60.0 sec 119 MBytes 16.6 Mbits/sec [ 4] 0.0-60.1 sec 124 MBytes 17.4 Mbits/sec [ 5] 0.0-60.2 sec 161 MBytes 22.4 Mbits/sec [ 7] 0.0-60.2 sec 122 MBytes 16.9 Mbits/sec [ 3] 0.0-60.2 sec 172 MBytes 24.0 Mbits/sec [SUM] 0.0-60.2 sec 698 MBytes 97.3 Mbits/sec }}} __Ten Clients__ {{{ [ ID] Interval Transfer Bandwidth [ 6] 0.0-60.0 sec 59.2 MBytes 8.28 Mbits/sec [ 10] 0.0-60.0 sec 87.6 MBytes 12.2 Mbits/sec [ 11] 0.0-60.1 sec 75.9 MBytes 10.6 Mbits/sec [ 3] 0.0-60.1 sec 62.5 MBytes 8.73 Mbits/sec [ 12] 0.0-60.1 sec 66.0 MBytes 9.21 Mbits/sec [ 7] 0.0-60.3 sec 71.1 MBytes 9.89 Mbits/sec [ 5] 0.0-60.4 sec 47.2 MBytes 6.56 Mbits/sec [ 4] 0.0-60.5 sec 60.0 MBytes 8.31 Mbits/sec [ 9] 0.0-60.7 sec 85.8 MBytes 11.9 Mbits/sec [ 8] 0.0-61.0 sec 91.2 MBytes 12.6 Mbits/sec [SUM] 0.0-61.0 sec 707 MBytes 97.2 Mbits/sec }}} '''Iperf GPO IG VM to NCSU2 CG (UDP) - UDP buffer size: 224 KByte (default) ''' {{{ [ ID] Interval Transfer Bandwidth [ 3] 0.0-60.0 sec 719 MBytes 101 Mbits/sec [ 3] Sent 512809 datagrams [ 3] Server Report: [ 3] 0.0-60.0 sec 694 MBytes 97.0 Mbits/sec 0.121 ms 17641/512808 (3.4%) [ 3] 0.0-60.0 sec 1 datagrams received out-of-order }}} '''Ping from GPO IG VM to the NCSU2 CG ''' {{{ 60 packets transmitted, 60 received, 0% packet loss, time 59099ms rtt min/avg/max/mdev = 17.163/17.569/18.046/0.234 ms }}} '''Iperf NCSU2 CG to GPO IG (TCP) - TCP window size: 85.5 KByte (default)''' ''Collected: 2015-01-09'' __One Client_ {{{ [ ID] Interval Transfer Bandwidth [ 3] 0.0-60.1 sec 35.0 MBytes 4.89 Mbits/sec }}} __Five Clients__ {{{ [ ID] Interval Transfer Bandwidth [ 6] 0.0-60.0 sec 37.0 MBytes 5.17 Mbits/sec [ 3] 0.0-60.2 sec 38.0 MBytes 5.30 Mbits/sec [ 7] 0.0-60.3 sec 33.6 MBytes 4.68 Mbits/sec [ 4] 0.0-60.4 sec 36.8 MBytes 5.11 Mbits/sec [ 5] 0.0-60.7 sec 36.9 MBytes 5.10 Mbits/sec [SUM] 0.0-60.7 sec 182 MBytes 25.2 Mbits/sec }}} __Ten Clients__ {{{ [ ID] Interval Transfer Bandwidth [ 3] 0.0-60.2 sec 38.6 MBytes 5.38 Mbits/sec [ 7] 0.0-60.2 sec 37.6 MBytes 5.24 Mbits/sec [ 11] 0.0-60.2 sec 38.2 MBytes 5.33 Mbits/sec [ 8] 0.0-60.2 sec 38.0 MBytes 5.29 Mbits/sec [ 12] 0.0-60.3 sec 37.8 MBytes 5.26 Mbits/sec [ 9] 0.0-60.3 sec 37.6 MBytes 5.24 Mbits/sec [ 4] 0.0-60.3 sec 38.1 MBytes 5.31 Mbits/sec [ 10] 0.0-60.3 sec 40.0 MBytes 5.56 Mbits/sec [ 5] 0.0-60.4 sec 37.6 MBytes 5.23 Mbits/sec [ 6] 0.0-60.6 sec 37.8 MBytes 5.23 Mbits/sec [SUM] 0.0-60.6 sec 381 MBytes 52.8 Mbits/sec }}} Note: If a larger TCP window size was specified, was able to get much better throughput. '''Iperf NCSU2 CG to GPO IG (UDP) - UDP buffer size: 208 KByte (default) ''' {{{ [ ID] Interval Transfer Bandwidth [ 3] 0.0-60.0 sec 719 MBytes 101 Mbits/sec [ 3] Sent 512821 datagrams [ 3] Server Report: [ 3] 0.0-60.1 sec 685 MBytes 95.6 Mbits/sec 0.067 ms 24133/512820 (4.7%) [ 3] 0.0-60.1 sec 9 datagrams received out-of-order }}} '''Ping from GPO IG VM to the NCSU2 CG ''' {{{ 60 packets transmitted, 60 received, 0% packet loss, time 59097ms rtt min/avg/max/mdev = 17.201/17.547/18.112/0.214 ms }}} === NCSU2 to IG AL2S site === Using the RSpec [stitch-eg-ncsu2-ig-chic.rspec] created a stitched sliver to a remote AL2S site: {{{ $ stitcher createsliver CG-CT-4x ./stitch-eg-ncsu2-ig-chic.rspec 15:14:21 INFO : Configured logging from file /home/lnevers/gcf-2.7/src/gcf/stitcher_logging.conf 15:14:21 INFO : Reading slice CG-CT-4z credential... 15:14:21 INFO : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+CG-CT-4z expires on 2015-01-16 20:14:20 UTC 15:14:21 INFO : Using SCS at http://nutshell.maxgigapop.net:8081/geni/xmlrpc 15:14:21 INFO : Calling SCS... 15:14:22 INFO : Multi-AM reservation will include resources from these aggregates: 15:14:22 INFO : 15:14:22 INFO : 15:14:22 INFO : 15:14:22 INFO : Stitcher doing createsliver at ... 15:14:27 INFO : ... Allocation at complete. 15:14:27 INFO : Stitcher doing createsliver at ... 15:14:46 INFO : ... Allocation at complete. 15:14:46 INFO : Stitcher doing createsliver at ... 15:14:49 ERROR : {'output': 'Exception: requested VLAN unavailable: sdn-sw.star.net.internet2.edu,eth7/1 VLAN=3076', 'code': {'geni_code': 24}} 15:14:49 WARNING : Stitching failed but will retry: Circuit reservation failed at . Try again from the SCS. (Error: Topology too complex for local redo - ask Stitching Service to find a VLAN tag (AMAPIError: Error from Aggregate: code 24: Exception: requested VLAN unavailable: sdn-sw.star.net.internet2.edu,eth7/1 VLAN=3076.)) 15:14:49 INFO : Deleting existing reservations... 15:14:49 INFO : Doing deletesliver at ... 15:14:50 INFO : Deleted reservation at . 15:14:50 INFO : Doing deletesliver at ... 15:16:20 ERROR : {'output': 'start/restart in progress; try again later', 'code': {'protogeni_error_log': 'urn:publicid:IDN+geni.uchicago.edu+log+d1498aed6f901cd5bbe792a8c9dde5b6', 'am_type': 'protogeni', 'geni_code': 14, 'am_code': 14, 'protogeni_error_url': 'https://www.geni.uchicago.edu/spewlogfile.php3?logfile=d1498aed6f901cd5bbe792a8c9dde5b6'}, 'value': 0} 15:18:00 INFO : Deleted reservation at . 15:18:00 INFO : Calling SCS for the 2nd time... 15:18:00 INFO : Pausing for 30 seconds for Aggregates to free up resources... 15:18:30 INFO : Multi-AM reservation will include resources from these aggregates: 15:18:30 INFO : 15:18:30 INFO : 15:18:30 INFO : 15:18:30 INFO : Stitcher doing createsliver at ... 15:18:35 INFO : ... Allocation at complete. 15:18:35 INFO : Stitcher doing createsliver at ... 15:19:14 INFO : ... Allocation at complete. 15:19:14 INFO : Stitcher doing createsliver at ... 15:19:19 INFO : ... Allocation at complete. 15:19:19 INFO : All aggregates are complete. 15:19:19 INFO : Your resources expire at 2015-01-15T20:08:30 (UTC). 15:19:19 INFO : Saved combined reservation RSpec at 3 AMs to file '/home/lnevers/gcf-test/cisco/CG-CT-4z-manifest-rspec-multiam-combined.xml' Success: Reserved resources in slice CG-CT-4z at 3 Aggregates (including 1 intermediate aggregate(s) not in the original request), creating 1 stitched link(s). }}} Determine login: {{{ $ readyToLogin CG-CT-4z -a chicago-ig ... ig-chic's geni_status is: ready User lnevers logs in to ig-chic using: ssh -p 30266 -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc3.geni.uchicago.edu $ readyToLogin CG-CT-4z -a ncsu2-eg ... eg-ncsu2's geni_status is: ready User lnevers logs in to eg-ncsu2 using: ssh -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@152.48.13.191 }}} ==== Measurements ==== '''Iperf Chicago IG VM to NCSU2 CG (TCP) - TCP window size: 23.5 KByte (default)''' ''Collected: 2015-01-09'' __One Client__ {{{ [ ID] Interval Transfer Bandwidth [ 3] 0.0-60.0 sec 680 MBytes 95.1 Mbits/sec }}} __Five Clients__ {{{ [ ID] Interval Transfer Bandwidth [ 4] 0.0-60.0 sec 158 MBytes 22.1 Mbits/sec [ 6] 0.0-60.1 sec 151 MBytes 21.1 Mbits/sec [ 3] 0.0-60.1 sec 145 MBytes 20.2 Mbits/sec [ 7] 0.0-60.4 sec 133 MBytes 18.5 Mbits/sec [ 5] 0.0-60.5 sec 120 MBytes 16.6 Mbits/sec [SUM] 0.0-60.5 sec 707 MBytes 98.0 Mbits/sec }}} __Ten Clients__ {{{ [ ID] Interval Transfer Bandwidth [ 4] 0.0-60.1 sec 71.2 MBytes 9.95 Mbits/sec [ 10] 0.0-60.1 sec 62.9 MBytes 8.78 Mbits/sec [ 12] 0.0-60.1 sec 67.2 MBytes 9.39 Mbits/sec [ 6] 0.0-60.1 sec 91.9 MBytes 12.8 Mbits/sec [ 9] 0.0-60.2 sec 85.1 MBytes 11.9 Mbits/sec [ 8] 0.0-60.2 sec 59.4 MBytes 8.28 Mbits/sec [ 11] 0.0-60.2 sec 67.4 MBytes 9.39 Mbits/sec [ 3] 0.0-60.3 sec 65.6 MBytes 9.13 Mbits/sec [ 7] 0.0-60.3 sec 83.5 MBytes 11.6 Mbits/sec [ 5] 0.0-60.4 sec 65.2 MBytes 9.06 Mbits/sec [SUM] 0.0-60.4 sec 720 MBytes 99.9 Mbits/sec }}} '''Iperf Chicago IG VM to NCSU2 CG (UDP) - UDP buffer size: 224 KByte (default) ''' {{{ [ ID] Interval Transfer Bandwidth [ 3] 0.0-60.0 sec 719 MBytes 101 Mbits/sec [ 3] Sent 512778 datagrams [ 3] Server Report: [ 3] 0.0-60.0 sec 693 MBytes 96.9 Mbits/sec 0.138 ms 18523/512777 (3.6%) [ 3] 0.0-60.0 sec 1 datagrams received out-of-order }}} '''Ping from Chicago IG VM to the NCSU2 CG ''' {{{ 60 packets transmitted, 60 received, 0% packet loss, time 59091ms rtt min/avg/max/mdev = 25.204/25.391/25.798/0.228 ms }}} '''Iperf NCSU2 CG to Chicago IG (TCP) - TCP window size: 85.0 KByte (default)''' ''Collected: 2015-01-09'' __One Client_ {{{ [ ID] Interval Transfer Bandwidth [ 3] 0.0-60.7 sec 21.9 MBytes 3.02 Mbits/sec }}} __Five Clients__ {{{ [ ID] Interval Transfer Bandwidth [ 3] 0.0-60.0 sec 27.1 MBytes 3.79 Mbits/sec [ 4] 0.0-60.5 sec 26.2 MBytes 3.64 Mbits/sec [ 6] 0.0-60.5 sec 28.6 MBytes 3.97 Mbits/sec [ 5] 0.0-60.6 sec 27.0 MBytes 3.74 Mbits/sec [ 7] 0.0-60.6 sec 28.8 MBytes 3.98 Mbits/sec [SUM] 0.0-60.6 sec 138 MBytes 19.1 Mbits/sec }}} __Ten Clients__ {{{ [ ID] Interval Transfer Bandwidth [ 5] 0.0-60.2 sec 27.5 MBytes 3.83 Mbits/sec [ 10] 0.0-60.2 sec 25.1 MBytes 3.50 Mbits/sec [ 3] 0.0-60.2 sec 27.2 MBytes 3.80 Mbits/sec [ 7] 0.0-60.2 sec 28.5 MBytes 3.97 Mbits/sec [ 12] 0.0-60.3 sec 26.2 MBytes 3.65 Mbits/sec [ 6] 0.0-60.4 sec 26.6 MBytes 3.70 Mbits/sec [ 8] 0.0-60.4 sec 25.0 MBytes 3.47 Mbits/sec [ 4] 0.0-60.5 sec 26.6 MBytes 3.69 Mbits/sec [ 9] 0.0-60.5 sec 26.6 MBytes 3.69 Mbits/sec [ 11] 0.0-60.5 sec 25.6 MBytes 3.55 Mbits/sec [SUM] 0.0-60.5 sec 265 MBytes 36.7 Mbits/sec }}} '''Iperf NCSU2 CG to Chicago IG (UDP) - UDP buffer size: 208 KByte (default) ''' {{{ [ ID] Interval Transfer Bandwidth [ 3] 0.0-60.0 sec 719 MBytes 101 Mbits/sec [ 3] Sent 512821 datagrams [ 3] Server Report: [ 3] 0.0-60.1 sec 681 MBytes 95.0 Mbits/sec 0.072 ms 26950/512820 (5.3%) [ 3] 0.0-60.1 sec 1 datagrams received out-of-order }}} '''Ping from Chicago IG VM to the NCSU2 CG ''' {{{ 60 packets transmitted, 60 received, 0% packet loss, time 59091ms rtt min/avg/max/mdev = 25.283/25.482/25.966/0.147 ms }}} === NCSU2 Multiple Stitched links === Using the RSpec [XX] requested the following topology: NYSERNet IGNCSU2<-AL2S->Stanford-IG The create sliver: {{{ $ stitcher createsliver 3linear ./stitch-3linear.rspec 15:27:46 INFO : Configured logging from file /home/lnevers/gcf-2.7/src/gcf/stitcher_logging.conf 15:27:46 INFO : Reading slice 3linear credential... 15:27:46 INFO : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+3linear expires on 2015-01-12 20:21:43 UTC 15:27:46 INFO : Using SCS at http://nutshell.maxgigapop.net:8081/geni/xmlrpc 15:27:46 INFO : Calling SCS... 15:27:48 INFO : Multi-AM reservation will include resources from these aggregates: 15:27:48 INFO : 15:27:48 INFO : 15:27:48 INFO : 15:27:48 INFO : 15:27:48 INFO : 15:27:48 INFO : Stitcher doing createsliver at ... 15:27:51 ERROR : {'output': 'Invalid slice urn urn:publicid:IDN+ch.geni.net:ln-test+slice+3linear', 'code': {'geni_code': 2}} 15:27:51 INFO : Got an error reserving resources in 3linear at 15:27:52 WARNING : Failed to delete sliver urn:publicid:IDN+ch.geni.net:ln-test+slice+3linear at ncsu2-eg (got result 'False'). (no reason given) 15:27:52 WARNING : Stitching failed but will retry: Circuit reservation failed at (AMAPIError: Error from Aggregate: code 2: Invalid slice urn urn:publicid:IDN+ch.geni.net:ln-test+slice+3linear.). Try again from the SCS }}}