wiki:GeniNetworkStitchingConfirmationTestStatus/UTC

Version 7 (modified by lnevers@bbn.com, 9 years ago) (diff)

--

University of Tennessee Stitching Confirmation Tests

This pages capture the detailed test logs for each test defined in the New Site Stitching Confirmation Tests page. For the sites status see the New Site Stitching Confirmation Tests Status page.

IG-ST-1 New Site to GPO IG topology

The UTC site advertises the following stitching details:

<stitching xmlns="http://hpn.east.isi.edu/rspec/ext/stitch/0.1/" lastUpdateTime="2015-02-11T16:47:07Z" >
<aggregate id="urn:publicid:IDN+instageni.utc.edu+authority+cm" url="https://www.instageni.utc.edu:12369/protogeni/xmlrpc/am">
<aggregatetype>protogeni</aggregatetype>
<stitchingmode>chainANDTree</stitchingmode>
<scheduledservices>false</scheduledservices>
<negotiatedservices>false</negotiatedservices>
<lifetime id="life">
    <start type="time">2015-02-11T16:47:07Z</start>
    <end type="time">2015-02-11T16:47:07Z</end>
</lifetime>
<node id="urn:publicid:IDN+instageni.utc.edu+node+procurve2">
  <port id="urn:publicid:IDN+instageni.utc.edu+stitchport+procurve2:1.21.al2s">
    <capacity>1000000</capacity>
    <maximumReservableCapacity>1000000</maximumReservableCapacity>
    <minimumReservableCapacity>1000</minimumReservableCapacity>
    <granularity>1</granularity>
    <link id="urn:publicid:IDN+instageni.utc.edu+interface+procurve2:1.21.al2s">
      <remoteLinkId>urn:publicid:IDN+al2s.internet2.edu+interface+sdn-sw.atla.net.internet2.edu:eth5/2:utc-ig</remoteLinkId>
      <trafficEngineeringMetric>10</trafficEngineeringMetric>
      <capacity>1000000</capacity>
      <maximumReservableCapacity>1000000</maximumReservableCapacity>
      <minimumReservableCapacity>1000</minimumReservableCapacity>
      <granularity>1</granularity>
      <switchingCapabilityDescriptor>
        <switchingcapType>l2sc</switchingcapType>
        <encodingType>ethernet</encodingType>
        <switchingCapabilitySpecificInfo>
         <switchingCapabilitySpecificInfo_L2sc>
          <interfaceMTU>1500</interfaceMTU>
          <vlanRangeAvailability>3500-3514</vlanRangeAvailability>
          <vlanTranslation>false</vlanTranslation>
         </switchingCapabilitySpecificInfo_L2sc>
        </switchingCapabilitySpecificInfo>
      </switchingCapabilityDescriptor>
    </link>
  </port>
</node>
</aggregate>
</stitching>

Experimenter may not need any of this data, but it is helpful to reference when trying to determine how many VLANs are delegated for stitching at the site or how much bandwidth can be requested.

Create a slice and then create the stitched slivers with the RSpec IG-ST-1-al2s.rspec:

$ stitcher.py createsliver IG-ST-1 IG-ST-1-al2s.rspec -o
11:48:29 INFO    : Configured logging from file /home/lnevers/gcf-2.8/src/gcf/stitcher_logging.conf
11:48:30 INFO    : Reading slice IG-ST-1 credential...
11:48:30 INFO    : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+IG-ST-1 expires on 2015-02-18 16:48:13 UTC
11:48:30 INFO    : Calling SCS...
11:48:32 INFO    : Multi-AM reservation will include resources from these aggregates:
11:48:32 INFO    : 	<Aggregate utc-ig>
11:48:32 INFO    : 	<Aggregate gpo-ig>
11:48:32 INFO    : 	<Aggregate al2s>
11:48:32 INFO    : 	<Aggregate ion>
11:48:32 INFO    : Stitcher doing createsliver at <Aggregate utc-ig>...
11:48:50 INFO    : ... Allocation at <Aggregate utc-ig> complete.
11:48:50 INFO    : Stitcher doing createsliver at <Aggregate gpo-ig>...
11:49:05 INFO    : ... Allocation at <Aggregate gpo-ig> complete.
11:49:05 INFO    : Stitcher doing createsliver at <Aggregate al2s>...
11:49:09 INFO    : ... Allocation at <Aggregate al2s> complete.
11:49:09 INFO    : Stitcher doing createsliver at <Aggregate ion>...
11:49:53 INFO    : DCN AM <Aggregate ion>: must wait for status ready....
11:49:53 INFO    : Pausing 30 seconds to let circuit become ready...
11:50:31 INFO    : Pausing 30 seconds to let circuit become ready...
11:51:09 WARNING : sliverstatus: 153231 is (still) failed at <Aggregate ion>. Delete and retry.
11:51:09 INFO    : Doing deletesliver at <Aggregate ion>...
11:51:14 WARNING : <Aggregate ion> says requested VLAN was unavailable at <Hop u'urn:publicid:IDN+ion.internet2.edu+interface+rtr.newy:et-5/0/0:gpo-ig' on path u'link'>
11:51:14 INFO    : Deleting some reservations to retry, avoiding failed VLAN...
11:51:14 INFO    : Doing deletesliver at <Aggregate gpo-ig>...
11:52:42 ERROR   :  {'output': 'start/restart in progress; try again later', 'code': {'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+85ad8828d76050e327e1bd90d32c7a79', 'am_type': 'protogeni', 'geni_code': 14, 'am_code': 14, 'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=85ad8828d76050e327e1bd90d32c7a79'}, 'value': 0}
11:52:52 INFO    :  ... aggregate was busy, will retry ...
11:54:20 ERROR   :  {'output': 'start/restart in progress; try again later', 'code': {'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+26af6c895a56890fa91b35c8b9002ef8', 'am_type': 'protogeni', 'geni_code': 14, 'am_code': 14, 'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=26af6c895a56890fa91b35c8b9002ef8'}, 'value': 0}
11:54:30 INFO    :  ... aggregate was busy, will retry ...
11:55:58 INFO    :  ... done.
11:55:58 INFO    : Will put <Aggregate ion> back in the pool to allocate. Got: Retrying reservations at earlier AMs to avoid unavailable VLAN tag at <Aggregate ion>....
11:55:58 INFO    : Pausing for 30 seconds for Aggregates to free up resources...


11:56:29 INFO    : Checking that prior reservation at <Aggregate gpo-ig> has been cleared up....
11:56:30 ERROR   :  {'output': 'No slice or aggregate here', 'code': {'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+b915d01503bf764d3451c0dff22b9f75', 'am_type': 'protogeni', 'geni_code': 12, 'am_code': 12, 'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=b915d01503bf764d3451c0dff22b9f75'}, 'value': 0}
11:56:30 INFO    : ... it is, so can try a new reservation.
11:56:30 INFO    : Stitcher doing createsliver at <Aggregate gpo-ig>...
11:57:10 INFO    : ... Allocation at <Aggregate gpo-ig> complete.
11:57:10 INFO    : Stitcher doing createsliver at <Aggregate ion>...
11:57:55 INFO    : DCN AM <Aggregate ion>: must wait for status ready....
11:57:55 INFO    : Pausing 30 seconds to let circuit become ready...
11:58:32 INFO    : Pausing 30 seconds to let circuit become ready...
11:59:10 INFO    : DCN circuit 153251 is ready at <Aggregate ion>
11:59:17 INFO    : ... Allocation at <Aggregate ion> complete.
11:59:17 INFO    : All aggregates are complete.
11:59:17 INFO    : Your resources expire at 2015-02-17T16:38:32 (UTC). 
11:59:17 INFO    : Saved combined reservation RSpec at 4 AMs to file '/home/lnevers/gcf-test/stitch-test/UTC/IG-ST-1-manifest-rspec-multiam-combined.xml'
Success: Reserved resources in slice IG-ST-1 at 4 Aggregates (including 2 intermediate aggregate(s) not in the original request), creating 1 stitched link(s).

Determined login information at each UTC and GPO aggregate:

$ readyToLogin.py IG-ST-1 --useSliceAggregates
  ....
ig-utc's geni_status is: ready 
User lnevers logs in to ig-utc using:
	ssh -p 30522  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc2.instageni.utc.edu
...
ig-gpo's geni_status is: changing 
User lnevers logs in to ig-gpo using:
	ssh -p 31802  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc5.instageni.gpolab.bbn.com

Measurements

Iperf InstaGENI GPO VM to InstaGENI UTC VM (TCP) - TCP window size: 23.5 KByte (default)

Collected: 2015-02-XX

One Client_

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.1 sec   142 MBytes  19.8 Mbits/sec

Five Clients

[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-60.3 sec  26.1 MBytes  3.63 Mbits/sec
[  5]  0.0-60.4 sec  23.2 MBytes  3.23 Mbits/sec
[  3]  0.0-60.5 sec  36.2 MBytes  5.03 Mbits/sec
[  7]  0.0-60.6 sec  33.2 MBytes  4.60 Mbits/sec
[  6]  0.0-60.7 sec  25.8 MBytes  3.56 Mbits/sec
[SUM]  0.0-60.7 sec   145 MBytes  20.0 Mbits/sec

Ten Clients

[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-59.6 sec  11.2 MBytes  1.58 Mbits/sec
[  7]  0.0-60.3 sec  17.8 MBytes  2.47 Mbits/sec
[ 10]  0.0-60.3 sec  13.1 MBytes  1.83 Mbits/sec
[ 12]  0.0-60.4 sec  15.9 MBytes  2.21 Mbits/sec
[  4]  0.0-60.7 sec  17.6 MBytes  2.44 Mbits/sec
[  6]  0.0-60.9 sec  13.1 MBytes  1.81 Mbits/sec
[  8]  0.0-61.0 sec  12.8 MBytes  1.75 Mbits/sec
[  3]  0.0-61.1 sec  10.1 MBytes  1.39 Mbits/sec
[ 11]  0.0-61.2 sec  15.8 MBytes  2.16 Mbits/sec
[  9]  0.0-62.0 sec  18.9 MBytes  2.55 Mbits/sec
[SUM]  0.0-62.0 sec   146 MBytes  19.8 Mbits/sec

Iperf InstaGENI GPO VM to InstaGENI UTC VM (UDP) - UDP buffer size: 224 KByte (default)

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec   719 MBytes   100 Mbits/sec
[  3] Sent 512770 datagrams
read failed: Connection refused
[  3] WARNING: did not receive ack of last datagram after 1 tries.

Ping from InstaGENI GPO VM to the InstaGENI UTC VM

60 packets transmitted, 60 received, 0% packet loss, time 59093ms
rtt min/avg/max/mdev = 27.722/27.995/29.328/0.274 ms

Iperf InstaGENI UTC VM to GPO InstaGENI VM (TCP) - TCP window size: 23.5 KByte (default)

Collected: 2015-02-XX

One Client

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.1 sec   142 MBytes  19.8 Mbits/sec

Five Clients

[ ID] Interval       Transfer     Bandwidth
[  7]  0.0-60.3 sec  29.4 MBytes  4.09 Mbits/sec
[  4]  0.0-60.3 sec  34.6 MBytes  4.82 Mbits/sec
[  3]  0.0-60.4 sec  25.5 MBytes  3.54 Mbits/sec
[  5]  0.0-60.8 sec  25.2 MBytes  3.49 Mbits/sec
[  6]  0.0-60.8 sec  29.5 MBytes  4.07 Mbits/sec
[SUM]  0.0-60.8 sec   144 MBytes  19.9 Mbits/sec

Ten Clients

[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-60.1 sec  14.6 MBytes  2.04 Mbits/sec
[  8]  0.0-60.2 sec  13.9 MBytes  1.93 Mbits/sec
[  6]  0.0-60.3 sec  14.9 MBytes  2.07 Mbits/sec
[ 11]  0.0-60.3 sec  13.9 MBytes  1.93 Mbits/sec
[  3]  0.0-60.6 sec  14.1 MBytes  1.96 Mbits/sec
[  7]  0.0-60.7 sec  13.4 MBytes  1.85 Mbits/sec
[  9]  0.0-60.9 sec  13.9 MBytes  1.91 Mbits/sec
[ 10]  0.0-60.9 sec  12.1 MBytes  1.67 Mbits/sec
[  4]  0.0-61.3 sec  12.1 MBytes  1.66 Mbits/sec
[ 12]  0.0-62.6 sec  22.4 MBytes  3.00 Mbits/sec
[SUM]  0.0-62.6 sec   145 MBytes  19.5 Mbits/sec

Iperf InstaGENI UTC VM to GPO InstaGENI VM (UDP) - UDP buffer size: 224 KByte (default)

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec   719 MBytes   101 Mbits/sec
[  3] Sent 512804 datagrams
[  3] Server Report:
[  3]  0.0-60.1 sec   139 MBytes  19.4 Mbits/sec   0.111 ms 413423/512799 (81%)
[  3]  0.0-60.1 sec  1 datagrams received out-of-order

Ping from InstaGENI UTC VM to GPO InstaGENI VM

60 packets transmitted, 60 received, 0% packet loss, time 59099ms
rtt min/avg/max/mdev = 27.635/27.995/29.817/0.366 ms

IG-ST-2 New Site to GPO IG Loop topology

Create a slice and then create the stitched slivers with the RSpec IG-ST-2.rspec:

$ stitcher createsliver IG-ST-2 ./IG-ST-2.rspec 
12:05:42 INFO    : Configured logging from file /home/lnevers/gcf-2.8/src/gcf/stitcher_logging.conf
12:05:43 INFO    : Reading slice IG-ST-2 credential...
12:05:43 INFO    : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+IG-ST-2 expires on 2015-02-18 16:50:26 UTC
12:05:43 INFO    : Calling SCS...
12:05:46 INFO    : Multi-AM reservation will include resources from these aggregates:
12:05:46 INFO    : 	<Aggregate utc-ig>
12:05:46 INFO    : 	<Aggregate gpo-ig>
12:05:46 INFO    : 	<Aggregate al2s>
12:05:46 INFO    : 	<Aggregate ion>
12:05:46 INFO    : Stitcher doing createsliver at <Aggregate utc-ig>...
12:06:03 INFO    : ... Allocation at <Aggregate utc-ig> complete.
12:06:03 INFO    : Stitcher doing createsliver at <Aggregate gpo-ig>...
12:06:19 INFO    : ... Allocation at <Aggregate gpo-ig> complete.
12:06:19 INFO    : Stitcher doing createsliver at <Aggregate al2s>...
12:06:24 INFO    : ... Allocation at <Aggregate al2s> complete.
12:06:24 INFO    : Stitcher doing createsliver at <Aggregate ion>...
12:07:09 INFO    : DCN AM <Aggregate ion>: must wait for status ready....
12:07:09 INFO    : Pausing 30 seconds to let circuit become ready...
12:07:46 INFO    : Pausing 30 seconds to let circuit become ready...
12:08:23 WARNING : sliverstatus: 153271 is (still) failed at <Aggregate ion>. Delete and retry.
12:08:23 WARNING : sliverstatus: 153261 is (still) failed at <Aggregate ion>. Delete and retry.
12:08:23 INFO    : Doing deletesliver at <Aggregate ion>...
12:08:29 WARNING : <Aggregate ion> says requested VLAN was unavailable at <Hop u'urn:publicid:IDN+ion.internet2.edu+interface+rtr.newy:et-5/0/0:gpo-ig' on path u'link'>
12:08:29 INFO    : Deleting some reservations to retry, avoiding failed VLAN...
12:08:29 INFO    : Doing deletesliver at <Aggregate gpo-ig>...
12:08:31 INFO    : Will put <Aggregate ion> back in the pool to allocate. Got: Retrying reservations at earlier AMs to avoid unavailable VLAN tag at <Aggregate ion>....
12:08:31 INFO    : Pausing for 30 seconds for Aggregates to free up resources...


12:09:01 INFO    : Checking that prior reservation at <Aggregate gpo-ig> has been cleared up....
12:09:23 ERROR   :  {'output': 'No slice or aggregate here', 'code': {'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+b5289d648dd066a3b9b547daf0686d84', 'am_type': 'protogeni', 'geni_code': 12, 'am_code': 12, 'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=b5289d648dd066a3b9b547daf0686d84'}, 'value': 0}
12:09:23 INFO    : ... it is, so can try a new reservation.
12:09:23 INFO    : Stitcher doing createsliver at <Aggregate gpo-ig>...
12:09:39 INFO    : ... Allocation at <Aggregate gpo-ig> complete.
12:09:39 INFO    : Stitcher doing createsliver at <Aggregate ion>...
12:10:24 INFO    : DCN AM <Aggregate ion>: must wait for status ready....
12:10:24 INFO    : Pausing 30 seconds to let circuit become ready...
12:11:02 INFO    : Pausing 30 seconds to let circuit become ready...
12:11:40 WARNING : sliverstatus: 153291 is (still) failed at <Aggregate ion>. Delete and retry.
12:11:40 WARNING :   Status had error message: VLAN cancelled by rollback from contingent failure
12:11:40 WARNING : sliverstatus: 153281 is (still) failed at <Aggregate ion>. Delete and retry.
12:11:40 INFO    : Doing deletesliver at <Aggregate ion>...
12:11:45 WARNING : <Aggregate ion> says requested VLAN was unavailable at <Hop u'urn:publicid:IDN+ion.internet2.edu+interface+rtr.newy:et-5/0/0:gpo-ig' on path u'link'>
12:11:45 INFO    : Deleting some reservations to retry, avoiding failed VLAN...
12:11:45 INFO    : Doing deletesliver at <Aggregate gpo-ig>...
12:11:47 INFO    : Will put <Aggregate ion> back in the pool to allocate. Got: Retrying reservations at earlier AMs to avoid unavailable VLAN tag at <Aggregate ion>....
12:11:47 INFO    : Pausing for 30 seconds for Aggregates to free up resources...


12:12:17 INFO    : Checking that prior reservation at <Aggregate gpo-ig> has been cleared up....
12:12:39 ERROR   :  {'output': 'No slice or aggregate here', 'code': {'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+5d447ffe50ae1110480186454c5b751c', 'am_type': 'protogeni', 'geni_code': 12, 'am_code': 12, 'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=5d447ffe50ae1110480186454c5b751c'}, 'value': 0}
12:12:39 INFO    : ... it is, so can try a new reservation.
12:12:39 INFO    : Stitcher doing createsliver at <Aggregate gpo-ig>...
12:12:55 INFO    : ... Allocation at <Aggregate gpo-ig> complete.
12:12:55 INFO    : Stitcher doing createsliver at <Aggregate ion>...
12:13:40 INFO    : DCN AM <Aggregate ion>: must wait for status ready....
12:13:40 INFO    : Pausing 30 seconds to let circuit become ready...
12:14:18 INFO    : Pausing 30 seconds to let circuit become ready...
12:14:55 INFO    : DCN circuit 153301 is ready at <Aggregate ion>
12:14:55 INFO    : DCN circuit 153311 is ready at <Aggregate ion>
12:15:02 INFO    : ... Allocation at <Aggregate ion> complete.
12:15:02 INFO    : All aggregates are complete.
12:15:02 INFO    : Your resources expire at 2015-02-17T16:55:46 (UTC). 
12:15:02 INFO    : Saved combined reservation RSpec at 4 AMs to file '/home/lnevers/gcf-test/stitch-test/UTC/IG-ST-2-manifest-rspec-multiam-combined.xml'
Success: Reserved resources in slice IG-ST-2 at 4 Aggregates (including 2 intermediate aggregate(s) not in the original request), creating 2 stitched link(s)

Determined login information at each UTC and GPO aggregate:

$ readyToLogin.py IG-ST-2 --useSliceAggregates
  ....
ig-utc's geni_status is: ready 
User lnevers logs in to ig-utc using:
	ssh -p 30778  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc5.instageni.utc.edu
  ....
ig-gpo's geni_status is: ready 
User lnevers logs in to ig-gpo using:
	ssh -p 32314  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc1.instageni.gpolab.bbn.com

Login to GPO host and ping the remote on each of the two interfaces. Below is the ping output for the GPO site:

$ remote-execute.py IG-ST-2 -a gpo-ig -m "ping 10.10.4.2 -c 60 -q ;ping 192.168.4.2 -c 60 -q"
1. Find login Info for hosts in slice IG-ST-2
Key file [/home/lnevers/.ssh/arendia-id_rsa] does NOT exist.
Key file [/home/lnevers/.ssh/sendaria-id_rsa] does NOT exist.
2. Send command 'ping 10.10.4.2 -c 60 -q ;ping 192.168.4.2 -c 60 -q' to ig-gpo

PING 10.10.4.2 (10.10.4.2) 56(84) bytes of data.

--- 10.10.4.2 ping statistics ---
60 packets transmitted, 60 received, 0% packet loss, time 59079ms
rtt min/avg/max/mdev = 27.157/27.814/55.272/3.601 ms
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.

--- 192.168.4.2 ping statistics ---
60 packets transmitted, 60 received, 0% packet loss, time 59079ms
rtt min/avg/max/mdev = 27.136/27.768/54.951/3.550 ms
... Done with command 'ping 10.10.4.2 -c 60 -q ;ping 192.168.4.2 -c 60 -q' at ig-gpo

IG-ST-3 IG-ST-3 New Site 3 node linear topology

Create a slice and then create the stitched slivers with the RSpec IG-ST-3.rspec:

$ stitcher createsliver IG-ST-3 ./IG-ST-3.rspec 
12:12:39 INFO    : Configured logging from file /home/lnevers/gcf-2.8/src/gcf/stitcher_logging.conf
12:12:40 INFO    : Reading slice IG-ST-3 credential...
12:12:40 INFO    : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+IG-ST-3 expires on 2015-02-18 17:10:47 UTC
12:12:40 INFO    : Calling SCS...
12:12:47 INFO    : Multi-AM reservation will include resources from these aggregates:
12:12:47 INFO    : 	<Aggregate ukypks2-ig>
12:12:47 INFO    : 	<Aggregate gpo-ig>
12:12:47 INFO    : 	<Aggregate al2s>
12:12:47 INFO    : 	<Aggregate utc-ig>
12:12:47 INFO    : 	<Aggregate ion>
12:12:47 INFO    : Stitcher doing createsliver at <Aggregate ukypks2-ig>...
12:13:03 INFO    : ... Allocation at <Aggregate ukypks2-ig> complete.
12:13:03 INFO    : Stitcher doing createsliver at <Aggregate gpo-ig>...
12:13:23 INFO    : ... Allocation at <Aggregate gpo-ig> complete.
12:13:23 INFO    : Stitcher doing createsliver at <Aggregate utc-ig>...
12:13:45 INFO    : ... Allocation at <Aggregate utc-ig> complete.
12:13:45 INFO    : Stitcher doing createsliver at <Aggregate al2s>...
12:13:50 INFO    : ... Allocation at <Aggregate al2s> complete.
12:13:50 INFO    : Stitcher doing createsliver at <Aggregate ion>...
12:14:36 INFO    : DCN AM <Aggregate ion>: must wait for status ready....
12:14:36 INFO    : Pausing 30 seconds to let circuit become ready...
12:15:13 INFO    : Pausing 30 seconds to let circuit become ready...
12:15:51 WARNING : sliverstatus: 153331 is (still) failed at <Aggregate ion>. Delete and retry.
12:15:51 WARNING : sliverstatus: 153321 is (still) failed at <Aggregate ion>. Delete and retry.
12:15:51 WARNING :   Status had error message: VLAN cancelled by rollback from contingent failure
12:15:51 INFO    : Doing deletesliver at <Aggregate ion>...
12:15:56 WARNING : <Aggregate ion> says requested VLAN was unavailable at <Hop u'urn:publicid:IDN+ion.internet2.edu+interface+rtr.newy:et-5/0/0:gpo-ig' on path u'link'>
12:15:56 INFO    : Deleting some reservations to retry, avoiding failed VLAN...
12:15:56 INFO    : Doing deletesliver at <Aggregate gpo-ig>...
12:17:28 ERROR   :  {'output': 'start/restart in progress; try again later', 'code': {'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+0bd79116b4fa73b9c08e39ec43460254', 'am_type': 'protogeni', 'geni_code': 14, 'am_code': 14, 'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=0bd79116b4fa73b9c08e39ec43460254'}, 'value': 0}
12:17:38 INFO    :  ... aggregate was busy, will retry ...
12:19:08 ERROR   :  {'output': 'start/restart in progress; try again later', 'code': {'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+beb0e5656130abb4e2f839e5b0ca9573', 'am_type': 'protogeni', 'geni_code': 14, 'am_code': 14, 'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=beb0e5656130abb4e2f839e5b0ca9573'}, 'value': 0}
12:19:18 INFO    :  ... aggregate was busy, will retry ...
12:20:47 ERROR   :  {'output': 'start/restart in progress; try again later', 'code': {'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+f950395b872abdba4a8f55b94ccccb6b', 'am_type': 'protogeni', 'geni_code': 14, 'am_code': 14, 'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=f950395b872abdba4a8f55b94ccccb6b'}, 'value': 0}
12:20:57 INFO    :  ... aggregate was busy, will retry ...
12:22:31 INFO    :  ... done.
12:22:31 INFO    : Will put <Aggregate ion> back in the pool to allocate. Got: Retrying reservations at earlier AMs to avoid unavailable VLAN tag at <Aggregate ion>....
12:22:31 INFO    : Pausing for 30 seconds for Aggregates to free up resources...


12:23:01 INFO    : Checking that prior reservation at <Aggregate gpo-ig> has been cleared up....
12:23:03 ERROR   :  {'output': 'No slice or aggregate here', 'code': {'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+74135c5ebf3faaf27657363b356da12a', 'am_type': 'protogeni', 'geni_code': 12, 'am_code': 12, 'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=74135c5ebf3faaf27657363b356da12a'}, 'value': 0}
12:23:03 INFO    : ... it is, so can try a new reservation.
12:23:03 INFO    : Stitcher doing createsliver at <Aggregate gpo-ig>...
12:23:44 INFO    : ... Allocation at <Aggregate gpo-ig> complete.
12:23:44 INFO    : Stitcher doing createsliver at <Aggregate ion>...
12:24:29 INFO    : DCN AM <Aggregate ion>: must wait for status ready....
12:24:29 INFO    : Pausing 30 seconds to let circuit become ready...
12:25:06 INFO    : Pausing 30 seconds to let circuit become ready...
12:25:44 INFO    : DCN circuit 153361 is ready at <Aggregate ion>
12:25:44 INFO    : DCN circuit 153351 is ready at <Aggregate ion>
12:25:51 INFO    : ... Allocation at <Aggregate ion> complete.
12:25:51 INFO    : All aggregates are complete.
12:25:51 INFO    : Your resources expire at 2015-02-17T17:02:47 (UTC). 
12:25:51 INFO    : Saved combined reservation RSpec at 5 AMs to file '/home/lnevers/gcf-test/stitch-test/UTC/IG-ST-3-manifest-rspec-multiam-combined.xml'
Success: Reserved resources in slice IG-ST-3 at 5 Aggregates (including 2 intermediate aggregate(s) not in the original request), creating 2 stitched link(s).

Determined login information the UTC host:

$ readyToLogin.py IG-ST-3 -a utc-ig
  ....

ig-utc's geni_status is: ready 
User lnevers logs in to ig-utc using:
	ssh -p 31034  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc1.instageni.utc.edu

Login to the UTC host and ping each remote:

#ping GPO IG 
lnevers@ig-utc:~$ ping 192.168.2.1 -c 60 -q
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.


--- 192.168.2.1 ping statistics ---
60 packets transmitted, 60 received, 0% packet loss, time 59093ms
rtt min/avg/max/mdev = 27.715/28.427/55.614/3.548 ms

# UKYPKS2 IG
lnevers@ig-utc:~$ ping 192.168.4.1 -c 60 -q
PING 192.168.4.1 (192.168.4.1) 56(84) bytes of data.

--- 192.168.4.1 ping statistics ---
60 packets transmitted, 60 received, 0% packet loss, time 59112ms
rtt min/avg/max/mdev = 17.657/18.186/39.184/2.736 ms

IG-ST-4 New Site to GPO EG interoperability

Create a slice and then create the stitched slivers with the RSpec IG-ST-4.rspec, which specifies a 100 Mbps link capacity. This was done to get around the mismatch in link capacity units that exists between IG and EG.

$ stitcher.py createsliver IG-ST-4a IG-ST-4.rspec -o 
12:41:39 INFO    : Configured logging from file /home/lnevers/gcf-2.8/src/gcf/stitcher_logging.conf
12:41:40 INFO    : Reading slice IG-ST-4a credential...
12:41:40 INFO    : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+IG-ST-4a expires on 2015-02-18 17:41:39 UTC
12:41:40 INFO    : Calling SCS...
12:41:42 INFO    : Multi-AM reservation will include resources from these aggregates:
12:41:42 INFO    : 	<Aggregate utc-ig>
12:41:42 INFO    : 	<Aggregate al2s>
12:41:42 INFO    : 	<Aggregate gpo-eg>
12:41:42 INFO    : 	<Aggregate ion>
12:41:42 INFO    : Stitcher doing createsliver at <Aggregate utc-ig>...
12:42:02 INFO    : ... Allocation at <Aggregate utc-ig> complete.
12:42:02 INFO    : Stitcher doing createsliver at <Aggregate gpo-eg>...
12:42:14 INFO    : ... Allocation at <Aggregate gpo-eg> complete.
12:42:14 INFO    : Stitcher doing createsliver at <Aggregate al2s>...
12:42:17 INFO    : ... Allocation at <Aggregate al2s> complete.
12:42:17 INFO    : Stitcher doing createsliver at <Aggregate ion>...
12:43:02 INFO    : DCN AM <Aggregate ion>: must wait for status ready....
12:43:02 INFO    : Pausing 30 seconds to let circuit become ready...
12:43:41 INFO    : Pausing 30 seconds to let circuit become ready...
12:44:18 INFO    : DCN circuit 153371 is ready at <Aggregate ion>
12:44:25 INFO    : ... Allocation at <Aggregate ion> complete.
12:44:25 INFO    : All aggregates are complete.
12:44:25 INFO    : Your resources expire at 2015-02-17T17:31:42 (UTC). 
12:44:25 INFO    : Saved combined reservation RSpec at 4 AMs to file '/home/lnevers/gcf-test/stitch-test/UTC/IG-ST-4a-manifest-rspec-multiam-combined.xml'
Success: Reserved resources in slice IG-ST-4a at 4 Aggregates (including 2 intermediate aggregate(s) not in the original request), creating 1 stitched link(s).

Determined login information at each UTC and GPO aggregate:

$ readyToLogin.py IG-ST-4 --useSliceAggregates
  ....
eg-gpo's geni_status is: ready 
User lnevers logs in to eg-gpo using:
	ssh -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@192.1.242.9
  ....
ig-utc's geni_status is: ready 
User lnevers logs in to ig-utc using:
	ssh -p 30522  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc2.instageni.utc.edu

Measurements

Iperf ExoGENI GPO VM to InstaGENI UTC VM (TCP) - TCP window size: 85.0 KByte (default)

Collected: 2015-02-11

One Client

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.1 sec   139 MBytes  19.4 Mbits/sec

Five Clients

[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-60.0 sec  28.9 MBytes  4.03 Mbits/sec
[  7]  0.0-60.2 sec  29.0 MBytes  4.04 Mbits/sec
[  5]  0.0-60.4 sec  28.4 MBytes  3.94 Mbits/sec
[  4]  0.0-60.8 sec  29.4 MBytes  4.05 Mbits/sec
[  3]  0.0-61.0 sec  28.5 MBytes  3.92 Mbits/sec
[SUM]  0.0-61.0 sec   144 MBytes  19.8 Mbits/sec

Ten Clients

[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-60.0 sec  15.6 MBytes  2.18 Mbits/sec
[  6]  0.0-60.1 sec  15.9 MBytes  2.22 Mbits/sec
[ 10]  0.0-60.3 sec  14.5 MBytes  2.02 Mbits/sec
[ 11]  0.0-60.3 sec  13.5 MBytes  1.88 Mbits/sec
[  7]  0.0-60.5 sec  14.8 MBytes  2.04 Mbits/sec
[  8]  0.0-60.6 sec  13.8 MBytes  1.90 Mbits/sec
[  9]  0.0-61.1 sec  14.1 MBytes  1.94 Mbits/sec
[  5]  0.0-61.3 sec  15.0 MBytes  2.05 Mbits/sec
[ 12]  0.0-61.7 sec  14.9 MBytes  2.02 Mbits/sec
[  3]  0.0-62.5 sec  16.2 MBytes  2.18 Mbits/sec
[SUM]  0.0-62.5 sec   148 MBytes  19.9 Mbits/sec

Iperf ExoGENI GPO VM to InstaGENI UTC VM (UDP) - UDP buffer size: 208 KByte (default)

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec   719 MBytes   101 Mbits/sec
[  3] Sent 512808 datagrams
[  3] Server Report:
[  3]  0.0-60.2 sec   140 MBytes  19.4 Mbits/sec   0.139 ms 413258/512807 (81%)
[  3]  0.0-60.2 sec  1 datagrams received out-of-order

Ping from ExoGENI GPO VM to the InstaGENI UTC VM

60 packets transmitted, 60 received, 0% packet loss, time 59079ms
rtt min/avg/max/mdev = 27.777/28.584/46.229/2.675 ms

Iperf InstaGENI UTC VM to ExoGENI GPO VM (TCP) - TCP window size: 23.5 KByte (default)

Collected: 2015-02-11

One Client_

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.1 sec   143 MBytes  20.0 Mbits/sec

Five Clients

[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-60.0 sec  21.4 MBytes  2.99 Mbits/sec
[  7]  0.0-60.1 sec  42.1 MBytes  5.88 Mbits/sec
[  3]  0.0-60.2 sec  32.8 MBytes  4.56 Mbits/sec
[  4]  0.0-60.2 sec  23.0 MBytes  3.20 Mbits/sec
[  6]  0.0-60.2 sec  25.1 MBytes  3.50 Mbits/sec
[SUM]  0.0-60.2 sec   144 MBytes  20.1 Mbits/sec

Ten Clients

[ ID] Interval       Transfer     Bandwidth
[  9]  0.0-60.2 sec  20.0 MBytes  2.79 Mbits/sec
[  7]  0.0-60.3 sec  13.8 MBytes  1.91 Mbits/sec
[  8]  0.0-60.6 sec  9.38 MBytes  1.30 Mbits/sec
[  6]  0.0-60.7 sec  15.2 MBytes  2.11 Mbits/sec
[  5]  0.0-60.8 sec  11.1 MBytes  1.54 Mbits/sec
[ 11]  0.0-60.9 sec  10.8 MBytes  1.48 Mbits/sec
[ 12]  0.0-61.0 sec  16.0 MBytes  2.20 Mbits/sec
[  3]  0.0-61.2 sec  21.4 MBytes  2.93 Mbits/sec
[ 10]  0.0-61.4 sec  11.2 MBytes  1.54 Mbits/sec
[  4]  0.0-61.6 sec  13.6 MBytes  1.85 Mbits/sec
[SUM]  0.0-61.6 sec   142 MBytes  19.4 Mbits/sec

Iperf InstaGENI UTC VM to ExoGENI GPO VM (UDP) - UDP buffer size: 224 KByte (default)

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec   719 MBytes   101 Mbits/sec
[  3] Sent 512765 datagrams
[  3] Server Report:
[  3]  0.0-60.2 sec   139 MBytes  19.4 Mbits/sec  14.610 ms 413662/512764 (81%)
[  3]  0.0-60.2 sec  1 datagrams received out-of-order

Ping from InstaGENI UTC VM to ExoGENI GPO VM

60 packets transmitted, 60 received, 0% packet loss, time 59082ms
rtt min/avg/max/mdev = 27.813/28.031/28.601/0.254 ms

IG-ST-5 Site Information

Various pages include stitching information for a site. Each of the following were verified for this site:

  1. Verified that stitching VLANs and Device URN information exists in the UTC Aggregate page.

  1. Verified that Delegated GENI Stitching VLANs for site are documented at the Delegated GENI Stitching VLANs.
  1. Stitching Computation Service logs were reviewed while testing stitching to this site, no issues found.
  1. Added site to the list of GENI Network Stitching Sites.

IG-ST-6 New Site OpenFlow topology

Create a slice and then create the stitched slivers using OpenFlow with the RSPec IG-ST-6.rspec:

$ stitcher createsliver IG-ST-6 ./IG-ST-6.rspec 
13:15:24 INFO    : Configured logging from file /home/lnevers/gcf-2.8/src/gcf/stitcher_logging.conf
13:15:25 INFO    : Reading slice IG-ST-6 credential...
13:15:25 INFO    : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+IG-ST-6 expires on 2015-02-14 00:00:00 UTC
13:15:25 INFO    : Calling SCS...
13:15:27 INFO    : Multi-AM reservation will include resources from these aggregates:
13:15:27 INFO    : 	<Aggregate utc-ig>
13:15:27 INFO    : 	<Aggregate gpo-ig>
13:15:27 INFO    : 	<Aggregate al2s>
13:15:27 INFO    : 	<Aggregate ion>
13:15:27 INFO    : Stitcher doing createsliver at <Aggregate utc-ig>...
13:15:43 INFO    : ... Allocation at <Aggregate utc-ig> complete.
13:15:43 INFO    : Stitcher doing createsliver at <Aggregate gpo-ig>...
13:15:59 INFO    : ... Allocation at <Aggregate gpo-ig> complete.
13:15:59 INFO    : Stitcher doing createsliver at <Aggregate al2s>...
13:16:02 INFO    : ... Allocation at <Aggregate al2s> complete.
13:16:02 INFO    : Stitcher doing createsliver at <Aggregate ion>...
13:16:47 INFO    : DCN AM <Aggregate ion>: must wait for status ready....
13:16:47 INFO    : Pausing 30 seconds to let circuit become ready...
13:17:25 WARNING : sliverstatus: 153401 is (still) failed at <Aggregate ion>. Delete and retry.
13:17:25 INFO    : Doing deletesliver at <Aggregate ion>...
13:17:30 WARNING : <Aggregate ion> says requested VLAN was unavailable at <Hop u'urn:publicid:IDN+ion.internet2.edu+interface+rtr.newy:et-5/0/0:gpo-ig' on path u'link'>
13:17:30 INFO    : Deleting some reservations to retry, avoiding failed VLAN...
13:17:30 INFO    : Doing deletesliver at <Aggregate gpo-ig>...
13:18:56 ERROR   :  {'output': 'start/restart in progress; try again later', 'code': {'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+21a926163fe00f8af0dac119a5a8ddf2', 'am_type': 'protogeni', 'geni_code': 14, 'am_code': 14, 'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=21a926163fe00f8af0dac119a5a8ddf2'}, 'value': 0}
13:19:06 INFO    :  ... aggregate was busy, will retry ...
13:20:36 ERROR   :  {'output': 'start/restart in progress; try again later', 'code': {'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+edba31bffa1ceec97db4fd88fc1d8ee0', 'am_type': 'protogeni', 'geni_code': 14, 'am_code': 14, 'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=edba31bffa1ceec97db4fd88fc1d8ee0'}, 'value': 0}
13:20:46 INFO    :  ... aggregate was busy, will retry ...
13:22:13 ERROR   :  {'output': 'start/restart in progress; try again later', 'code': {'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+57f24bde8b5e708252b19f744f8ef19d', 'am_type': 'protogeni', 'geni_code': 14, 'am_code': 14, 'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=57f24bde8b5e708252b19f744f8ef19d'}, 'value': 0}
13:22:23 INFO    :  ... aggregate was busy, will retry ...
13:23:29 INFO    :  ... done.
13:23:29 INFO    : Will put <Aggregate ion> back in the pool to allocate. Got: Retrying reservations at earlier AMs to avoid unavailable VLAN tag at <Aggregate ion>....
13:23:29 INFO    : Pausing for 30 seconds for Aggregates to free up resources...


13:23:59 INFO    : Checking that prior reservation at <Aggregate gpo-ig> has been cleared up....
13:24:00 ERROR   :  {'output': 'No slice or aggregate here', 'code': {'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+c15fcaa28e576d5c77be7edc426334d3', 'am_type': 'protogeni', 'geni_code': 12, 'am_code': 12, 'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=c15fcaa28e576d5c77be7edc426334d3'}, 'value': 0}
13:24:00 INFO    : ... it is, so can try a new reservation.
13:24:00 INFO    : Stitcher doing createsliver at <Aggregate gpo-ig>...
13:24:39 INFO    : ... Allocation at <Aggregate gpo-ig> complete.
13:24:39 INFO    : Stitcher doing createsliver at <Aggregate ion>...
13:25:24 INFO    : DCN AM <Aggregate ion>: must wait for status ready....
13:25:24 INFO    : Pausing 30 seconds to let circuit become ready...
13:26:01 INFO    : DCN circuit 153411 is ready at <Aggregate ion>
13:26:08 INFO    : ... Allocation at <Aggregate ion> complete.
13:26:08 INFO    : All aggregates are complete.
13:26:08 INFO    : Your resources expire at 2015-02-13T18:05:27 (UTC). 
13:26:08 INFO    : Saved combined reservation RSpec at 4 AMs to file '/home/lnevers/gcf-test/stitch-test/UTC/IG-ST-6-manifest-rspec-multiam-combined.xml'
Success: Reserved resources in slice IG-ST-6 at 4 Aggregates (including 2 intermediate aggregate(s) not in the original request), creating 1 stitched link(s).

Determined login information at each UTC and GPO aggregate:

$ readyToLogin.py IG-ST-6 --useSliceAggregates
 ....
ig-utc's geni_status is: ready 
User lnevers logs in to ig-utc using:
	ssh -p 30522  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc3.instageni.utc.edu
 ....
ig-gpo's geni_status is: ready 
User lnevers logs in to ig-gpo using:
	ssh -p 31802  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc4.instageni.gpolab.bbn.com

Measurements

Iperf InstaGENI GPO VM to InstaGENI UTC VM (TCP) - TCP window size: 23.5 KByte (default)

Collected: 2015-01-xx

One Client_

Five Clients

Ten Clients

Iperf InstaGENI GPO VM to InstaGENI UTC VM (UDP) - UDP buffer size: 224 KByte (default)

Ping from InstaGENI GPO VM to InstaGENI UTC VM

Iperf InstaGENI UTC VM to InstaGENI GPO VM (TCP) - TCP window size: 23.5 KByte (default)

Collected: 2015-01-xx

One Client_

Five Clients

Ten Clients

Iperf InstaGENI UTC VM to InstaGENI GPO VM (UDP) - UDP buffer size: 224 KByte (default)

Ping from InstaGENI UTC VM to InstaGENI GPO VM


Email help@geni.net for GENI support or email me with feedback on this page!