Changes between Version 27 and Version 28 of GENIRacksHome/USIgniteInstageniRacks/ConfirmationTestStatus/UL-IGN


Ignore:
Timestamp:
06/19/17 10:14:38 (7 years ago)
Author:
lnevers@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • GENIRacksHome/USIgniteInstageniRacks/ConfirmationTestStatus/UL-IGN

    v27 v28  
    10671067Create a slice and then create the stitched slivers with the RSpec [http://groups.geni.net/geni/browser/trunk/stitch-examples/ULLafayette/IG-ST-1.rspec IG-ST-1.rspec]:
    10681068{{{
    1069 $ stitcher.py createsliver IG-ST-1a IG-ST-1.rspec -o
    1070 14:12:15 INFO    : Configured logging from file /home/lnevers/geni-tools-2.10/src/gcf/stitcher_logging.conf
    1071 14:12:15 INFO    : Reading slice IG-ST-1a credential...
    1072 14:12:16 INFO    : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+IG-ST-1a expires on 2017-06-02 18:12:15 UTC
    1073 14:12:16 INFO    : Calling SCS...
    1074 14:12:22 INFO    : Multi-AM reservation will include resources from these aggregates:
    1075 14:12:22 INFO    :      <Aggregate ulla-ig>
    1076 14:12:22 INFO    :      <Aggregate gpo-ig>
    1077 14:12:22 INFO    :      <Aggregate al2s>
    1078 14:12:22 INFO    : Stitcher doing createsliver at <Aggregate ulla-ig>...
    1079 14:12:44 INFO    : ... Allocation at <Aggregate ulla-ig> complete.
    1080 14:12:44 INFO    : Stitcher doing createsliver at <Aggregate gpo-ig>...
    1081 14:13:04 INFO    : ... Allocation at <Aggregate gpo-ig> complete.
    1082 14:13:04 INFO    : Stitcher doing createsliver at <Aggregate al2s>...
    1083 14:13:12 INFO    : ... Allocation at <Aggregate al2s> complete.
    1084 14:13:12 INFO    : All aggregates are complete.
    1085 14:13:12 INFO    : Your resources expire at 2017-06-01T18:02:22 (UTC).
    1086 14:13:12 INFO    : Saved combined reservation RSpec at 3 AM(s) to file '/home/lnevers/gcf-test/instarspec/ULLafayette/IG-ST-1a-manifest-rspec-multiam-combined.xml'
    1087 Success: Reserved resources in slice IG-ST-1a at 3 Aggregates (including 1 intermediate aggregate(s) not in the original request), creating 1 stitched link(s).
     1069$ stitcher.py createsliver IG-ST-1 IG-ST-1.rspec -o
     107009:04:29 INFO    : Configured logging from file /home/lnevers/geni-tools-2.10/src/gcf/stitcher_logging.conf
     107109:04:29 INFO    : Reading slice IG-ST-1 credential...
     107209:04:29 INFO    : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+IG-ST-1 expires on 2017-06-26 13:04:18 UTC
     107309:04:29 INFO    : Calling SCS...
     107409:04:35 INFO    : Multi-AM reservation will include resources from these aggregates:
     107509:04:35 INFO    :      <Aggregate ulla-ig>
     107609:04:35 INFO    :      <Aggregate gpo-ig>
     107709:04:35 INFO    :      <Aggregate al2s>
     107809:04:35 INFO    : Stitcher doing createsliver at <Aggregate ulla-ig>...
     107909:04:54 INFO    : ... Allocation at <Aggregate ulla-ig> complete.
     108009:04:54 INFO    : Stitcher doing createsliver at <Aggregate gpo-ig>...
     108109:05:15 INFO    : ... Allocation at <Aggregate gpo-ig> complete.
     108209:05:15 INFO    : Stitcher doing createsliver at <Aggregate al2s>...
     108309:05:23 INFO    : ... Allocation at <Aggregate al2s> complete.
     108409:05:23 INFO    : All aggregates are complete.
     108509:05:23 INFO    : Your resources expire at 2017-06-25T12:54:35 (UTC).
     108609:05:23 INFO    : Saved combined reservation RSpec at 3 AM(s) to file '/home/lnevers/gcf-test/instarspec/ULLafayette/IG-ST-1-manifest-rspec-multiam-combined.xml'
    10881087}}}
    10891088
    … …  
    10951094ig-ulla's geni_status is: ready
    10961095User lnevers logs in to ig-ulla using:
    1097         ssh -p 25010  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc1.instageni.louisiana.edu
     1096        ssh -p 25010  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc3.instageni.louisiana.edu
    10981097
    10991098
    11001099ig-gpo's geni_status is: ready
    11011100User lnevers logs in to ig-gpo using:
    1102         ssh -p 25266  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc2.instageni.gpolab.bbn.com
     1101        ssh -p 25042  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc2.instageni.gpolab.bbn.com
    11031102}}}
    11041103
    … …  
    11101109'''Iperf InstaGENI GPO VM to InstaGENI ULLafayette VM (TCP) - TCP window size: 85.0 KByte (default)'''
    11111110
    1112 ''Collected: 2017-05-09''
     1111''Collected: 2017-06-19''
    11131112
    11141113__One Client_
    11151114{{{
    1116 
     1115[ ID] Interval       Transfer     Bandwidth
     1116[  3]  0.0-60.1 sec   575 MBytes  80.3 Mbits/sec
    11171117}}}
    11181118
    11191119__Five Clients__
    11201120{{{
    1121 
     1121[ ID] Interval       Transfer     Bandwidth
     1122[  3]  0.0-60.0 sec   153 MBytes  21.3 Mbits/sec
     1123[  4]  0.0-60.1 sec   133 MBytes  18.5 Mbits/sec
     1124[  5]  0.0-60.1 sec   131 MBytes  18.3 Mbits/sec
     1125[  7]  0.0-60.1 sec   129 MBytes  18.0 Mbits/sec
     1126[  6]  0.0-60.1 sec   119 MBytes  16.5 Mbits/sec
     1127[SUM]  0.0-60.1 sec   664 MBytes  92.6 Mbits/sec
    11221128}}}
    11231129
    11241130__Ten Clients__
    11251131{{{
    1126 
     1132[ ID] Interval       Transfer     Bandwidth
     1133[  8]  0.0-60.0 sec  72.2 MBytes  10.1 Mbits/sec
     1134[ 10]  0.0-60.0 sec  68.5 MBytes  9.57 Mbits/sec
     1135[ 12]  0.0-60.1 sec  69.0 MBytes  9.64 Mbits/sec
     1136[  4]  0.0-60.1 sec  64.8 MBytes  9.04 Mbits/sec
     1137[  3]  0.0-60.2 sec  61.4 MBytes  8.55 Mbits/sec
     1138[  9]  0.0-60.2 sec  62.0 MBytes  8.64 Mbits/sec
     1139[  5]  0.0-60.2 sec  66.0 MBytes  9.19 Mbits/sec
     1140[  6]  0.0-60.3 sec  63.4 MBytes  8.82 Mbits/sec
     1141[ 11]  0.0-60.3 sec  78.5 MBytes  10.9 Mbits/sec
     1142[  7]  0.0-60.3 sec  67.1 MBytes  9.34 Mbits/sec
     1143[SUM]  0.0-60.3 sec   673 MBytes  93.7 Mbits/sec
    11271144}}}
    11281145
    11291146'''Iperf InstaGENI GPO VM to InstaGENI ULLafayette VM (UDP) - UDP buffer size:  208 KByte (default) '''
    11301147{{{
    1131 
     1148[ ID] Interval       Transfer     Bandwidth
     1149[  3]  0.0-60.0 sec   719 MBytes   101 Mbits/sec
     1150[  3] Sent 512816 datagrams
     1151[  3] Server Report:
     1152[  3]  0.0-60.0 sec   691 MBytes  96.5 Mbits/sec   0.023 ms 20228/512815 (3.9%)
     1153[  3]  0.0-60.0 sec  1 datagrams received out-of-order
    11321154}}}
    11331155
    11341156'''Ping from InstaGENI GPO VM to the InstaGENI ULLafayette VM '''
    11351157{{{
    1136 
     115860 packets transmitted, 60 received, 0% packet loss, time 59076ms
     1159rtt min/avg/max/mdev = 47.994/48.246/50.873/0.470 ms
    11371160}}}
    11381161
    … …  
    11401163'''Iperf InstaGENI ULLafayette VM to GPO InstaGENI VM (TCP) - TCP window size: 85.0 KByte (default)'''
    11411164
    1142 ''Collected: 2017-05-09''
     1165''Collected: 2017-06-19''
    11431166
    11441167__One Client_
    11451168{{{
    1146 
     1169[ ID] Interval       Transfer     Bandwidth
     1170[  3]  0.0-60.2 sec   598 MBytes  83.4 Mbits/sec
    11471171}}}
    11481172
    11491173__Five Clients__
    11501174{{{
    1151 
     1175[ ID] Interval       Transfer     Bandwidth
     1176[  5]  0.0-60.0 sec   110 MBytes  15.4 Mbits/sec
     1177[  3]  0.0-60.1 sec  93.1 MBytes  13.0 Mbits/sec
     1178[  6]  0.0-60.1 sec   269 MBytes  37.6 Mbits/sec
     1179[  4]  0.0-60.1 sec   118 MBytes  16.4 Mbits/sec
     1180[  7]  0.0-60.2 sec  84.5 MBytes  11.8 Mbits/sec
     1181[SUM]  0.0-60.2 sec   674 MBytes  94.0 Mbits/sec
    11521182}}}
    11531183
    11541184__Ten Clients__
    11551185{{{
    1156 
     1186[ ID] Interval       Transfer     Bandwidth
     1187[ 10]  0.0-60.0 sec  70.6 MBytes  9.87 Mbits/sec
     1188[  5]  0.0-60.0 sec  71.5 MBytes  9.99 Mbits/sec
     1189[  7]  0.0-60.0 sec  71.8 MBytes  10.0 Mbits/sec
     1190[  4]  0.0-60.0 sec  59.6 MBytes  8.33 Mbits/sec
     1191[  9]  0.0-60.1 sec  71.2 MBytes  9.94 Mbits/sec
     1192[ 11]  0.0-60.1 sec  59.2 MBytes  8.27 Mbits/sec
     1193[  3]  0.0-60.2 sec  70.2 MBytes  9.79 Mbits/sec
     1194[ 12]  0.0-60.2 sec  64.4 MBytes  8.97 Mbits/sec
     1195[  6]  0.0-60.2 sec  75.8 MBytes  10.5 Mbits/sec
     1196[  8]  0.0-60.3 sec  61.6 MBytes  8.58 Mbits/sec
     1197[SUM]  0.0-60.3 sec   676 MBytes  94.1 Mbits/sec
    11571198}}}
    11581199
    11591200'''Iperf InstaGENI ULLafayette VM to GPO InstaGENI VM (UDP)  - UDP buffer size:  208 KByte (default) '''
    11601201{{{
    1161 
     1202[ ID] Interval       Transfer     Bandwidth
     1203[  3]  0.0-60.0 sec   719 MBytes   101 Mbits/sec
     1204[  3] Sent 512816 datagrams
     1205[  3] Server Report:
     1206[  3]  0.0-60.0 sec   692 MBytes  96.7 Mbits/sec   0.016 ms 19538/512815 (3.8%)
     1207[  3]  0.0-60.0 sec  1 datagrams received out-of-order
    11621208}}}
    11631209
    11641210'''Ping from InstaGENI ULLafayette VM to GPO InstaGENI VM '''
    11651211{{{
    1166 
     121260 packets transmitted, 60 received, 0% packet loss, time 59072ms
     1213rtt min/avg/max/mdev = 48.045/48.161/48.595/0.243 ms
    11671214}}}
    11681215
    … …  
    11721219Create a slice and then create the stitched slivers with the RSpec [http://groups.geni.net/geni/browser/trunk/stitch-examples/ULLafayette/IG-ST-2.rspec IG-ST-2.rspec]:
    11731220{{{
    1174 $ stitcher.py createsliver IG-ST-2x IG-ST-2.rspec -o
    1175 14:01:33 INFO    : Configured logging from file /home/lnevers/geni-tools-2.10/src/gcf/stitcher_logging.conf
    1176 14:01:33 INFO    : Reading slice IG-ST-2x credential...
    1177 14:01:33 INFO    : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+IG-ST-2x expires on 2017-05-16 18:01:32 UTC
    1178 14:01:33 INFO    : Calling SCS...
    1179 14:01:34 INFO    : Multi-AM reservation will include resources from these aggregates:
    1180 14:01:34 INFO    :      <Aggregate ulla-ig>
    1181 14:01:34 INFO    :      <Aggregate gpo-ig>
    1182 14:01:34 INFO    :      <Aggregate al2s>
    1183 14:01:34 INFO    : Stitcher doing createsliver at <Aggregate ulla-ig>...
    1184 14:01:56 INFO    : ... Allocation at <Aggregate ulla-ig> complete.
    1185 14:01:56 INFO    : Stitcher doing createsliver at <Aggregate gpo-ig>...
    1186 14:02:16 INFO    : ... Allocation at <Aggregate gpo-ig> complete.
    1187 14:02:16 INFO    : Stitcher doing createsliver at <Aggregate al2s>...
    1188 14:02:29 INFO    : ... Allocation at <Aggregate al2s> complete.
    1189 14:02:29 INFO    : All aggregates are complete.
    1190 14:02:29 INFO    : Your resources expire at 2017-05-15T17:51:34 (UTC).
    1191 14:02:29 INFO    : Saved combined reservation RSpec at 3 AM(s) to file '/home/lnevers/gcf-test/instarspec/ULLafayette/IG-ST-2x-manifest-rspec-multiam-combined.xml'
    1192 Success: Reserved resources in slice IG-ST-2x at 3 Aggregates (including 1 intermediate aggregate(s) not in the original request), creating 2 stitched link(s).
     1221$ stitcher.py createsliver IG-ST-2 IG-ST-2.rspec -o
     122209:44:34 INFO    : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+IG-ST-2 expires on 2017-06-26 13:44:33 UTC
     122309:44:34 INFO    : Calling SCS...
     122409:44:35 INFO    : Multi-AM reservation will include resources from these aggregates:
     122509:44:35 INFO    :      <Aggregate ulla-ig>
     122609:44:35 INFO    :      <Aggregate gpo-ig>
     122709:44:35 INFO    :      <Aggregate al2s>
     122809:44:35 INFO    : Stitcher doing createsliver at <Aggregate ulla-ig>...
     122909:44:58 INFO    : ... Allocation at <Aggregate ulla-ig> complete.
     123009:44:58 INFO    : Stitcher doing createsliver at <Aggregate gpo-ig>...
     123109:45:18 INFO    : ... Allocation at <Aggregate gpo-ig> complete.
     123209:45:18 INFO    : Stitcher doing createsliver at <Aggregate al2s>...
     123309:45:32 INFO    : ... Allocation at <Aggregate al2s> complete.
     123409:45:32 INFO    : All aggregates are complete.
     123509:45:32 INFO    : Your resources expire at 2017-06-25T13:34:35 (UTC).
     123609:45:32 INFO    : Saved combined reservation RSpec at 3 AM(s) to file '/home/lnevers/gcf-test/instarspec/ULLafayette/IG-ST-2-manifest-rspec-multiam-combined.xml'
     1237Success: Reserved resources in slice IG-ST-2 at 3 Aggregates (including 1 intermediate aggregate(s) not in the original request), creating 2 stitched link(s).
    11931238}}}
    11941239
    11951240Determined login information at each ULLafayette and GPO aggregate:
    11961241{{{
    1197 $ readyToLogin.py IG-ST-2x --useSliceAggregates
     1242$ readyToLogin.py IG-ST-2 --useSliceAggregates
    11981243 ....
    11991244ig-ulla's geni_status is: ready
    12001245User lnevers logs in to ig-ulla using:
    1201         ssh -p 25106  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc1.instageni.louisiana.edu
     1246        ssh -p 25042  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc1.instageni.louisiana.edu
    12021247
    12031248ig-gpo's geni_status is: ready
    12041249User lnevers logs in to ig-gpo using:
    1205         ssh -p 25298  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc2.instageni.gpolab.bbn.com
     1250        ssh -p 25298  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc5.instageni.gpolab.bbn.com
    12061251}}}
    12071252
    12081253Login to GPO host and ping the remote on each of the two interfaces.  Below is the ping output for the GPO site:
    12091254{{{
    1210 $ remote-execute.py IG-ST-2x -a gpo-ig -m "ping 10.10.4.2 -c 60 -q ;ping 192.168.4.2 -c 60 -q"
     1255$ remote-execute.py IG-ST-2 -a gpo-ig -m "ping 10.10.4.2 -c 60 -q ;ping 192.168.4.2 -c 60 -q"
    121112561. Find login Info for hosts in slice IG-ST-2x
    121212572. Send command 'ping 10.10.4.2 -c 60 -q ;ping 192.168.4.2 -c 60 -q' to ig-gpo
     12582. Send command 'ping 10.10.4.2 -c 60 -q ;ping 192.168.4.2 -c 60 -q' to ig-gpo
    12131259
    12141260PING 10.10.4.2 (10.10.4.2) 56(84) bytes of data.
    12151261
    12161262--- 10.10.4.2 ping statistics ---
    1217 60 packets transmitted, 0 received, +60 errors, 100% packet loss, time 59319ms
    1218 pipe 3
     126360 packets transmitted, 60 received, 0% packet loss, time 59067ms
     1264rtt min/avg/max/mdev = 47.992/48.893/96.224/6.167 ms
    12191265PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
    12201266
    12211267--- 192.168.4.2 ping statistics ---
    1222 60 packets transmitted, 0 received, +60 errors, 100% packet loss, time 59319ms
    1223 pipe 3
     126860 packets transmitted, 60 received, 0% packet loss, time 59014ms
     1269rtt min/avg/max/mdev = 47.930/49.009/96.294/6.182 ms
    12241270... Done with command 'ping 10.10.4.2 -c 60 -q ;ping 192.168.4.2 -c 60 -q' at ig-gpo
     1271
    12251272}}}
    12261273
    … …  
    12301277{{{
    12311278$ stitcher.py createsliver IG-ST-3 IG-ST-3.rspec -o
    1232 14:24:00 INFO    : Configured logging from file /home/lnevers/geni-tools-2.10/src/gcf/stitcher_logging.conf
    1233 14:24:00 INFO    : Reading slice IG-ST-3 credential...
    1234 14:24:01 INFO    : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+IG-ST-3 expires on 2017-05-16 18:23:59 UTC
    1235 14:24:01 INFO    : Calling SCS...
    1236 14:24:04 INFO    : Multi-AM reservation will include resources from these aggregates:
    1237 14:24:04 INFO    :      <Aggregate ulla-ig>
    1238 14:24:04 INFO    :      <Aggregate gpo-ig>
    1239 14:24:04 INFO    :      <Aggregate al2s>
    1240 14:24:04 INFO    :      <Aggregate uwashington-ig>
    1241 14:24:04 INFO    : Stitcher doing createsliver at <Aggregate ulla-ig>...
    1242 14:24:26 INFO    : ... Allocation at <Aggregate ulla-ig> complete.
    1243 14:24:26 INFO    : Stitcher doing createsliver at <Aggregate gpo-ig>...
    1244 14:24:44 INFO    : ... Allocation at <Aggregate gpo-ig> complete.
    1245 14:24:44 INFO    : Stitcher doing createsliver at <Aggregate uwashington-ig>...
    1246 14:25:07 INFO    : ... Allocation at <Aggregate uwashington-ig> complete.
    1247 14:25:07 INFO    : Stitcher doing createsliver at <Aggregate al2s>...
    1248 14:25:20 INFO    : ... Allocation at <Aggregate al2s> complete.
    1249 14:25:20 INFO    : All aggregates are complete.
    1250 14:25:20 INFO    : Your resources expire at 2017-05-15T18:14:04 (UTC).
    1251 14:25:20 INFO    : Saved combined reservation RSpec at 4 AM(s) to file '/home/lnevers/gcf-test/instarspec/ULLafayette/IG-ST-3-manifest-rspec-multiam-combined.xml'
     127909:50:14 INFO    : Configured logging from file /home/lnevers/geni-tools-2.10/src/gcf/stitcher_logging.conf
     128009:50:14 INFO    : Reading slice IG-ST-3 credential...
     128109:50:14 INFO    : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+IG-ST-3 expires on 2017-06-26 13:50:14 UTC
     128209:50:14 INFO    : Calling SCS...
     128309:50:17 INFO    : Multi-AM reservation will include resources from these aggregates:
     128409:50:17 INFO    :      <Aggregate ulla-ig>
     128509:50:17 INFO    :      <Aggregate gpo-ig>
     128609:50:17 INFO    :      <Aggregate al2s>
     128709:50:17 INFO    :      <Aggregate uwashington-ig>
     128809:50:17 INFO    : Stitcher doing createsliver at <Aggregate ulla-ig>...
     128909:50:40 INFO    : ... Allocation at <Aggregate ulla-ig> complete.
     129009:50:40 INFO    : Stitcher doing createsliver at <Aggregate gpo-ig>...
     129109:51:00 ERROR   :  {'output': "*** SetNodeHistory: No control interface for [Node: interconnect-poblanovm-6]\n*** WARNING: mapper:\n***   DB Query failed:\n***     Query: insert into interface_state set   node_id='pcvm5-2',   card=0, port=1, iface='eth0'\n***     Error: Duplicate entry 'pcvm5-2-0-1' for key 1 (1062)\n*** CreateVnodes: Could not create interface for pcvm5-2\n*** ERROR: mapper: Could not allocate vnodes on pc5\n*** ERROR: mapper: Could not allocate virtual nodes\n*** ERROR: mapper: Unretriable error. Giving up.\n*** ERROR: mapper: 1 warnings.\n", 'code': {'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+ccacc0d7bf93f99e64c104504c2e8e1f', 'am_type': 'protogeni', 'geni_code': 2, 'am_code': 28, 'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=ccacc0d7bf93f99e64c104504c2e8e1f'}, 'value': 'Could not map to resources'}
     129209:51:00 WARNING : Stitching failed but will retry: Circuit reservation failed at <Aggregate gpo-ig> (AMAPIError: Error from Aggregate: code 2. protogeni AM code: 28: *** SetNodeHistory: No control interface for [Node: interconnect-poblanovm-6]
     1293*** WARNING: mapper:
     1294***   DB Query failed:
     1295***     Query: insert into interface_state set   node_id='pcvm5-2',   card=0, port=1, iface='eth0'
     1296***     Error: Duplicate entry 'pcvm5-2-0-1' for key 1 (1062)
     1297*** CreateVnodes: Could not create interface for pcvm5-2
     1298*** ERROR: mapper: Could not allocate vnodes on pc5
     1299*** ERROR: mapper: Could not allocate virtual nodes
     1300*** ERROR: mapper: Unretriable error. Giving up.
     1301*** ERROR: mapper: 1 warnings.
     1302.). Try again from the SCS
     130309:51:00 INFO    : Deleting existing reservations...
     130409:51:00 INFO    : Doing deletesliver at <Aggregate ulla-ig>...
     130509:51:03 INFO    : Deleted reservation at <Aggregate ulla-ig>.
     130609:51:03 INFO    : Calling SCS for the 2nd time...
     130709:51:03 INFO    : Pausing for 30 seconds for Aggregates to free up resources...
     1308
     1309
     131009:51:33 INFO    : Multi-AM reservation will include resources from these aggregates:
     131109:51:33 INFO    :      <Aggregate ulla-ig>
     131209:51:33 INFO    :      <Aggregate gpo-ig>
     131309:51:33 INFO    :      <Aggregate al2s>
     131409:51:33 INFO    :      <Aggregate uwashington-ig>
     131509:51:33 INFO    : Stitcher doing createsliver at <Aggregate ulla-ig>...
     131609:51:56 INFO    : ... Allocation at <Aggregate ulla-ig> complete.
     131709:51:56 INFO    : Stitcher doing createsliver at <Aggregate gpo-ig>...
     131809:52:14 INFO    : ... Allocation at <Aggregate gpo-ig> complete.
     131909:52:14 INFO    : Stitcher doing createsliver at <Aggregate uwashington-ig>...
     132009:52:37 INFO    : ... Allocation at <Aggregate uwashington-ig> complete.
     132109:52:37 INFO    : Stitcher doing createsliver at <Aggregate al2s>...
     132209:52:45 ERROR   :  {'output': 'Exception: requested VLAN unavailable: rtsw.newy32aoa.net.internet2.edu,et-7/3/0.0 VLAN=3718', 'code': {'geni_code': 24}}
     132309:52:45 INFO    : Deleting some reservations to retry, avoiding failed VLAN...
     132409:52:45 INFO    : Doing deletesliver at <Aggregate gpo-ig>...
     132509:54:16 ERROR   :  {'output': 'start/restart in progress; try again later', 'code': {'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+ccf919fcca89cc6691f88b1c203d0c17', 'am_type': 'protogeni', 'geni_code': 14, 'am_code': 14, 'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=ccf919fcca89cc6691f88b1c203d0c17'}, 'value': 0}
     132609:54:26 INFO    :  ... aggregate was busy, will retry ...
     132709:55:11 INFO    :  ... done.
     132809:55:11 INFO    : Will put <Aggregate al2s> back in the pool to allocate. Got: Retrying reservations at earlier AMs to avoid unavailable VLAN tag at <Aggregate al2s>....
     132909:55:11 INFO    : Pausing for 30 seconds for Aggregates to free up resources...
     1330
     1331
     133209:55:41 INFO    : Checking that prior reservation at <Aggregate gpo-ig> has been cleared up....
     133309:55:43 ERROR   :  {'output': 'No slice or aggregate here', 'code': {'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+3587969c16fb07f29dc1b83303e49170', 'am_type': 'protogeni', 'geni_code': 12, 'am_code': 12, 'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=3587969c16fb07f29dc1b83303e49170'}, 'value': 0}
     133409:55:43 INFO    : ... it is, so can try a new reservation.
     133509:55:43 INFO    : Stitcher doing createsliver at <Aggregate gpo-ig>...
     133609:56:23 INFO    : ... Allocation at <Aggregate gpo-ig> complete.
     133709:56:23 INFO    : Stitcher doing createsliver at <Aggregate al2s>...
     133809:56:37 INFO    : ... Allocation at <Aggregate al2s> complete.
     133909:56:37 INFO    : All aggregates are complete.
     134009:56:37 INFO    : Your resources expire at 2017-06-25T13:41:33 (UTC).
     134109:56:37 INFO    : Saved combined reservation RSpec at 4 AM(s) to file '/home/lnevers/gcf-test/instarspec/ULLafayette/IG-ST-3-manifest-rspec-multiam-combined.xml'
    12521342Success: Reserved resources in slice IG-ST-3 at 4 Aggregates (including 1 intermediate aggregate(s) not in the original request), creating 2 stitched link(s).
     1343
    12531344}}}
    12541345
    … …  
    12591350ig-ulla's geni_status is: ready
    12601351User lnevers logs in to ig-ulla using:
    1261         ssh -p 25106  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc3.instageni.louisiana.edu
     1352        ssh -p 25010  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc3.instageni.louisiana.edu
    12621353
    12631354ig-gpo's geni_status is: ready
    12641355User lnevers logs in to ig-gpo using:
    1265         ssh -p 25234  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc5.instageni.gpolab.bbn.com
     1356        ssh -p 25042  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc2.instageni.gpolab.bbn.com
    12661357
    12671358ig-wash's geni_status is: ready
    12681359User lnevers logs in to ig-wash using:
    1269         ssh -p 25138  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc2.instageni.washington.edu
     1360        ssh -p 25074  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc3.instageni.washington.edu
    12701361}}}
    12711362Login to the ULLafayette host and ping each remote:
    12721363{{{
     1364
    12731365#ping GPO IG
     1366
    12741367lnevers@ig-ulla:~$ ping 192.168.2.1 -c 60 -q
    12751368PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
    12761369
    12771370--- 192.168.2.1 ping statistics ---
    1278 60 packets transmitted, 0 received, +60 errors, 100% packet loss, time 58999ms
    1279 pipe 3
    1280 lnevers@ig-ulla:~$
    1281 
     137160 packets transmitted, 60 received, 0% packet loss, time 59071ms
     1372rtt min/avg/max/mdev = 48.002/48.935/96.357/6.176 ms
     1373lnevers@ig-ulla:~$
    12821374
    12831375# Washington IG
    … …  
    12871379
    12881380--- 192.168.4.1 ping statistics ---
    1289 60 packets transmitted, 0 received, +60 errors, 100% packet loss, time 58999ms
    1290 pipe 3
    1291 lnevers@ig-ulla:~$
     138160 packets transmitted, 60 received, 0% packet loss, time 59098ms
     1382rtt min/avg/max/mdev = 67.794/69.061/135.858/8.703 ms
     1383lnevers@ig-ulla:~$ 
    12921384}}}
    12931385
    … …  
    12981390
    12991391{{{
    1300 $ stitcher.py createsliver IG-ST-4h IG-ST-4.rspec -o --defaultCapacity=100000
    1301 15:13:39 INFO    : Configured logging from file /home/lnevers/geni-tools-2.10/src/gcf/stitcher_logging.conf
    1302 15:13:40 INFO    : Reading slice IG-ST-4i credential...
    1303 15:13:40 INFO    : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+IG-ST-4i expires on 2017-05-16 19:13:39 UTC
    1304 15:13:40 INFO    : Calling SCS...
    1305 15:13:40 INFO    : Multi-AM reservation will include resources from these aggregates:
    1306 15:13:40 INFO    :      <Aggregate ulla-ig>
    1307 15:13:40 INFO    :      <Aggregate al2s>
    1308 15:13:40 INFO    :      <Aggregate gpo-eg>
    1309 15:13:40 INFO    : Stitcher doing createsliver at <Aggregate ulla-ig>...
    1310 15:14:01 INFO    : ... Allocation at <Aggregate ulla-ig> complete.
    1311 15:14:01 INFO    : Stitcher doing createsliver at <Aggregate gpo-eg>...
    1312 15:14:11 INFO    : ... Allocation at <Aggregate gpo-eg> complete.
    1313 15:14:11 INFO    : Stitcher doing createsliver at <Aggregate al2s>...
    1314 15:14:13 ERROR   :  {'output': 'Exception: requested VLAN unavailable: rtsw.newy32aoa.net.internet2.edu,et-7/3/0.0 VLAN=3741', 'code': {'geni_code': 24}}
    1315 15:14:13 WARNING : Stitching failed but will retry: Circuit reservation failed at <Aggregate al2s>. Try again from the SCS. (Error: Topology too complex for local redo - ask Stitching Service to find a VLAN tag (AMAPIError: Error from Aggregate: code 24: Exception: requested VLAN unavailable: rtsw.newy32aoa.net.internet2.edu,et-7/3/0.0 VLAN=3741.))
    1316 15:14:13 INFO    : Deleting existing reservations...
    1317 15:14:13 INFO    : Doing deletesliver at <Aggregate ulla-ig>...
    1318 15:14:16 INFO    : Deleted reservation at <Aggregate ulla-ig>.
    1319 15:14:16 INFO    : Doing deletesliver at <Aggregate gpo-eg>...
    1320 15:14:21 INFO    : Deleted reservation at <Aggregate gpo-eg>.
    1321 15:14:21 INFO    : Calling SCS for the 2nd time...
    1322 15:14:21 INFO    : Pausing for 30 seconds for Aggregates to free up resources...
    1323 
    1324 
    1325 15:14:51 INFO    : Multi-AM reservation will include resources from these aggregates:
    1326 15:14:51 INFO    :      <Aggregate ulla-ig>
    1327 15:14:51 INFO    :      <Aggregate al2s>
    1328 15:14:51 INFO    :      <Aggregate gpo-eg>
    1329 15:14:51 INFO    : Stitcher doing createsliver at <Aggregate ulla-ig>...
    1330 15:15:34 INFO    : ... Allocation at <Aggregate ulla-ig> complete.
    1331 15:15:34 INFO    : Stitcher doing createsliver at <Aggregate gpo-eg>...
    1332 15:15:43 INFO    : ... Allocation at <Aggregate gpo-eg> complete.
    1333 15:15:43 INFO    : Stitcher doing createsliver at <Aggregate al2s>...
    1334 15:15:45 ERROR   :  {'output': 'Exception: requested VLAN unavailable: rtsw.newy32aoa.net.internet2.edu,et-7/3/0.0 VLAN=3739', 'code': {'geni_code': 24}}
    1335 15:15:45 WARNING : Stitching failed but will retry: Circuit reservation failed at <Aggregate al2s>. Try again from the SCS. (Error: Topology too complex for local redo - ask Stitching Service to find a VLAN tag (AMAPIError: Error from Aggregate: code 24: Exception: requested VLAN unavailable: rtsw.newy32aoa.net.internet2.edu,et-7/3/0.0 VLAN=3739.))
    1336 15:15:45 INFO    : Deleting existing reservations...
    1337 15:15:45 INFO    : Doing deletesliver at <Aggregate ulla-ig>...
    1338 15:15:48 INFO    : Deleted reservation at <Aggregate ulla-ig>.
    1339 15:15:48 INFO    : Doing deletesliver at <Aggregate gpo-eg>...
    1340 15:15:51 INFO    : Deleted reservation at <Aggregate gpo-eg>.
    1341 15:15:51 INFO    : Calling SCS for the 3rd time...
    1342 15:15:51 INFO    : Pausing for 30 seconds for Aggregates to free up resources...
    1343 
    1344 
    1345 15:16:21 INFO    : Multi-AM reservation will include resources from these aggregates:
    1346 15:16:21 INFO    :      <Aggregate ulla-ig>
    1347 15:16:21 INFO    :      <Aggregate al2s>
    1348 15:16:21 INFO    :      <Aggregate gpo-eg>
    1349 15:16:21 INFO    : Stitcher doing createsliver at <Aggregate ulla-ig>...
    1350 15:16:43 INFO    : ... Allocation at <Aggregate ulla-ig> complete.
    1351 15:16:43 INFO    : Stitcher doing createsliver at <Aggregate gpo-eg>...
    1352 15:16:53 INFO    : ... Allocation at <Aggregate gpo-eg> complete.
    1353 15:16:53 INFO    : Stitcher doing createsliver at <Aggregate al2s>...
    1354 15:16:55 ERROR   :  {'output': 'Exception: requested VLAN unavailable: rtsw.newy32aoa.net.internet2.edu,et-7/3/0.0 VLAN=3738', 'code': {'geni_code': 24}}
    1355 15:16:55 WARNING : Stitching failed but will retry: Circuit reservation failed at <Aggregate al2s>. Try again from the SCS. (Error: Topology too complex for local redo - ask Stitching Service to find a VLAN tag (AMAPIError: Error from Aggregate: code 24: Exception: requested VLAN unavailable: rtsw.newy32aoa.net.internet2.edu,et-7/3/0.0 VLAN=3738.))
    1356 15:16:55 INFO    : Deleting existing reservations...
    1357 15:16:55 INFO    : Doing deletesliver at <Aggregate ulla-ig>...
    1358 15:16:58 INFO    : Deleted reservation at <Aggregate ulla-ig>.
    1359 15:16:58 INFO    : Doing deletesliver at <Aggregate gpo-eg>...
    1360 15:17:01 INFO    : Deleted reservation at <Aggregate gpo-eg>.
    1361 15:17:01 INFO    : Calling SCS for the 4th time...
    1362 15:17:02 INFO    : Pausing for 30 seconds for Aggregates to free up resources...
    1363 
    1364 
    1365 15:17:32 INFO    : Multi-AM reservation will include resources from these aggregates:
    1366 15:17:32 INFO    :      <Aggregate ulla-ig>
    1367 15:17:32 INFO    :      <Aggregate al2s>
    1368 15:17:32 INFO    :      <Aggregate gpo-eg>
    1369 15:17:32 INFO    : Stitcher doing createsliver at <Aggregate ulla-ig>...
    1370 15:18:15 INFO    : ... Allocation at <Aggregate ulla-ig> complete.
    1371 15:18:15 INFO    : Stitcher doing createsliver at <Aggregate gpo-eg>...
    1372 15:18:24 INFO    : ... Allocation at <Aggregate gpo-eg> complete.
    1373 15:18:24 INFO    : Stitcher doing createsliver at <Aggregate al2s>...
    1374 15:18:32 INFO    : ... Allocation at <Aggregate al2s> complete.
    1375 15:18:32 INFO    : All aggregates are complete.
    1376 15:18:32 INFO    : Your resources expire at 2017-05-15T19:07:32 (UTC).
    1377 15:18:32 INFO    : Saved combined reservation RSpec at 3 AM(s) to file '/home/lnevers/gcf-test/instarspec/ULLafayette/IG-ST-4i-manifest-rspec-multiam-combined.xml'
    1378 Success: Reserved resources in slice IG-ST-4i at 3 Aggregates (including 1 intermediate aggregate(s) not in the original request), creating 1 stitched link(s).
    1379 
     1392$ stitcher.py createsliver IG-ST-4 IG-ST-4.rspec -o --defaultCapacity=100000
     139310:06:48 INFO    : Configured logging from file /home/lnevers/geni-tools-2.10/src/gcf/stitcher_logging.conf
     139410:06:48 INFO    : Reading slice IG-ST-4 credential...
     139510:06:48 INFO    : Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+IG-ST-4 expires on 2017-06-26 14:06:47 UTC
     139610:06:48 INFO    : Calling SCS...
     139710:06:49 INFO    : Multi-AM reservation will include resources from these aggregates:
     139810:06:49 INFO    :      <Aggregate ulla-ig>
     139910:06:49 INFO    :      <Aggregate al2s>
     140010:06:49 INFO    :      <Aggregate gpo-eg>
     140110:06:49 INFO    : Stitcher doing createsliver at <Aggregate ulla-ig>...
     140210:07:11 INFO    : ... Allocation at <Aggregate ulla-ig> complete.
     140310:07:11 INFO    : Stitcher doing createsliver at <Aggregate gpo-eg>...
     140410:07:21 INFO    : ... Allocation at <Aggregate gpo-eg> complete.
     140510:07:21 INFO    : Stitcher doing createsliver at <Aggregate al2s>...
     140610:07:29 INFO    : ... Allocation at <Aggregate al2s> complete.
     140710:07:29 INFO    : All aggregates are complete.
     140810:07:29 INFO    : Your resources expire at 2017-06-25T13:56:49 (UTC).
     140910:07:29 INFO    : Saved combined reservation RSpec at 3 AM(s) to file '/home/lnevers/gcf-test/instarspec/ULLafayette/IG-ST-4-manifest-rspec-multiam-combined.xml'
     1410Success: Reserved resources in slice IG-ST-4 at 3 Aggregates (including 1 intermediate aggregate(s) not in the original request), creating 1 stitched link(s).
    13801411}}}
    13811412
    13821413Determined login information at each ULLafayette and GPO aggregate:
    13831414{{{
    1384 $ readyToLogin.py IG-ST-4i --useSliceAggregates
    1385 }}}
    1386 
    1387 .... NO IP CONNECTIVITY OVER STITCHED LINK ....
     1415$ readyToLogin.py IG-ST-4 --useSliceAggregates
     1416
     1417eg-gpo's geni_status is: ready
     1418User lnevers logs in to eg-gpo using:
     1419        ssh -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@192.1.242.9
     1420
     1421ig-ulla's geni_status is: ready
     1422User lnevers logs in to ig-ulla using:
     1423        ssh -p 25042  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@pc2.instageni.louisiana.edu
     1424}}}
     1425
     1426
    13881427
    13891428=== Measurements ===
    … …  
    13911430'''Iperf ExoGENI GPO VM to InstaGENI ULLafayette VM (TCP) - TCP window size: 85.0 KByte (default)'''
    13921431
     1432''Collected: 2017-06-19''
     1433
     1434__One Client_
     1435{{{
     1436
     1437}}}
     1438
     1439__Five Clients__
     1440{{{
     1441
     1442}}}
     1443
     1444__Ten Clients__
     1445{{{
     1446
     1447}}}
     1448
     1449'''Iperf ExoGENI GPO VM to InstaGENI ULLafayette VM (UDP) - UDP buffer size:  208 KByte (default) '''
     1450{{{
     1451
     1452}}}
     1453
     1454'''Ping from ExoGENI GPO VM to the InstaGENI ULLafayette VM '''
     1455{{{
     1456
     1457}}}
     1458
     1459
     1460'''Iperf InstaGENI ULLafayette VM to ExoGENI GPO VM (TCP) - TCP window size: 85.0 KByte (default)'''
     1461
    13931462''Collected: 2017-05-XX''
    13941463
    … …  
    14081477}}}
    14091478
    1410 '''Iperf ExoGENI GPO VM to InstaGENI ULLafayette VM (UDP) - UDP buffer size:  208 KByte (default) '''
    1411 {{{
    1412 
    1413 }}}
    1414 
    1415 '''Ping from ExoGENI GPO VM to the InstaGENI ULLafayette VM '''
    1416 {{{
    1417 
    1418 }}}
    1419 
    1420 
    1421 '''Iperf InstaGENI ULLafayette VM to ExoGENI GPO VM (TCP) - TCP window size: 85.0 KByte (default)'''
    1422 
    1423 ''Collected: 2017-05-XX''
    1424 
    1425 __One Client_
    1426 {{{
    1427 
    1428 }}}
    1429 
    1430 __Five Clients__
    1431 {{{
    1432 
    1433 }}}
    1434 
    1435 __Ten Clients__
    1436 {{{
    1437 
    1438 }}}
    1439 
    14401479'''Iperf InstaGENI ULLafayette VM to ExoGENI GPO VM (UDP)  - UDP buffer size:  208 KByte (default) '''
    14411480{{{