wiki:GENIEducation/SampleAssignments/TcpAssignment/ExerciseLayout/Execute

Version 6 (modified by shuang@bbn.com, 12 years ago) (diff)

--

<TCP ASSIGNMENT>

Hello GENI index Hello GENI index Hello GENI index

STEPS FOR EXECUTING EXERCISE

Now that we have reserved the nodes. Let's log on to each nodes and do experiment.
Recall that the nodes reserved are pc73, pc81, pc55, pc47, pc84.
pc47 is top; pc84 is bottom; pc81 is left; pc55 is right; pc73 is center. We can figure this out from the output of "createsliver" step.
I typically draw a topology, naming the nodes correctly for quick reference; but again we can use the GENI Portal to see a graphical representation of the topology, with named nodes.
However, the flack interface (we can launch it via GENI Portal) still does not show the interface numbers of each nodes (we might need those interface numbers when we debug our experiment).
For example, we might want to see the traffic sent from center to top, then we need to do "tcpdump" on the interface that center uses to connect to top.
Right now we can not do this via the flack interface. But we can see it from the output of "createsliver" step (another reason I prefer to use command line omni :-))

Useful commands:
Change the use of congestion control algorithm:

echo reno | sudo tee /proc/sys/net/ipv4/tcp_congestion_control
echo cubic | sudo tee /proc/sys/net/ipv4/tcp_congestion_control

Change the delay/loss of a particular interface:

sudo /sbin/tc qdisc add dev eth1 root handle 1:0 netem delay 200ms loss 5%

Restore network delay/loss on nic card:

sudo /sbin/tc qdisc del dev eth1 root

It is a little bit tricky to configure delay/loss on a virtual machine
Step 1: find our qdisc family number by executing "sudo /sbin/tc qdisc", a sample output could be like the following:

[shufeng@center ~]$ sudo /sbin/tc qdisc
qdisc mq 0: dev eth0 root 
qdisc htb 150: dev eth1 root refcnt 9 r2q 10 default 1 direct_packets_stat 0
qdisc netem 140: dev eth1 parent 150:1 limit 1000
qdisc htb 170: dev eth2 root refcnt 9 r2q 10 default 1 direct_packets_stat 0
qdisc netem 160: dev eth2 parent 170:1 limit 1000
qdisc htb 130: dev eth3 root refcnt 9 r2q 10 default 1 direct_packets_stat 0
qdisc netem 120: dev eth3 parent 130:1 limit 1000

Now if the ethernet card you want to change is eth3, you can find the "parent qdisc id" you want to use is "120:1" by looking at the following line:

qdisc netem 120: dev eth3 parent 130:1 limit 1000

As a result, you change the delay/loss by executing the following:

sudo /sbin/tc qdisc add dev eth3 parent 120:1 netem delay 100ms loss 5%
sudo /sbin/tc qdisc change dev eth3 parent 120:1 netem delay 100ms loss 5%
sudo /sbin/tc qdisc del dev eth3 parent 120:1 netem delay 100ms loss 5%

1. compare cubic and reno under no loss/delay introduced:

use default TCP congestion control (cubic) on left and right, run iperf between them:
On left, run:

/usr/local/etc/emulab/emulab-iperf -s 

On right, run (10.10.1.1 is the ip address for left):

/usr/local/etc/emulab/emulab-iperf -c 10.10.1.1 -t 60

Results: 94.2 Mbps for Cubic when there is no delay introduced

------------------------------------------------------------
Client connecting to 10.10.1.1, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 10.10.2.1 port 53755 connected with 10.10.1.1 port 5001
[  3]  0.0-60.0 sec    674 MBytes  94.2 Mbits/sec

Let both left and right use reno as the TCP congestion control mechanism, repeat the experiments:

echo reno | sudo tee /proc/sys/net/ipv4/tcp_congestion_control

Results: 94.2 Mbps for reno when there is no delay introduced

------------------------------------------------------------
Client connecting to 10.10.1.1, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 10.10.2.1 port 53073 connected with 10.10.1.1 port 5001
[  3]  0.0-60.0 sec    674 MBytes  94.2 Mbits/sec

Answer: they are the same under no loss/delay

2. add 300ms delay and see how it goes:

With Cubic, here is the result:

[  3]  0.0-1800.2 sec  6.57 GBytes  31.3 Mbits/sec
[  3]  0.0-60.2 sec    213 MBytes  29.7 Mbits/sec

[  3]  0.0- 1.0 sec  56.0 KBytes    459 Kbits/sec
[  3]  1.0- 2.0 sec    312 KBytes  2.56 Mbits/sec
[  3]  2.0- 3.0 sec    640 KBytes  5.24 Mbits/sec
[  3]  3.0- 4.0 sec  2.67 MBytes  22.4 Mbits/sec
[  3]  4.0- 5.0 sec  3.57 MBytes  29.9 Mbits/sec
[  3]  5.0- 6.0 sec  3.65 MBytes  30.6 Mbits/sec
[  3]  6.0- 7.0 sec  3.70 MBytes  31.1 Mbits/sec
[  3]  7.0- 8.0 sec  3.66 MBytes  30.7 Mbits/sec

With Reno, here is the result:

[  3]  0.0-1800.1 sec  6.57 GBytes  31.3 Mbits/sec
[  3]  0.0-60.2 sec    214 MBytes  29.8 Mbits/sec

[  3]  0.0- 1.0 sec  56.0 KBytes    459 Kbits/sec
[  3]  1.0- 2.0 sec    232 KBytes  1.90 Mbits/sec
[  3]  2.0- 3.0 sec    680 KBytes  5.57 Mbits/sec
[  3]  3.0- 4.0 sec  2.76 MBytes  23.1 Mbits/sec
[  3]  4.0- 5.0 sec  4.11 MBytes  34.5 Mbits/sec
[  3]  5.0- 6.0 sec  3.68 MBytes  30.9 Mbits/sec

Answer: I was hoping to see Cubic out-performs Reno but it seems that they are the same in this case.
For the long run (e.g., 1800 seconds), Cubic and Reno perform similar under no loss and big delay
For slow start, Cubic out-performs Reno (1.0 - 2.0 seconds) when no loss and high delay.

3. repeat the experiments with 30 parallel connections (-P 30 option in iperf) and see how it goes:

With Cubic, here is the result:

[ 12]  0.0-57.0 sec  13.1 MBytes  1.93 Mbits/sec
[ 16]  0.0-57.0 sec  13.8 MBytes  2.03 Mbits/sec
[ 14]  0.0-57.3 sec  20.2 MBytes  2.96 Mbits/sec
[ 18]  0.0-57.4 sec  18.4 MBytes  2.69 Mbits/sec
[  8]  0.0-57.4 sec  20.3 MBytes  2.97 Mbits/sec
[  7]  0.0-57.3 sec  23.7 MBytes  3.48 Mbits/sec
[  6]  0.0-57.3 sec  23.3 MBytes  3.41 Mbits/sec
[  5]  0.0-57.3 sec  29.4 MBytes  4.30 Mbits/sec
[  4]  0.0-57.3 sec  21.0 MBytes  3.07 Mbits/sec
[  3]  0.0-57.5 sec  23.3 MBytes  3.41 Mbits/sec
[ 11]  0.0-57.5 sec  18.5 MBytes  2.70 Mbits/sec
[ 15]  0.0-57.5 sec  23.7 MBytes  3.46 Mbits/sec
[ 13]  0.0-57.6 sec  26.4 MBytes  3.85 Mbits/sec
[ 17]  0.0-57.6 sec  19.3 MBytes  2.81 Mbits/sec
[  9]  0.0-57.8 sec  15.3 MBytes  2.22 Mbits/sec
[ 10]  0.0-57.9 sec  20.5 MBytes  2.97 Mbits/sec
[ 28]  0.0-60.0 sec  23.8 MBytes  3.32 Mbits/sec
[ 30]  0.0-60.0 sec  15.9 MBytes  2.22 Mbits/sec
[ 29]  0.0-60.1 sec  14.7 MBytes  2.05 Mbits/sec
[ 32]  0.0-60.1 sec  27.3 MBytes  3.81 Mbits/sec
[ 19]  0.0-60.1 sec  20.5 MBytes  2.86 Mbits/sec
[ 23]  0.0-60.1 sec  16.2 MBytes  2.25 Mbits/sec
[ 20]  0.0-60.1 sec  30.0 MBytes  4.19 Mbits/sec
[ 26]  0.0-60.1 sec  14.6 MBytes  2.04 Mbits/sec
[ 21]  0.0-60.2 sec  22.1 MBytes  3.07 Mbits/sec
[ 27]  0.0-60.3 sec  19.9 MBytes  2.77 Mbits/sec
[ 22]  0.0-60.4 sec  24.7 MBytes  3.44 Mbits/sec
[ 24]  0.0-60.4 sec  26.1 MBytes  3.62 Mbits/sec
[ 25]  0.0-60.5 sec  28.0 MBytes  3.88 Mbits/sec
[ 31]  0.0-60.5 sec  34.2 MBytes  4.74 Mbits/sec
[SUM]  0.0-60.5 sec    648 MBytes  89.8 Mbits/sec

[  6]  0.0-1797.2 sec    684 MBytes  3.19 Mbits/sec
[  4]  0.0-1797.3 sec    678 MBytes  3.17 Mbits/sec
[  3]  0.0-1797.3 sec    675 MBytes  3.15 Mbits/sec
[ 10]  0.0-1797.8 sec    602 MBytes  2.81 Mbits/sec
[ 12]  0.0-1797.8 sec    664 MBytes  3.10 Mbits/sec
[ 17]  0.0-1797.9 sec    642 MBytes  3.00 Mbits/sec
[ 13]  0.0-1797.9 sec    686 MBytes  3.20 Mbits/sec
[  9]  0.0-1797.9 sec    707 MBytes  3.30 Mbits/sec
[ 14]  0.0-1798.0 sec    679 MBytes  3.17 Mbits/sec
[  5]  0.0-1798.2 sec    620 MBytes  2.89 Mbits/sec
[  8]  0.0-1798.2 sec    671 MBytes  3.13 Mbits/sec
[  7]  0.0-1798.2 sec    723 MBytes  3.37 Mbits/sec
[ 11]  0.0-1798.3 sec    696 MBytes  3.25 Mbits/sec
[ 16]  0.0-1798.3 sec    657 MBytes  3.07 Mbits/sec
[ 15]  0.0-1798.4 sec    624 MBytes  2.91 Mbits/sec
[ 18]  0.0-1798.8 sec    695 MBytes  3.24 Mbits/sec
[ 28]  0.0-1800.1 sec    705 MBytes  3.29 Mbits/sec
[ 23]  0.0-1800.1 sec    689 MBytes  3.21 Mbits/sec
[ 32]  0.0-1800.1 sec    686 MBytes  3.20 Mbits/sec
[ 31]  0.0-1800.2 sec    703 MBytes  3.28 Mbits/sec
[ 21]  0.0-1800.2 sec    671 MBytes  3.13 Mbits/sec
[ 30]  0.0-1800.4 sec    699 MBytes  3.26 Mbits/sec
[ 20]  0.0-1800.5 sec    668 MBytes  3.11 Mbits/sec
[ 22]  0.0-1800.6 sec    652 MBytes  3.04 Mbits/sec
[ 27]  0.0-1800.6 sec    701 MBytes  3.27 Mbits/sec
[ 19]  0.0-1800.6 sec    594 MBytes  2.77 Mbits/sec
[ 29]  0.0-1800.7 sec    680 MBytes  3.17 Mbits/sec
[ 26]  0.0-1800.8 sec    709 MBytes  3.30 Mbits/sec
[ 25]  0.0-1800.9 sec    646 MBytes  3.01 Mbits/sec
[ 24]  0.0-1801.1 sec    672 MBytes  3.13 Mbits/sec
[SUM]  0.0-1801.1 sec  19.7 GBytes  94.0 Mbits/sec

With Reno, here is the result:

[ 17]  0.0-57.1 sec  7.38 MBytes  1.08 Mbits/sec
[ 15]  0.0-57.0 sec  7.33 MBytes  1.08 Mbits/sec
[ 14]  0.0-57.0 sec  7.35 MBytes  1.08 Mbits/sec
[ 18]  0.0-57.0 sec  7.16 MBytes  1.05 Mbits/sec
[ 13]  0.0-57.1 sec  7.31 MBytes  1.08 Mbits/sec
[  3]  0.0-57.2 sec  25.7 MBytes  3.77 Mbits/sec
[ 12]  0.0-57.2 sec  7.33 MBytes  1.08 Mbits/sec
[  5]  0.0-57.2 sec  87.5 MBytes  12.8 Mbits/sec
[  4]  0.0-57.2 sec  26.5 MBytes  3.88 Mbits/sec
[ 11]  0.0-57.2 sec  7.32 MBytes  1.07 Mbits/sec
[ 10]  0.0-57.3 sec  7.38 MBytes  1.08 Mbits/sec
[ 16]  0.0-57.3 sec  7.41 MBytes  1.09 Mbits/sec
[  8]  0.0-57.4 sec  29.6 MBytes  4.33 Mbits/sec
[  7]  0.0-57.7 sec  23.7 MBytes  3.45 Mbits/sec
[  9]  0.0-57.7 sec  23.3 MBytes  3.38 Mbits/sec
[  6]  0.0-58.1 sec  64.6 MBytes  9.33 Mbits/sec
[ 25]  0.0-60.0 sec  43.4 MBytes  6.06 Mbits/sec
[ 21]  0.0-60.0 sec  36.2 MBytes  5.05 Mbits/sec
[ 20]  0.0-60.2 sec  27.3 MBytes  3.81 Mbits/sec
[ 24]  0.0-60.1 sec  28.2 MBytes  3.94 Mbits/sec
[ 23]  0.0-60.1 sec  30.3 MBytes  4.23 Mbits/sec
[ 27]  0.0-60.0 sec  7.80 MBytes  1.09 Mbits/sec
[ 26]  0.0-60.1 sec  7.84 MBytes  1.09 Mbits/sec
[ 30]  0.0-60.1 sec  7.84 MBytes  1.09 Mbits/sec
[ 29]  0.0-60.1 sec  7.74 MBytes  1.08 Mbits/sec
[ 31]  0.0-60.1 sec  7.82 MBytes  1.09 Mbits/sec
[ 19]  0.0-60.3 sec  29.1 MBytes  4.04 Mbits/sec
[ 22]  0.0-60.2 sec  30.9 MBytes  4.31 Mbits/sec
[ 32]  0.0-60.1 sec  32.8 MBytes  4.58 Mbits/sec
[ 28]  0.0-60.1 sec  7.82 MBytes  1.09 Mbits/sec
[SUM]  0.0-60.3 sec    652 MBytes  90.7 Mbits/sec

Answer: The above results show that the total performance is about the same for Cubic and Reno.
Apparently when u use multiple TCP connections the bandwidth utilization will be higher.
But the single throughput for each TCP connection varies.

4. remove 300ms delay, add 5% lossrate and see how it goes:

With Cubic, here is the result:

[  3]  0.0-60.0 sec  73.7 MBytes  10.3 Mbits/sec

10% lossrate: [  3]  0.0-60.6 sec  17.3 MBytes  2.39 Mbits/sec

With Reno, here is the result:

[  3]  0.0-60.0 sec  59.5 MBytes  8.32 Mbits/sec

10% lossrate: [  3]  0.0-60.2 sec  13.5 MBytes  1.89 Mbits/sec

Answer: Apparently Cubic out-performs Reno under 5% lossrate.

5. restore NIC back to no loss and no delay, run 10 TCP connections from right to left, while running 20Mbps UDP session from top to left

UDP throughput:

[  3]  0.0-60.1 sec    141 MBytes  19.6 Mbits/sec  0.416 ms  431/100735 (0.43%)

TCP throughput:

[  5]  0.0-60.1 sec  50.2 MBytes  7.01 Mbits/sec
[  4]  0.0-60.0 sec  78.8 MBytes  11.0 Mbits/sec
[  7]  0.0-60.0 sec  55.0 MBytes  7.69 Mbits/sec
[  6]  0.0-60.0 sec  71.1 MBytes  9.94 Mbits/sec
[  8]  0.0-60.1 sec  39.5 MBytes  5.52 Mbits/sec
[ 10]  0.0-60.0 sec  37.7 MBytes  5.27 Mbits/sec
[ 11]  0.0-60.1 sec  39.5 MBytes  5.51 Mbits/sec
[ 12]  0.0-60.0 sec  73.6 MBytes  10.3 Mbits/sec
[  9]  0.0-60.1 sec  46.8 MBytes  6.54 Mbits/sec
[  3]  0.0-60.3 sec  49.1 MBytes  6.83 Mbits/sec
[SUM]  0.0-60.3 sec    541 MBytes  75.3 Mbits/sec

Answer: Apparently UDP will not care about loss: the client keeps sending at a rate of 20Mbps despite 0.43% loss.
On the other hand, TCP will do its rate control/congestion control mechanism when facing with packets loss and hence has smaller throughputs.

6. follow question 5, how to enforce fairness using tc qdisc for this 11 flows? Prove it

Let's try the following command and see how it goes (it simply uses fair-queuing discipline):

sudo /sbin/tc qdisc add dev eth2 root handle 1:0 sfq

UDP throughput:

[  3]  0.0-60.0 sec    141 MBytes  19.7 Mbits/sec
[  3] Sent 100367 datagrams
[  3] Server Report:
[  3]  0.0-60.0 sec  67.3 MBytes  9.40 Mbits/sec  2.355 ms 52361/100366 (52%)
[  3]  0.0-60.0 sec  1 datagrams received out-of-order

TCP throughput:

[  5]  0.0-57.0 sec  58.6 MBytes  8.62 Mbits/sec
[  4]  0.0-57.0 sec  58.7 MBytes  8.63 Mbits/sec
[  3]  0.0-57.0 sec  58.6 MBytes  8.63 Mbits/sec
[  9]  0.0-57.0 sec  58.3 MBytes  8.57 Mbits/sec
[  8]  0.0-57.0 sec  58.6 MBytes  8.63 Mbits/sec
[  7]  0.0-57.0 sec  58.2 MBytes  8.57 Mbits/sec
[ 10]  0.0-57.1 sec  57.4 MBytes  8.44 Mbits/sec
[  6]  0.0-57.0 sec  58.5 MBytes  8.61 Mbits/sec
[ 11]  0.0-57.0 sec  57.4 MBytes  8.44 Mbits/sec
[ 12]  0.0-60.0 sec  90.4 MBytes  12.6 Mbits/sec
[SUM]  0.0-60.0 sec    615 MBytes  86.0 Mbits/sec

Answer: It works. UDP throughput is slightly bigger than TCP, probably because of TCP's slow start.
A little bit surprising that one of the TCP flows has way better throughput than the rest.
Maybe it is because that I ran both UDP and TCP for 60 seconds and that TCP connection is the last one being created
As a result, when the UDP session ends, the last TCP session is still active for about 1 second, boosting up the total throughput. Just a guess

7. change NIC delay to 100ms, remove fair queuing, see how it goes:

Result: (I am using Cubic)

[  3]  0.0-60.0 sec    567 MBytes  79.3 Mbits/sec

Now add a 75ms delay variance and see how it goes:
Result: (again, using Cubic)

[  3]  0.0-60.0 sec  24.4 MBytes  3.41 Mbits/sec

Answer: WoW! It surprised me that reordering can affect TCP's performance so much'''

Now tweak the parameters in /proc/sys/net/ipv4/tcp_reordering and see what's the best you can get:
Default value is 3, meaning TCP will retransmit when 3 duplicate ACK is received.
In our case, since no packet is really lost, meaning there is no retransmission needed.
I changed the number to 100, here is the result:

[  3]  0.0-60.0 sec  32.6 MBytes  4.55 Mbits/sec

Well, not a big boost. Let me change it to 100000 and here is the result:

[  3]  0.0-60.3 sec  62.4 MBytes  8.69 Mbits/sec

Well, let me try a HUGE number 1000000000000000 which basically disables TCP's fast-retransmission and see how it goes:

[  3]  0.0-60.3 sec  71.0 MBytes  9.88 Mbits/sec

What about if I am using Reno? Just curious
tcp_reordering = 3, result:

[  3]  0.0-60.1 sec  40.6 MBytes  5.67 Mbits/sec

tcp_reordering = 100000000000000, result:

[  3]  0.0-60.0 sec  71.8 MBytes  10.0 Mbits/sec

Answer: A too high value of tcp_reordering disables TCP's fast retransmission. A too low value will cause unnecessary retransmissions, which is a waste of bandwidth.

8. use Cubic, with SACK on(default), set loss to 10%, see how it goes

Result (repeated for 5 times):

[  3]  0.0-60.9 sec  14.3 MBytes  1.97 Mbits/sec
[  3]  0.0-60.0 sec  15.3 MBytes  2.13 Mbits/sec
[  3]  0.0-60.0 sec  19.3 MBytes  2.70 Mbits/sec
[  3]  0.0-60.2 sec  16.5 MBytes  2.30 Mbits/sec
[  3]  0.0-60.1 sec  19.1 MBytes  2.67 Mbits/sec

Disable tcp_sack and here is the result:

[  3]  0.0-60.0 sec  9.91 MBytes  1.39 Mbits/sec
[  3]  0.0-60.1 sec  11.4 MBytes  1.59 Mbits/sec
[  3]  0.0-60.2 sec  13.4 MBytes  1.87 Mbits/sec
[  3]  0.0-60.0 sec  10.0 MBytes  1.40 Mbits/sec
[  3]  0.0-60.1 sec  10.5 MBytes  1.47 Mbits/sec

Answer: SACK is most beneficial when receiver keeps sending duplicated ACKs back to the sender
So if it is a long delay high bandwidth lossy network, SACK will be very useful.

9. compile and use a customized congestion control mechanism exp and see how it goes:

In the new exp congestion control module, we use:

a slow start exponential factor of 3 instead of 2 in Reno;
ssthresh x 3 / 4 when entering loss recovery instead of ssthresh/2 as in Reno

Comparison: Apparently this will increase the sending rate of TCP during slow start time compared with Reno;
This new mechanism will also cut less slow start threshold when entering loss recovery.
Thus, it is a more aggressive algorithm and should out-perform Reno in one connection facing loss/delay.
However, when number of connections is big, it can be defeated by Reno; simply because its aggressiveness will introduce more loss when network condition is bad

Performance Results:
Under 500ms delay:
Single Reno connection:

[  3]  0.0-60.3 sec    127 MBytes  17.7 Mbits/sec

Single exp connection:

[  3]  0.0-60.3 sec  11.1 MBytes  1.54 Mbits/sec

30 Reno connection:

[ 12]  0.0-51.0 sec  3.06 MBytes    504 Kbits/sec
[ 15]  0.0-51.0 sec  2.52 MBytes    414 Kbits/sec
[ 10]  0.0-51.0 sec  2.64 MBytes    434 Kbits/sec
[  3]  0.0-51.0 sec  3.00 MBytes    493 Kbits/sec
[  4]  0.0-51.1 sec  4.94 MBytes    811 Kbits/sec
[ 13]  0.0-51.1 sec  2.95 MBytes    485 Kbits/sec
[ 14]  0.0-51.2 sec  2.88 MBytes    471 Kbits/sec
[ 16]  0.0-51.2 sec  2.38 MBytes    390 Kbits/sec
[ 11]  0.0-51.3 sec  2.55 MBytes    418 Kbits/sec
[ 18]  0.0-51.3 sec  3.09 MBytes    505 Kbits/sec
[  7]  0.0-51.3 sec  3.92 MBytes    641 Kbits/sec
[  6]  0.0-51.4 sec  5.17 MBytes    845 Kbits/sec
[ 17]  0.0-51.4 sec  2.41 MBytes    393 Kbits/sec
[  9]  0.0-51.9 sec  5.90 MBytes    954 Kbits/sec
[  8]  0.0-52.3 sec  4.63 MBytes    744 Kbits/sec
[  5]  0.0-52.3 sec  4.33 MBytes    694 Kbits/sec
[ 19]  0.0-54.3 sec  9.04 MBytes  1.40 Mbits/sec
[ 23]  0.0-54.4 sec  6.91 MBytes  1.07 Mbits/sec
[ 22]  0.0-54.4 sec  10.8 MBytes  1.67 Mbits/sec
[ 21]  0.0-54.4 sec  6.48 MBytes  1.00 Mbits/sec
[ 24]  0.0-54.4 sec  5.59 MBytes    862 Kbits/sec
[ 25]  0.0-54.5 sec  9.11 MBytes  1.40 Mbits/sec
[ 20]  0.0-54.9 sec  5.80 MBytes    887 Kbits/sec
[ 32]  0.0-60.0 sec  3.20 MBytes    447 Kbits/sec
[ 31]  0.0-60.1 sec  3.12 MBytes    435 Kbits/sec
[ 27]  0.0-60.1 sec  2.52 MBytes    351 Kbits/sec
[ 28]  0.0-60.1 sec  2.86 MBytes    399 Kbits/sec
[ 30]  0.0-60.2 sec  2.01 MBytes    280 Kbits/sec
[ 29]  0.0-60.3 sec  2.62 MBytes    365 Kbits/sec
[ 26]  0.0-60.4 sec  2.92 MBytes    406 Kbits/sec
[SUM]  0.0-60.4 sec    129 MBytes  18.0 Mbits/sec

30 exp connection:

[  5]  0.0-57.1 sec  8.42 MBytes  1.24 Mbits/sec
[ 16]  0.0-57.2 sec  2.67 MBytes    392 Kbits/sec
[ 14]  0.0-57.2 sec  2.63 MBytes    386 Kbits/sec
[ 10]  0.0-57.3 sec  2.60 MBytes    381 Kbits/sec
[  4]  0.0-57.3 sec  7.45 MBytes  1.09 Mbits/sec
[ 11]  0.0-57.3 sec  2.32 MBytes    340 Kbits/sec
[ 17]  0.0-57.3 sec  2.79 MBytes    408 Kbits/sec
[ 12]  0.0-57.3 sec  3.04 MBytes    445 Kbits/sec
[ 15]  0.0-57.4 sec  2.55 MBytes    372 Kbits/sec
[ 13]  0.0-57.4 sec  2.93 MBytes    428 Kbits/sec
[  7]  0.0-57.6 sec  4.09 MBytes    595 Kbits/sec
[  3]  0.0-57.7 sec  9.19 MBytes  1.34 Mbits/sec
[  8]  0.0-57.9 sec  2.77 MBytes    402 Kbits/sec
[  6]  0.0-58.0 sec  28.8 MBytes  4.16 Mbits/sec
[ 18]  0.0-58.7 sec  3.04 MBytes    434 Kbits/sec
[ 31]  0.0-60.0 sec  10.1 MBytes  1.41 Mbits/sec
[ 32]  0.0-60.0 sec  3.24 MBytes    453 Kbits/sec
[ 24]  0.0-60.2 sec  4.41 MBytes    614 Kbits/sec
[ 23]  0.0-60.3 sec  8.37 MBytes  1.16 Mbits/sec
[ 28]  0.0-60.3 sec  3.45 MBytes    480 Kbits/sec
[ 29]  0.0-60.3 sec  2.55 MBytes    356 Kbits/sec
[ 30]  0.0-60.4 sec  3.30 MBytes    459 Kbits/sec
[ 27]  0.0-60.3 sec  2.64 MBytes    367 Kbits/sec
[ 26]  0.0-60.4 sec  2.66 MBytes    370 Kbits/sec
[ 22]  0.0-60.3 sec  3.71 MBytes    516 Kbits/sec
[ 19]  0.0-60.8 sec  3.48 MBytes    480 Kbits/sec
[ 20]  0.0-61.0 sec  3.55 MBytes    489 Kbits/sec
[ 25]  0.0-61.3 sec  4.31 MBytes    590 Kbits/sec
[ 21]  0.0-61.5 sec  5.57 MBytes    759 Kbits/sec
[  9]  0.0-61.9 sec  4.15 MBytes    563 Kbits/sec
[SUM]  0.0-61.9 sec    151 MBytes  20.4 Mbits/sec

Under 5% loss:
Single Reno connection:

[  3]  0.0-60.0 sec  64.0 MBytes  8.95 Mbits/sec

Single exp connection:

[  3]  0.0-60.0 sec    124 MBytes  17.3 Mbits/sec

30 Reno connection:

[ 12]  0.0-51.0 sec  17.8 MBytes  2.92 Mbits/sec
[ 11]  0.0-51.0 sec  18.8 MBytes  3.09 Mbits/sec
[ 10]  0.0-51.0 sec  19.1 MBytes  3.14 Mbits/sec
[  4]  0.0-51.0 sec  16.5 MBytes  2.71 Mbits/sec
[  6]  0.0-51.0 sec  18.6 MBytes  3.06 Mbits/sec
[  8]  0.0-51.0 sec  18.8 MBytes  3.10 Mbits/sec
[  3]  0.0-51.0 sec  19.9 MBytes  3.27 Mbits/sec
[  7]  0.0-51.2 sec  18.3 MBytes  2.99 Mbits/sec
[  9]  0.0-51.3 sec  19.5 MBytes  3.18 Mbits/sec
[ 14]  0.0-54.0 sec  19.3 MBytes  3.00 Mbits/sec
[ 13]  0.0-54.0 sec  19.5 MBytes  3.02 Mbits/sec
[ 17]  0.0-54.0 sec  19.5 MBytes  3.03 Mbits/sec
[ 24]  0.0-54.0 sec  19.8 MBytes  3.07 Mbits/sec
[ 22]  0.0-54.0 sec  19.8 MBytes  3.08 Mbits/sec
[ 23]  0.0-54.0 sec  19.2 MBytes  2.98 Mbits/sec
[ 21]  0.0-54.0 sec  18.8 MBytes  2.91 Mbits/sec
[ 20]  0.0-54.0 sec  19.6 MBytes  3.05 Mbits/sec
[ 19]  0.0-54.1 sec  19.5 MBytes  3.03 Mbits/sec
[ 32]  0.0-54.0 sec  19.5 MBytes  3.03 Mbits/sec
[ 18]  0.0-54.2 sec  19.7 MBytes  3.06 Mbits/sec
[ 15]  0.0-54.2 sec  19.2 MBytes  2.98 Mbits/sec
[  5]  0.0-54.7 sec  19.3 MBytes  2.96 Mbits/sec
[ 27]  0.0-60.0 sec  24.2 MBytes  3.39 Mbits/sec
[ 28]  0.0-60.0 sec  25.7 MBytes  3.59 Mbits/sec
[ 26]  0.0-60.0 sec  25.7 MBytes  3.59 Mbits/sec
[ 25]  0.0-60.1 sec  25.0 MBytes  3.49 Mbits/sec
[ 31]  0.0-60.0 sec  27.3 MBytes  3.82 Mbits/sec
[ 30]  0.0-60.0 sec  24.7 MBytes  3.45 Mbits/sec
[ 16]  0.0-60.0 sec  27.5 MBytes  3.85 Mbits/sec
[ 29]  0.0-60.6 sec  23.4 MBytes  3.24 Mbits/sec
[SUM]  0.0-60.6 sec    623 MBytes  86.3 Mbits/sec

30 exp connection:

[ 20]  0.0-39.0 sec  13.9 MBytes  2.99 Mbits/sec
[ 10]  0.0-39.0 sec  13.8 MBytes  2.96 Mbits/sec
[ 14]  0.0-39.0 sec  13.4 MBytes  2.89 Mbits/sec
[  8]  0.0-39.0 sec  12.7 MBytes  2.73 Mbits/sec
[  6]  0.0-39.0 sec  14.7 MBytes  3.15 Mbits/sec
[  4]  0.0-39.1 sec  13.9 MBytes  2.97 Mbits/sec
[  5]  0.0-39.0 sec  13.0 MBytes  2.79 Mbits/sec
[  3]  0.0-39.0 sec  13.1 MBytes  2.81 Mbits/sec
[ 11]  0.0-39.0 sec  14.4 MBytes  3.09 Mbits/sec
[ 12]  0.0-39.0 sec  13.9 MBytes  2.98 Mbits/sec
[  9]  0.0-39.0 sec  13.7 MBytes  2.95 Mbits/sec
[ 13]  0.0-39.0 sec  14.8 MBytes  3.19 Mbits/sec
[ 19]  0.0-39.0 sec  12.7 MBytes  2.73 Mbits/sec
[ 18]  0.0-39.0 sec  12.9 MBytes  2.76 Mbits/sec
[ 17]  0.0-39.0 sec  13.5 MBytes  2.90 Mbits/sec
[  7]  0.0-39.2 sec  14.3 MBytes  3.07 Mbits/sec
[ 23]  0.0-42.0 sec  16.7 MBytes  3.34 Mbits/sec
[ 22]  0.0-42.0 sec  15.9 MBytes  3.18 Mbits/sec
[ 27]  0.0-42.0 sec  16.9 MBytes  3.38 Mbits/sec
[ 26]  0.0-42.0 sec  16.7 MBytes  3.33 Mbits/sec
[ 25]  0.0-42.0 sec  16.6 MBytes  3.32 Mbits/sec
[ 24]  0.0-42.0 sec  15.9 MBytes  3.18 Mbits/sec
[ 28]  0.0-42.0 sec  16.3 MBytes  3.25 Mbits/sec
[ 21]  0.0-42.0 sec  16.5 MBytes  3.28 Mbits/sec
[ 16]  0.0-42.0 sec  16.5 MBytes  3.29 Mbits/sec
[ 30]  0.0-48.0 sec  29.2 MBytes  5.09 Mbits/sec
[ 29]  0.0-48.0 sec  27.8 MBytes  4.86 Mbits/sec
[ 31]  0.0-48.0 sec  29.8 MBytes  5.21 Mbits/sec
[ 32]  0.0-48.1 sec  25.5 MBytes  4.44 Mbits/sec
[ 15]  0.0-60.0 sec  52.9 MBytes  7.40 Mbits/sec
[SUM]  0.0-60.0 sec    532 MBytes  74.3 Mbits/sec

Under 500ms delay and 5% loss:
Single Reno connection:

[  3]  0.0-61.0 sec    880 KBytes    118 Kbits/sec

Single exp connection:

[  3]  0.0-60.5 sec  1016 KBytes    138 Kbits/sec

30 Reno connection:

[ 16]  0.0-39.2 sec    528 KBytes    110 Kbits/sec
[ 13]  0.0-39.4 sec    600 KBytes    125 Kbits/sec
[ 12]  0.0-39.6 sec    368 KBytes  76.1 Kbits/sec
[ 11]  0.0-39.7 sec    584 KBytes    120 Kbits/sec
[ 14]  0.0-39.8 sec    560 KBytes    115 Kbits/sec
[  8]  0.0-39.8 sec    448 KBytes  92.1 Kbits/sec
[ 10]  0.0-40.0 sec    456 KBytes  93.5 Kbits/sec
[ 15]  0.0-40.0 sec    392 KBytes  80.2 Kbits/sec
[  5]  0.0-40.3 sec    448 KBytes  91.0 Kbits/sec
[  6]  0.0-40.5 sec    400 KBytes  80.9 Kbits/sec
[  3]  0.0-40.5 sec    512 KBytes    103 Kbits/sec
[  4]  0.0-40.9 sec    416 KBytes  83.3 Kbits/sec
[ 17]  0.0-41.3 sec    480 KBytes  95.1 Kbits/sec
[  9]  0.0-41.6 sec    536 KBytes    105 Kbits/sec
[ 18]  0.0-42.5 sec    496 KBytes  95.5 Kbits/sec
[ 25]  0.0-42.6 sec    392 KBytes  75.5 Kbits/sec
[ 29]  0.0-42.6 sec    504 KBytes  96.9 Kbits/sec
[ 24]  0.0-42.7 sec    608 KBytes    117 Kbits/sec
[ 19]  0.0-42.7 sec    520 KBytes  99.8 Kbits/sec
[  7]  0.0-43.1 sec    584 KBytes    111 Kbits/sec
[ 26]  0.0-43.1 sec    464 KBytes  88.1 Kbits/sec
[ 23]  0.0-43.2 sec    512 KBytes  97.1 Kbits/sec
[ 30]  0.0-43.2 sec    376 KBytes  71.3 Kbits/sec
[ 32]  0.0-43.2 sec    576 KBytes    109 Kbits/sec
[ 27]  0.0-43.5 sec    584 KBytes    110 Kbits/sec
[ 31]  0.0-43.6 sec    456 KBytes  85.7 Kbits/sec
[ 28]  0.0-43.8 sec    488 KBytes  91.3 Kbits/sec
[ 21]  0.0-49.4 sec    592 KBytes  98.3 Kbits/sec
[ 22]  0.0-51.6 sec    664 KBytes    105 Kbits/sec
[ 20]  0.0-60.8 sec    696 KBytes  93.8 Kbits/sec
[SUM]  0.0-60.8 sec  14.9 MBytes  2.05 Mbits/sec

30 exp connection:

[  3]  0.0-51.1 sec    824 KBytes    132 Kbits/sec
[ 19]  0.0-51.2 sec    720 KBytes    115 Kbits/sec
[ 14]  0.0-51.2 sec    816 KBytes    130 Kbits/sec
[  5]  0.0-51.3 sec    888 KBytes    142 Kbits/sec
[  8]  0.0-51.3 sec  1008 KBytes    161 Kbits/sec
[ 13]  0.0-51.3 sec    832 KBytes    133 Kbits/sec
[  6]  0.0-51.4 sec    776 KBytes    124 Kbits/sec
[  4]  0.0-51.5 sec    808 KBytes    129 Kbits/sec
[ 18]  0.0-51.5 sec    664 KBytes    106 Kbits/sec
[  9]  0.0-51.7 sec    712 KBytes    113 Kbits/sec
[ 15]  0.0-51.8 sec    944 KBytes    149 Kbits/sec
[  7]  0.0-51.9 sec    600 KBytes  94.7 Kbits/sec
[ 11]  0.0-51.9 sec    776 KBytes    122 Kbits/sec
[ 17]  0.0-52.0 sec    744 KBytes    117 Kbits/sec
[ 16]  0.0-52.0 sec    824 KBytes    130 Kbits/sec
[ 12]  0.0-52.0 sec    656 KBytes    103 Kbits/sec
[ 22]  0.0-54.4 sec  1.08 MBytes    166 Kbits/sec
[ 25]  0.0-54.4 sec    888 KBytes    134 Kbits/sec
[ 26]  0.0-54.6 sec  1.05 MBytes    161 Kbits/sec
[ 21]  0.0-54.7 sec  1.00 MBytes    153 Kbits/sec
[ 30]  0.0-54.8 sec    952 KBytes    142 Kbits/sec
[ 23]  0.0-55.0 sec    960 KBytes    143 Kbits/sec
[ 20]  0.0-55.0 sec  1008 KBytes    150 Kbits/sec
[ 27]  0.0-55.2 sec  1.04 MBytes    158 Kbits/sec
[ 28]  0.0-55.3 sec    872 KBytes    129 Kbits/sec
[ 24]  0.0-55.5 sec    728 KBytes    107 Kbits/sec
[ 29]  0.0-57.1 sec    848 KBytes    122 Kbits/sec
[ 10]  0.0-60.4 sec    952 KBytes    129 Kbits/sec
[ 31]  0.0-60.8 sec    808 KBytes    109 Kbits/sec
[ 32]  0.0-61.7 sec  1.12 MBytes    152 Kbits/sec
[SUM]  0.0-61.7 sec  25.4 MBytes  3.45 Mbits/sec

Next: Teardown Experiment