= IG-MON-4: Infrastructure Device Performance Test = This page captures status for the test case IG-MON-4, which verifies that the rack head node performs well enough to run all the services it needs to run while !OpenFlow and non-!OpenFlow experiments are running. For overall status see the [wiki:GENIRacksHome/InstageniRacks/AcceptanceTestStatus InstaGENI Acceptance Test Status] page. ''Last update: 2013/02/01'' = Test Status = This section captures the status for each step in the acceptance test plan. ||'''Step'''||'''State'''|| ''' Ticket ''' || '''Notes''' || || Step 1 ||[[Color(green,Pass)]] || || || || Step 2 ||[[Color(green,Pass)]]|| || || || Step 3 ||[[Color(green,Pass)]]|| || || [[BR]] || '''State Legend''' || '''Description''' || || [[Color(green,Pass)]] || Test completed and met all criteria || || [[Color(#98FB98,Pass: most criteria)]] || Test completed and met most criteria. Exceptions documented || || [[Color(red,Fail)]] || Test completed and failed to meet criteria. || || [[Color(yellow,Complete)]] || Test completed but will require re-execution due to expected changes || || [[Color(orange,Blocked)]] || Blocked by ticketed issue(s). || || [[Color(#63B8FF,In Progress)]] || Currently under test. || [[BR]] = Test Plan Steps = This test cases sets up several experiments to generate resource usage for both compute and network resources in an InstaGENI Rack. The rack used is the GPO rack and the following experiments are set up before head node device performance is reviewed: - IG-MON-4-exp1: IG GPO non-!OpenFlow experiment with 10 VM nodes, all nodes exchanging traffic. - IG-MON-4-exp2: IG GPO non-!OpenFlow experiment with 1 VM and one dedicated raw-pc, both exchanging traffic. - IG-MON-4-exp3: IG GPO !OpenFlow experiment with 2 nodes in rack exchanging traffic with 2 site GPO !OpenFlow campus resources. - IG-MON-4-exp4: IG GPO !OpenFlow experiment with 4 nodes exchanging traffic within the GPO rack. The setup of the experiments above is not captured in this test case, but the RSpec are available [insert_link_here]. Also traffic levels and types will be captured when this test is run. In addition to the 4 experiment listed above the following experiments were running on the GPO IG rack: [Insert_capture for https://boss.instageni.gpolab.bbn.com/showexp_list.php3] While the 4 experiments involving !OpenFlow and compute slivers are running, execute the steps below. == 1. View !OpenFlow control monitoring at GMOC == Before starting any experiments that request compute or network resources, collected information for baseline performance for the GPO rack for the boss, !FlowVisor and FOAM nodes. Checked existing experiments within the rack, which also shows node allocation by node type: [[Image(IG-MON-4-pre-exp.jpg)]] '''Baseline statistics for Boss Node:''' Top statistics: {{{ load averages: 0.00, 0.01, 0.06 160 processes: 1 running, 158 sleeping, 1 zombie CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 483M Active, 1205M Inact, 221M Wired, 31M Cache, 95M Buf, 60M Free Swap: 2047M Total, 2900K Used, 2044M Free }}} The above shows the boss node CPU Load averages: * 1 minutes: 0.00 * 5 minutes: 0.01 * 15 minutes: 0.06 I/O Statistics for boss node: {{{ tty da4 da1 da2 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 0 3 26.04 0 0.01 20.62 0 0.00 8.01 1 0.01 1 0 1 0 98 }}} '''Baseline statistics for FOAM Node:''' Top statistics: {{{ load average: 0.01, 0.03, 0.05 Tasks: 66 total, 1 running, 63 sleeping, 0 stopped, 2 zombie Cpu(s): 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 756268k total, 533536k used, 222732k free, 134708k buffers Swap: 794620k total, 5428k used, 789192k free, 280268k cached }}} The above shows the FOAM node CPU Load averages: * 1 minutes: 0.01 * 5 minutes: 0.03 * 15 minutes: 0.05 I/O Statistics for FOAM node: {{{ Linux 3.2.0-34-generic (foam) 02/06/2013 _x86_64_ (1 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 0.71 0.00 0.37 0.00 0.00 98.91 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn xvda 1.92 0.09 14.80 411133 70367069 dm-0 2.69 0.09 14.80 406773 70360928 dm-1 0.00 0.00 0.00 2356 6092 }}} Checked for existing FOAM sliver (also shown in the EID column of the "Active Nodes" above: {{{ lnevers@foam:~$ foamctl geni:list-slivers --passwd-file=/etc/foam.passwd |grep sliver_urn "sliver_urn": "urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs15:8a0abd6f-0f5a-469f-91d2-c7f990b8494e", "sliver_urn": "urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs16:a92990b6-1ede-4dd7-b6f6-7b4a4bd36fd7", "sliver_urn": "urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tuptymon:b7850c93-110f-4e63-a121-26f3449dac44", }}} '''Baseline statistics for !FlowVisor Node:''' Top statistics for !FlowVisor node: {{{ load average: 0.00, 0.01, 0.05 Tasks: 67 total, 1 running, 66 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4031252k total, 965296k used, 3065956k free, 92468k buffers Swap: 4202492k total, 0k used, 4202492k free, 340944k cached }}} The above shows the !FlowVisor node CPU Load averages: * 1 minutes: 0.00 * 5 minutes: 0.01 * 15 minutes: 0.05 I/O Statistics for !FlowVisor node: {{{ Linux 3.2.0-34-generic (flowvisor) 02/06/2013 _x86_64_ (2 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 0.09 0.00 0.07 0.00 0.00 99.83 Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn xvda 0.30 0.06 1.98 268395 9372505 dm-0 0.43 0.06 1.98 265253 9372452 dm-1 0.00 0.00 0.00 1148 0 }}} Checked for existing !FlowSpace rules and found 6: {{{ lnevers@flowvisor:~$ fvctl --passwd-file=/etc/flowvisor.passwd listFlowSpace Got reply: rule 0: FlowEntry[dpid=[06:d6:84:34:97:c6:c9:00],ruleMatch=[OFMatch[dl_type=0x800,nw_dst=10.42.15.0/24,nw_src=10.42.15.0/24]],actionsList=[Slice:8a0abd6f-0f5a-469f-91d2-c7f990b8494e=4],id=[13],priority=[2000],] rule 1: FlowEntry[dpid=[06:d6:84:34:97:c6:c9:00],ruleMatch=[OFMatch[dl_type=0x806,nw_dst=10.42.15.0/24,nw_src=10.42.15.0/24]],actionsList=[Slice:8a0abd6f-0f5a-469f-91d2-c7f990b8494e=4],id=[14],priority=[2000],] rule 2: FlowEntry[dpid=[06:d6:84:34:97:c6:c9:00],ruleMatch=[OFMatch[dl_type=0x800,nw_dst=10.42.16.0/24,nw_src=10.42.16.0/24]],actionsList=[Slice:a92990b6-1ede-4dd7-b6f6-7b4a4bd36fd7=4],id=[15],priority=[2000],] rule 3: FlowEntry[dpid=[06:d6:84:34:97:c6:c9:00],ruleMatch=[OFMatch[dl_type=0x806,nw_dst=10.42.16.0/24,nw_src=10.42.16.0/24]],actionsList=[Slice:a92990b6-1ede-4dd7-b6f6-7b4a4bd36fd7=4],id=[16],priority=[2000],] rule 4: FlowEntry[dpid=[06:d6:84:34:97:c6:c9:00],ruleMatch=[OFMatch[dl_type=0x800,nw_dst=10.50.0.0/16,nw_src=10.50.0.0/16]],actionsList=[Slice:b7850c93-110f-4e63-a121-26f3449dac44=4],id=[21],priority=[2000],] rule 5: FlowEntry[dpid=[06:d6:84:34:97:c6:c9:00],ruleMatch=[OFMatch[dl_type=0x806,nw_dst=10.50.0.0/16,nw_src=10.50.0.0/16]],actionsList=[Slice:b7850c93-110f-4e63-a121-26f3449dac44=4],id=[22],priority=[2000],] }}} '''Baseline statistics for OPS/users Node:''' Top statistics: {{{ load averages: 0.00, 0.00, 0.00 97 processes: 1 running, 96 sleeping CPU: 0.0% user, 0.0% nice, 1.2% system, 0.0% interrupt, 98.8% idle Mem: 152M Active, 873M Inact, 157M Wired, 6944K Cache, 95M Buf, 811M Free Swap: 2047M Total, 712K Used, 2047M Free }}} The above shows the !FlowVisor node CPU Load averages: * 1 minutes: 0.00 * 5 minutes: 0.00 * 15 minutes: 0.00 I/O Statistics for OPS node: {{{ tty da3 da0 da4 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 0 1 19.46 0 0.00 11.03 0 0.01 11.33 0 0.00 0 0 0 0 100 }}} Set up the first experiment IG-MON-4-exp1 at the GPO InstaGENI. Experiment included 10 VM nodes exchanging traffic, without !OpenFlow. Verified all nodes in the sliver were ready, and checked statistics for boss and ops nodes: '''Boss after EG-MON-4-exp1 sliver creation:''' {{{ load averages: 0.28, 0.24, 0.11 141 processes: 1 running, 139 sleeping, 1 zombie CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 505M Active, 1068M Inact, 198M Wired, 46M Cache, 96M Buf, 183M Free Swap: 2047M Total, 4164K Used, 2043M Free }}} '''Ops after EG-MON-4 sliver creation:''' {{{ load averages: 0.00, 0.00, 0.00 99 processes: 1 running, 98 sleeping CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 155M Active, 874M Inact, 158M Wired, 6944K Cache, 95M Buf, 807M Free Swap: 2047M Total, 712K Used, 2047M Free }}} '''PC2 node statistics''' The node pc2 was used to allocate the 10 VMs, following are the top statistics: {{{ load average: 0.19, 2.30, 3.23 Tasks: 643 total, 1 running, 642 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 49311300k total, 15130436k used, 34180864k free, 902764k buffers Swap: 1050168k total, 0k used, 1050168k free, 12125232k cached }}} Started a ping on each of the ten nodes IN IG-MON-4-EXP1 to another node in the sliver using 64 bytes packets with an interval of 3 packet/sec. All nodes allocated were on pc2. After 10 minutes of traffic checked statistics: [[BR]] '''EG-MON-4-exp1 statistics for PC2 Node:''' {{{ load average: 0.22, 0.64, 2.06 Tasks: 683 total, 1 running, 682 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 49311300k total, 15160596k used, 34150704k free, 904076k buffers Swap: 1050168k total, 0k used, 1050168k free, 12125384k cached }}} The above shows CPU Load averages: * 1 minutes: 0.22 * 5 minutes: 0.64 * 15 minutes: 2.06 '''EG-MON-4-exp1 statistics for Boss Node:''' {{{ oad averages: 0.04, 0.06, 0.07 146 processes: 1 running, 144 sleeping, 1 zombie CPU: 0.4% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.2% idle Mem: 410M Active, 1182M Inact, 198M Wired, 46M Cache, 95M Buf, 164M Free Swap: 2047M Total, 4136K Used, 2043M Free }}} The above shows CPU Load averages: * 1 minutes: 0.04 * 5 minutes: 0.06 * 15 minutes: 0.07 '''EG-MON-4-exp1 statistics for FOAM Node:''' {{{ load average: 0.00, 0.01, 0.05 Tasks: 66 total, 1 running, 63 sleeping, 0 stopped, 2 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 756268k total, 533072k used, 223196k free, 134744k buffers Swap: 794620k total, 5428k used, 789192k free, 280116k cached }}} The above shows CPU Load averages: * 1 minutes: 0.00 * 5 minutes: 0.01 * 15 minutes: 0.05 '''EG-MON-4-exp1 statistics for !FlowVisor Node:''' {{{ load average: 0.00, 0.01, 0.05 Tasks: 67 total, 1 running, 66 sleeping, 0 stopped, 0 zombie Cpu(s): 0.2%us, 0.0%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4031252k total, 965832k used, 3065420k free, 92468k buffers Swap: 4202492k total, 0k used, 4202492k free, 340948k cached }}} The above shows CPU Load averages: * 1 minutes: 0.00 * 5 minutes: 0.01 * 15 minutes: 0.05 '''EG-MON-4-exp1 statistics for OPS Node:''' {{{ load averages: 0.01, 0.03, 0.00 99 processes: 1 running, 98 sleeping CPU: 0.4% user, 0.0% nice, 0.0% system, 0.0% interrupt, 99.6% idle Mem: 155M Active, 874M Inact, 158M Wired, 6944K Cache, 96M Buf, 807M Free Swap: 2047M Total, 712K Used, 2047M Free }}} The above shows CPU Load averages: * 1 minutes: 0.01 * 5 minutes: 0.03 * 15 minutes: 0.00 Leave first experiment IG-MON-4-exp1 running and start experiment IG-MON-4-exp2. The experiment IG-MON-4-exp2 includes 1 PC and 1 VM. Once the sliver was ready started running iperf traffic for 15 minutes. Iperf results: {{{ [lnevers@VM ~]$ /usr/bin/iperf -c 192.168.1.2 -t 900 ------------------------------------------------------------ Client connecting to 192.168.1.2, TCP port 5001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.1.1 port 44231 connected with 192.168.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-900.0 sec 10.5 GBytes 100 Mbits/sec }}} At iperf completion checked statistics: '''EG-MON-4-exp2 statistics for VM Node (pc4):" {{{ load average: 0.00, 0.00, 0.01 Tasks: 429 total, 1 running, 428 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 49311300k total, 2813428k used, 46497872k free, 91040k buffers Swap: 1050168k total, 0k used, 1050168k free, 2222324k cached }}} '''EG-MON-4-exp2 statistics for Dedicated Node (pc5):" {{{ load average: 0.00, 0.01, 0.04 Tasks: 182 total, 1 running, 181 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.1%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 49714548k total, 290696k used, 49423852k free, 9768k buffers Swap: 1050172k total, 0k used, 1050172k free, 182852k cached }}} '''EG-MON-4-exp2 statistics for Boss Node:''' {{{ load averages: 0.38, 0.14, 0.09 157 processes: 1 running, 155 sleeping, 1 zombie CPU: 0.4% user, 0.0% nice, 0.0% system, 0.0% interrupt, 99.6% idle Mem: 407M Active, 1205M Inact, 195M Wired, 44M Cache, 95M Buf, 149M Free Swap: 2047M Total, 4128K Used, 2043M Free }}} '''EG-MON-4-exp2 statistics for FOAM Node:''' {{{ load average: 0.00, 0.01, 0.05 Tasks: 66 total, 1 running, 63 sleeping, 0 stopped, 2 zombie Cpu(s): 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 756268k total, 533908k used, 222360k free, 134808k buffers Swap: 794620k total, 5428k used, 789192k free, 280904k cached }}} '''EG-MON-4-exp2 statistics for !FlowVisor Node:''' {{{ load average: 0.00, 0.01, 0.05 Tasks: 67 total, 1 running, 66 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.2%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4031252k total, 965832k used, 3065420k free, 92468k buffers Swap: 4202492k total, 0k used, 4202492k free, 340948k cached }}} '''EG-MON-4-exp2 statistics for OPS Node:''' {{{ load average: 0.00, 0.00, 0.18 Tasks: 683 total, 1 running, 682 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 49311300k total, 15166288k used, 34145012k free, 910020k buffers Swap: 1050168k total, 0k used, 1050168k free, 12125464k cached }}} Ping statistic for PC to VM: - without iperf: "rtt min/avg/max/mdev = 0.083/0.168/0.193/0.032 ms" - with iperf: "rtt min/avg/max/mdev = 52.344/53.343/54.618/0.800 ms" Restarted iperf to run for the remainder of this test. == 2. View VLAN 1750 data plane monitoring == In this section we verify the VLAN 1750 data plane monitoring which pings the rack's interface on VLAN 1750, and verify that packets are not being dropped. Leave first experiment IG-MON-4-exp2 running and add the !OpenFlow experiment IG-MON-4-exp3. The experiment IG-MON-4-exp3 includes 2 nodes in the rack exchanging traffic with 2 campus nodes also connected on the !OpenFlow network. Two nodes were allocated on pc1: '''EG-MON-4-exp3 statistics for PC1:''' {{{ load average: 0.00, 0.30, 0.49 Tasks: 543 total, 1 running, 542 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 49311300k total, 9917688k used, 39393612k free, 751168k buffers Swap: 1050168k total, 0k used, 1050168k free, 7748796k cached }}} '''EG-MON-4-exp3 statistics for Boss Node:''' {{{ load averages: 0.05, 0.09, 0.08 156 processes: 1 running, 154 sleeping, 1 zombie CPU: 1.2% user, 0.0% nice, 3.5% system, 0.0% interrupt, 95.3% idle Mem: 403M Active, 1230M Inact, 187M Wired, 44M Cache, 94M Buf, 135M Free Swap: 2047M Total, 4120K Used, 2043M Free }}} '''EG-MON-4-exp3 statistics for FOAM Node:''' {{{ load average: 0.15, 0.08, 0.06 Tasks: 67 total, 1 running, 63 sleeping, 0 stopped, 3 zombie Cpu(s): 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 756268k total, 534816k used, 221452k free, 134888k buffers Swap: 794620k total, 5428k used, 789192k free, 281560k cached }}} '''EG-MON-4-exp3 statistics for !FlowVisor Node:''' {{{ load average: 0.00, 0.01, 0.05 Tasks: 67 total, 1 running, 66 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4031252k total, 965816k used, 3065436k free, 92468k buffers Swap: 4202492k total, 0k used, 4202492k free, 340976k cached }}} '''EG-MON-4-exp3 statistics for OPS Node:''' {{{ load averages: 0.00, 0.00, 0.00 105 processes: 1 running, 104 sleeping CPU: 0.0% user, 0.0% nice, 1.6% system, 0.0% interrupt, 98.4% idle Mem: 162M Active, 876M Inact, 159M Wired, 6932K Cache, 96M Buf, 797M Free Swap: 2047M Total, 712K Used, 2047M Free }}} Leave !OpenFlow experiment IG-MON-4-exp3 running and add the other !OpenFlow experiment IG-MON-4-exp4. Experiment includes 4 nodes exchanging pings withing the GPO Rack, all nodes in experiment were assigned on pc1. '''EG-MON-4-exp4 statistics for Boss Node:''' {{{ load averages: 0.03, 0.07, 0.07 156 processes: 1 running, 154 sleeping, 1 zombie CPU: 0.4% user, 0.0% nice, 0.0% system, 0.0% interrupt, 99.6% idle Mem: 403M Active, 1234M Inact, 187M Wired, 44M Cache, 96M Buf, 131M Free Swap: 2047M Total, 4104K Used, 2043M Free }}} '''EG-MON-4-exp4 statistics for FOAM Node:''' {{{ load average: 0.00, 0.04, 0.05 Tasks: 67 total, 1 running, 63 sleeping, 0 stopped, 3 zombie Cpu(s): 0.3%us, 0.0%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 756268k total, 535420k used, 220848k free, 134908k buffers Swap: 794620k total, 5428k used, 789192k free, 281848k cached }}} '''EG-MON-4-exp4 statistics for !FlowVisor Node:''' {{{ load average: 0.00, 0.01, 0.05 Tasks: 67 total, 1 running, 66 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.2%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4031252k total, 966080k used, 3065172k free, 92468k buffers Swap: 4202492k total, 0k used, 4202492k free, 340992k cached }}} '''EG-MON-4-exp4 statistics for OPS Node:''' {{{ load averages: 0.00, 0.00, 0.00 105 processes: 1 running, 104 sleeping CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 162M Active, 876M Inact, 159M Wired, 6932K Cache, 96M Buf, 796M Free Swap: 2047M Total, 712K Used, 2047M Free }}} '''Statistics for pc1 Node:''' {{{ load average: 0.01, 0.22, 0.37 Tasks: 595 total, 1 running, 594 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 49311300k total, 11733656k used, 37577644k free, 818580k buffers Swap: 1050168k total, 0k used, 1050168k free, 9252840k cached }}} With all traffic from the above 4 experiments running, captured ping round trip time from boss node to each of the other nodes in the rack: || Dest || min rtt (ms)|| avg rtt (ms) || max (ms) || stddev (ms)|| || ops || 0.166 ||0.202 ||0.236 || 0.023 || || pc1 || 0.222 ||0.257 ||0.354 || 0.039 || || pc2 || 0.221 ||0.241 ||0.256 || 0.010 || || pc3 || 0.212 ||0.228 ||0.267 || 0.014 || || pc4 || 0.207 ||0.222 ||0.252 || 0.012 || || pc5 || 0.217 ||0.232 ||0.248 || 0.012 || || foam || 0.154 ||2.193 ||20.119 || 5.976 || || foam-priv ||0.187 ||1.546 ||9.995 || 3.005 || || flowvisor || 0.167 ||0.208 ||0.385 || 0.063 || || flowvisor-priv ||0.185||0.223 ||0.381 || 0.057 || Ping round trip time were collected after the 4 experiments (and traffic) were delete to provide comparison. Again captured from the boss to each of the other nodes in the rack: || Dest || min rtt (ms)|| avg rtt (ms) || max (ms) || stddev (ms)|| || ops || 0.168 || 1.190 || 9.990 || 2.933 || || pc1 || 0.223 || 0.248 || 0.271 || 0.016 || || pc2 || 0.239 || 0.259 || 0.307 || 0.018 || || pc3 || 0.222 || 0.240 || 0.268 || 0.015 || || pc4 || 0.235 || 0.247 || 0.266 || 0.009 || || pc5 || 0.175 || 0.246 || 0.418 || 0.083 || || foam || 0.139 || 0.172 || 0.211 || 0.020 || || foam-priv || 0.212 || 0.236 || 0.272 || 0.018 || || flowvisor || 0.193 || 0.208 || 0.221 || 0.009 || || flowvisor-priv|| 0.192 || 0.212 || 0.228 || 0.013 || After the 4 experiment were deleted also collected top statistics: '''Boss Node Statistics:''' {{{ load averages: 0.00, 0.07, 0.11 160 processes: 2 running, 157 sleeping, 1 zombie CPU: 0.4% user, 0.0% nice, 0.0% system, 0.0% interrupt, 99.6% idle Mem: 484M Active, 1200M Inact, 223M Wired, 32M Cache, 95M Buf, 61M Free Swap: 2047M Total, 2900K Used, 2044M Free }}} '''OPS Node Statistics:''' {{{ load averages: 0.00, 0.03, 0.00 99 processes: 1 running, 98 sleeping CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle Mem: 155M Active, 911M Inact, 163M Wired, 6928K Cache, 95M Buf, 764M Free Swap: 2047M Total, 712K Used, 2047M Free }}} '''FOAM Node Statistics:''' {{{ load average: 0.00, 0.01, 0.05 Tasks: 67 total, 1 running, 63 sleeping, 0 stopped, 3 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 756268k total, 535296k used, 220972k free, 134996k buffers Swap: 794620k total, 5428k used, 789192k free, 281964k cached }}} '''!FlowVisor Node Statistics:''' {{{ load average: 0.00, 0.01, 0.05 Tasks: 67 total, 1 running, 66 sleeping, 0 stopped, 0 zombie Cpu(s): 0.2%us, 0.0%sy, 0.0%ni, 99.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 4031252k total, 966080k used, 3065172k free, 92468k buffers Swap: 4202492k total, 0k used, 4202492k free, 340992k cached }}} '''PC1 Node Statistics:''' {{{ load average: 0.06, 0.02, 0.03 Tasks: 491 total, 1 running, 490 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 49311300k total, 8113708k used, 41197592k free, 686612k buffers Swap: 1050168k total, 0k used, 1050168k free, 6244968k cached }}} '''PC2 Node Statistics:''' {{{ load average: 0.00, 0.04, 0.09 Tasks: 463 total, 1 running, 462 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.9%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 49311300k total, 7008116k used, 42303184k free, 603528k buffers Swap: 1050168k total, 0k used, 1050168k free, 5357824k cached }}} '''PC4 Node Statistics:''' {{{ load average: 0.06, 0.02, 0.00 Tasks: 403 total, 1 running, 402 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 5.9%sy, 0.0%ni, 94.1%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 49311300k total, 1781088k used, 47530212k free, 57424k buffers Swap: 1050168k total, 0k used, 1050168k free, 1353432k cached }}} == 3. Verify CPU idle percentage on the boss and FOAM/FV hosts are nonzero == Reviewed all CPU statistics gathered in each of the previous steps and found that all "% idle" values are nonzero.