Changes between Version 27 and Version 28 of GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-MON-4


Ignore:
Timestamp:
02/06/13 15:11:28 (11 years ago)
Author:
lnevers@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-MON-4

    v27 v28  
    5252Top statistics:
    5353{{{
    54 last pid:  9865;  load averages:  0.07,  0.03,  0.00               
    55 145 processes: 3 running, 141 sleeping, 1 zombie
    56 CPU:  0.8% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.2% idle
    57 Mem: 488M Active, 1189M Inact, 189M Wired, 31M Cache, 94M Buf, 103M Free
    58 Swap: 2047M Total, 2628K Used, 2045M Free
    59 }}}
     54load averages:  0.00,  0.00,  0.00   
     55141 processes: 1 running, 139 sleeping, 1 zombie
     56CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
     57Mem: 405M Active, 1303M Inact, 202M Wired, 10M Cache, 94M Buf, 80M Free
     58Swap: 2047M Total, 3820K Used, 2043M Free
     59
    6060The above shows the boss node CPU Load averages:
    61   * 1 minutes:  0.07
    62   * 5 minutes:  0.03
     61  * 1 minutes:  0.00
     62  * 5 minutes:  0.00
    6363  * 15 minutes:  0.00
    6464
    … …  
    6767       tty             da4              da1              da2             cpu
    6868 tin  tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
    69    0     3 25.98   0  0.01  20.73   0  0.00   7.90   1  0.01   1  0  1  0 98
     69   0     3 26.04   0  0.01  20.62   0  0.00   8.01   1  0.01   1  0  1  0 98
    7070}}}
    7171
    … …  
    7474Top statistics:
    7575{{{
    76 load average: 0.00, 0.01, 0.05
    77 Tasks:  67 total,   1 running,  64 sleeping,   0 stopped,   2 zombie
    78 Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
    79 Mem:    756268k total,   607344k used,   148924k free,   149308k buffers
    80 Swap:   794620k total,     4444k used,   790176k free,   334760k cached
     76load average: 0.01, 0.03, 0.05
     77Tasks:  66 total,   1 running,  63 sleeping,   0 stopped,   2 zombie
     78Cpu(s):  0.0%us,  0.3%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
     79Mem:    756268k total,   533536k used,   222732k free,   134708k buffers
     80Swap:   794620k total,     5428k used,   789192k free,   280268k cached
     81
    8182}}}
    8283The above shows the FOAM node CPU Load averages:
    83   * 1 minutes:  0.00
    84   * 5 minutes:  0.01
     84  * 1 minutes:  0.01
     85  * 5 minutes:  0.03
    8586  * 15 minutes:  0.05
    8687
    8788I/O Statistics for FOAM node:
    8889{{{
    89 Linux 3.2.0-34-generic (foam)   02/01/2013      _x86_64_        (1 CPU)
     90Linux 3.2.0-34-generic (foam)   02/06/2013      _x86_64_        (1 CPU)
    9091avg-cpu:  %user   %nice %system %iowait  %steal   %idle
    91            0.64    0.00    0.35    0.00    0.00   99.00
     92           0.71    0.00    0.37    0.00    0.00   98.91
    9293Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
    93 xvda              1.80         0.09        13.93     371249   60046465
    94 dm-0              2.52         0.09        13.93     366921   60041316
    95 dm-1              0.00         0.00         0.00       2324       5104
     94xvda              1.92         0.09        14.80     411133   70367069
     95dm-0              2.69         0.09        14.80     406773   70360928
     96dm-1              0.00         0.00         0.00       2356       6092
     97
    9698}}}
    9799
    … …  
    102104   "sliver_urn": "urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs16:a92990b6-1ede-4dd7-b6f6-7b4a4bd36fd7",
    103105   "sliver_urn": "urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tuptymon:b7850c93-110f-4e63-a121-26f3449dac44",
    104    "sliver_urn": "urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs17:fd82bb82-eef3-407d-b092-c1393773791c",
    105106}}}
    106107
    … …  
    110111{{{
    111112load average: 0.00, 0.01, 0.05
    112 Tasks:  68 total,   1 running,  67 sleeping,   0 stopped,   0 zombie
    113 Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
    114 Mem:   4031252k total,   967100k used,  3064152k free,    92444k buffers
    115 Swap:  4202492k total,        0k used,  4202492k free,   366332k cached
     113Tasks:  67 total,   1 running,  66 sleeping,   0 stopped,   0 zombie
     114Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
     115Mem:   4031252k total,   965296k used,  3065956k free,    92468k buffers
     116Swap:  4202492k total,        0k used,  4202492k free,   340944k cached
    116117}}}
    117118The above shows the !FlowVisor node CPU Load averages:
    … …  
    122123I/O Statistics for !FlowVisor node:
    123124{{{
    124 Linux 3.2.0-34-generic (flowvisor)      02/01/2013      _x86_64_        (2 CPU)
    125 
     125Linux 3.2.0-34-generic (flowvisor)      02/06/2013      _x86_64_        (2 CPU)
    126126avg-cpu:  %user   %nice %system %iowait  %steal   %idle
    127            0.09    0.00    0.07    0.00    0.00   99.84
     127           0.09    0.00    0.07    0.00    0.00   99.83
    128128Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
    129 xvda              0.31         0.06         2.01     268011    8634017
    130 dm-0              0.44         0.06         2.01     264869    8633968
     129xvda              0.30         0.06         1.98     268395    9372505
     130dm-0              0.43         0.06         1.98     265253    9372452
    131131dm-1              0.00         0.00         0.00       1148          0
    132132}}}
    133133
    134 Checked for existing !FlowSpace rules:
     134Checked for existing !FlowSpace rules and found 6:
    135135{{{
    136136lnevers@flowvisor:~$ fvctl --passwd-file=/etc/flowvisor.passwd listFlowSpace
    … …  
    142142rule 4: FlowEntry[dpid=[06:d6:84:34:97:c6:c9:00],ruleMatch=[OFMatch[dl_type=0x800,nw_dst=10.50.0.0/16,nw_src=10.50.0.0/16]],actionsList=[Slice:b7850c93-110f-4e63-a121-26f3449dac44=4],id=[21],priority=[2000],]
    143143rule 5: FlowEntry[dpid=[06:d6:84:34:97:c6:c9:00],ruleMatch=[OFMatch[dl_type=0x806,nw_dst=10.50.0.0/16,nw_src=10.50.0.0/16]],actionsList=[Slice:b7850c93-110f-4e63-a121-26f3449dac44=4],id=[22],priority=[2000],]
    144 rule 6: FlowEntry[dpid=[06:d6:84:34:97:c6:c9:00],ruleMatch=[OFMatch[dl_type=0x800,nw_dst=10.42.17.0/24,nw_src=10.42.17.0/24]],actionsList=[Slice:fd82bb82-eef3-407d-b092-c1393773791c=4],id=[50],priority=[2000],]
    145 rule 7: FlowEntry[dpid=[06:d6:84:34:97:c6:c9:00],ruleMatch=[OFMatch[dl_type=0x806,nw_dst=10.42.17.0/24,nw_src=10.42.17.0/24]],actionsList=[Slice:fd82bb82-eef3-407d-b092-c1393773791c=4],id=[51],priority=[2000],]
    146 lnevers@flowvisor:~$
    147144}}}
    148145
    … …  
    151148Top statistics:
    152149{{{
    153 load averages:  0.63,  0.16,  0.05   
     150load averages:  0.00,  0.00,  0.00 
     15197 processes:  1 running, 96 sleeping
     152CPU:  0.0% user,  0.0% nice,  1.2% system,  0.0% interrupt, 98.8% idle
     153Mem: 152M Active, 873M Inact, 157M Wired, 6944K Cache, 95M Buf, 811M Free
     154Swap: 2047M Total, 712K Used, 2047M Free
     155
     156}}}
     157The above shows the !FlowVisor node CPU Load averages:
     158  * 1 minutes:  0.00
     159  * 5 minutes:  0.00
     160  * 15 minutes:  0.00
     161
     162I/O Statistics for OPS node:
     163{{{
     164       tty             da3              da0              da4             cpu
     165 tin  tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
     166   0     1 19.46   0  0.00  11.03   0  0.01  11.33   0  0.00   0  0  0  0 100
     167}}}
     168
     169Set up the first experiment IG-MON-4-exp1 at the GPO InstaGENI. Experiment included 10 VM nodes exchanging traffic, without !OpenFlow. Verified all nodes in the sliver were ready, and checked statistics for boss and ops nodes:
     170
     171'''Boss after EG-MON-4-exp1 sliver creation:'''
     172{{{
     173load averages:  0.28,  0.24,  0.11   
     174141 processes: 1 running, 139 sleeping, 1 zombie
     175CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
     176Mem: 505M Active, 1068M Inact, 198M Wired, 46M Cache, 96M Buf, 183M Free
     177Swap: 2047M Total, 4164K Used, 2043M Free
     178}}}
     179'''Ops after EG-MON-4 sliver creation:'''
     180{{{
     181load averages:  0.00,  0.00,  0.00   
    15418299 processes:  1 running, 98 sleeping
    155183CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
    156 Mem: 158M Active, 851M Inact, 163M Wired, 7660K Cache, 95M Buf, 820M Free
    157 Swap: 2047M Total, 852K Used, 2046M Free
    158 }}}
    159 The above shows the !FlowVisor node CPU Load averages:
    160   * 1 minutes:  0.63
    161   * 5 minutes:  0.16
    162   * 15 minutes:  0.05
    163 
    164 I/O Statistics for OPS node:
    165 {{{
    166        tty             da3              da0              da4             cpu
    167  tin  tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
    168    0     1 20.01   0  0.00  11.03   0  0.01  11.19   0  0.00   0  0  0  0 100
    169 }}}
    170 
    171 Set up the first experiment IG-MON-4-exp1 at the GPO InstaGENI. Experiment included 10 VM nodes exchanging traffic, without !OpenFlow. Verified all nodes in the sliver were ready, and checked statistics for boss and ops nodes:
    172 
    173 '''Boss after EG-MON-4-exp1 sliver creation:'''
    174 {{{
    175 load averages:  0.05,  0.11,  0.08             
    176 145 processes: 1 running, 143 sleeping, 1 zombie
    177 CPU:  0.0% user,  0.0% nice,  0.8% system,  0.0% interrupt, 99.2% idle
    178 Mem: 432M Active, 1171M Inact, 188M Wired, 27M Cache, 93M Buf, 183M Free
    179 Swap: 2047M Total, 2628K Used, 2045M Free
    180 }}}
    181 '''Ops after EG-MON-4 sliver creation:'''
    182 {{{
    183 load averages:  0.00,  0.02,  0.00 
    184 101 processes: 2 running, 99 sleeping
    185 CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
    186 Mem: 160M Active, 852M Inact, 164M Wired, 7660K Cache, 95M Buf, 817M Free
    187 Swap: 2047M Total, 852K Used, 2046M Free
     184Mem: 155M Active, 874M Inact, 158M Wired, 6944K Cache, 95M Buf, 807M Free
     185Swap: 2047M Total, 712K Used, 2047M Free
     186}}}
     187
     188'''PC2 node statistics'''
     189The node pc2 was used to allocate the 10 VMs, following are the top statistics:
     190{{{
     191load average: 0.19, 2.30, 3.23
     192Tasks: 643 total,   1 running, 642 sleeping,   0 stopped,   0 zombie
     193Cpu(s):  0.0%us,  0.1%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
     194Mem:  49311300k total, 15130436k used, 34180864k free,   902764k buffers
     195Swap:  1050168k total,        0k used,  1050168k free, 12125232k cached
    188196}}}
    189197
    190198Started a ping on each of the ten nodes IN IG-MON-4-EXP1 to another node in the sliver using 64 bytes packets with an interval of 3 packet/sec. All nodes allocated were on pc2.  After 10 minutes of traffic checked statistics:
    191 
    192199 
    193200'''EG-MON-4-exp1 statistics for PC2 Node:'''
    194201{{{
    195 load average: 0.00, 0.45, 1.87
    196 Tasks: 650 total,   1 running, 649 sleeping,   0 stopped,   0 zombie
    197 Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
    198 Mem:  49311300k total, 15063252k used, 34248048k free,   904700k buffers
    199 Swap:  1050168k total,        0k used,  1050168k free, 12100764k cached
     202load average: 0.22, 0.64, 2.06
     203Tasks: 683 total,   1 running, 682 sleeping,   0 stopped,   0 zombie
     204Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.9%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
     205Mem:  49311300k total, 15160596k used, 34150704k free,   904076k buffers
     206Swap:  1050168k total,        0k used,  1050168k free, 12125384k cached
     207}}}
     208The above shows CPU Load averages:
     209  * 1 minutes:  0.22
     210  * 5 minutes:  0.64
     211  * 15 minutes:  2.06
     212
     213
     214'''EG-MON-4-exp1 statistics for Boss Node:'''
     215{{{
     216oad averages:  0.04,  0.06,  0.07 
     217146 processes: 1 running, 144 sleeping, 1 zombie
     218CPU:  0.4% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.2% idle
     219Mem: 410M Active, 1182M Inact, 198M Wired, 46M Cache, 95M Buf, 164M Free
     220Swap: 2047M Total, 4136K Used, 2043M Free
     221
     222
     223}}}
     224The above shows CPU Load averages:
     225  * 1 minutes:  0.04
     226  * 5 minutes:  0.06
     227  * 15 minutes: 0.07
     228
     229'''EG-MON-4-exp1 statistics for FOAM Node:'''
     230{{{
     231load average: 0.00, 0.01, 0.05
     232Tasks:  66 total,   1 running,  63 sleeping,   0 stopped,   2 zombie
     233Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
     234Mem:    756268k total,   533072k used,   223196k free,   134744k buffers
     235Swap:   794620k total,     5428k used,   789192k free,   280116k cached
    200236}}}
    201237The above shows CPU Load averages:
    202238  * 1 minutes:  0.00
    203   * 5 minutes:  0.45
    204   * 15 minutes:  1.87
    205 
    206 
    207 '''EG-MON-4-exp1 statistics for Boss Node:'''
    208 {{{
    209 load averages:  0.01,  0.05,  0.03             
    210 146 processes: 1 running, 144 sleeping, 1 zombie
    211 CPU:  0.8% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.2% idle
    212 Mem: 396M Active, 1211M Inact, 189M Wired, 27M Cache, 94M Buf, 177M Free
    213 Swap: 2047M Total, 2628K Used, 2045M Free
    214 
    215 }}}
    216 The above shows CPU Load averages:
    217   * 1 minutes:  0.01
    218   * 5 minutes:  0.05
    219   * 15 minutes: 1.87
    220 
    221 '''EG-MON-4-exp1 statistics for FOAM Node:'''
    222 {{{
    223 load average: 0.08, 0.06, 0.05
    224 Tasks:  67 total,   1 running,  64 sleeping,   0 stopped,   2 zombie
    225 Cpu(s):  0.3%us,  0.3%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
    226 Mem:    756268k total,   608972k used,   147296k free,   149320k buffers
    227 Swap:   794620k total,     4444k used,   790176k free,   336216k cached
    228 }}}
    229 The above shows CPU Load averages:
    230   * 1 minutes:  0.08
    231   * 5 minutes:  0.06
     239  * 5 minutes:  0.01
    232240  * 15 minutes: 0.05
    233241
    … …  
    236244{{{
    237245load average: 0.00, 0.01, 0.05
    238 Tasks:  68 total,   1 running,  67 sleeping,   0 stopped,   0 zombie
    239 Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
    240 Mem:   4031252k total,   966976k used,  3064276k free,    92444k buffers
    241 Swap:  4202492k total,        0k used,  4202492k free,   366360k cached
    242 
     246Tasks:  67 total,   1 running,  66 sleeping,   0 stopped,   0 zombie
     247Cpu(s):  0.2%us,  0.0%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
     248Mem:   4031252k total,   965832k used,  3065420k free,    92468k buffers
     249Swap:  4202492k total,        0k used,  4202492k free,   340948k cached
    243250}}}
    244251The above shows CPU Load averages:
    … …  
    250257'''EG-MON-4-exp1 statistics for OPS Node:'''
    251258{{{
    252 load averages:  0.05,  0.04,  0.00   
    253 101 processes: 1 running, 100 sleeping
    254 CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
    255 Mem: 160M Active, 852M Inact, 164M Wired, 7660K Cache, 96M Buf, 817M Free
    256 Swap: 2047M Total, 852K Used, 2046M Free
     259load averages:  0.01,  0.03,  0.00 
     26099 processes:  1 running, 98 sleeping
     261CPU:  0.4% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.6% idle
     262Mem: 155M Active, 874M Inact, 158M Wired, 6944K Cache, 96M Buf, 807M Free
     263Swap: 2047M Total, 712K Used, 2047M Free
     264
    257265}}}
    258266The above shows CPU Load averages:
    259   * 1 minutes:  0.05
    260   * 5 minutes:  0.04
     267  * 1 minutes:  0.01
     268  * 5 minutes:  0.03
    261269  * 15 minutes:  0.00
    262270
    263271
    264 Leave first experiment IG-MON-4-exp1 running and start experiment IG-MON-4-exp2. This experiment includes 1 PC and 1 VM. Once the sliver was ready started running iperf traffic for 15 minutes. Iperf results:
     272Leave first experiment IG-MON-4-exp1 running and start experiment IG-MON-4-exp2. The experiment IG-MON-4-exp2 includes 1 PC and 1 VM. Once the sliver was ready started running iperf traffic for 15 minutes. Iperf results:
    265273{{{
    266274[lnevers@VM ~]$ /usr/bin/iperf -c 192.168.1.2  -t 900
    … …  
    269277TCP window size: 16.0 KByte (default)
    270278------------------------------------------------------------
    271 [  3] local 192.168.1.1 port 55187 connected with 192.168.1.2 port 5001
     279[  3] local 192.168.1.1 port 44231 connected with 192.168.1.2 port 5001
    272280[ ID] Interval       Transfer     Bandwidth
    273 [  3]  0.0-900.1 sec  10.5 GBytes   100 Mbits/sec
     281[  3]  0.0-900.0 sec  10.5 GBytes   100 Mbits/sec
    274282}}}
    275283At iperf completion checked statistics:
    276284
    277 
    278 '''EG-MON-4-exp2 statistics for VM Node (pc1):"
    279 {{{
    280 load average: 0.00, 0.00, 0.00
    281 Tasks: 457 total,   1 running, 456 sleeping,   0 stopped,   0 zombie
    282 Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
    283 Mem:  49311300k total,  5608100k used, 43703200k free,   438612k buffers
    284 Swap:  1050168k total,        0k used,  1050168k free,  4265232k cached
    285 }}}
    286 
    287 '''EG-MON-4-exp2 statistics for Dedicated Node (pc4):"
    288 {{{
    289 load average: 0.00, 0.01, 0.02
     285'''EG-MON-4-exp2 statistics for VM Node (pc4):"
     286{{{
     287load average: 0.00, 0.00, 0.01
     288Tasks: 429 total,   1 running, 428 sleeping,   0 stopped,   0 zombie
     289Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
     290Mem:  49311300k total,  2813428k used, 46497872k free,    91040k buffers
     291Swap:  1050168k total,        0k used,  1050168k free,  2222324k cached
     292
     293}}}
     294
     295'''EG-MON-4-exp2 statistics for Dedicated Node (pc5):"
     296{{{
     297load average: 0.00, 0.01, 0.04
    290298Tasks: 182 total,   1 running, 181 sleeping,   0 stopped,   0 zombie
    291 Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
    292 Mem:  49714548k total,   281332k used, 49433216k free,     9764k buffers
    293 Swap:  1050172k total,        0k used,  1050172k free,   172524k cached
     299Cpu(s):  0.0%us,  0.1%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.2%si,  0.0%st
     300Mem:  49714548k total,   290696k used, 49423852k free,     9768k buffers
     301Swap:  1050172k total,        0k used,  1050172k free,   182852k cached
    294302}}}
    295303
    296304'''EG-MON-4-exp2 statistics for Boss Node:'''
    297305{{{
    298 load averages:  0.08,  0.08,  0.08             
    299 161 processes: 1 running, 159 sleeping, 1 zombie
    300 CPU:  0.0% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.6% idle
    301 Mem: 392M Active, 1255M Inact, 181M Wired, 26M Cache, 90M Buf, 146M Free
    302 Swap: 2047M Total, 2628K Used, 2045M Free
     306load averages:  0.38,  0.14,  0.09 
     307157 processes: 1 running, 155 sleeping, 1 zombie
     308CPU:  0.4% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.6% idle
     309Mem: 407M Active, 1205M Inact, 195M Wired, 44M Cache, 95M Buf, 149M Free
     310Swap: 2047M Total, 4128K Used, 2043M Free
     311
    303312}}}
    304313
    … …  
    306315{{{
    307316load average: 0.00, 0.01, 0.05
    308 Tasks:  67 total,   1 running,  64 sleeping,   0 stopped,   2 zombie
     317Tasks:  66 total,   1 running,  63 sleeping,   0 stopped,   2 zombie
    309318Cpu(s):  0.0%us,  0.3%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
    310 Mem:    756268k total,   608988k used,   147280k free,   149324k buffers
    311 Swap:   794620k total,     4444k used,   790176k free,   336272k cached
     319Mem:    756268k total,   533908k used,   222360k free,   134808k buffers
     320Swap:   794620k total,     5428k used,   789192k free,   280904k cached
    312321}}}
    313322
    … …  
    316325{{{
    317326load average: 0.00, 0.01, 0.05
    318 Tasks:  68 total,   1 running,  67 sleeping,   0 stopped,   0 zombie
     327Tasks:  67 total,   1 running,  66 sleeping,   0 stopped,   0 zombie
    319328Cpu(s):  0.0%us,  0.2%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
    320 Mem:   4031252k total,   967216k used,  3064036k free,    92444k buffers
    321 Swap:  4202492k total,        0k used,  4202492k free,   366364k cached
     329Mem:   4031252k total,   965832k used,  3065420k free,    92468k buffers
     330Swap:  4202492k total,        0k used,  4202492k free,   340948k cached
    322331}}}
    323332
    … …  
    325334'''EG-MON-4-exp2 statistics for OPS Node:'''
    326335{{{
    327 load averages:  0.19,  0.09,  0.02   
    328 101 processes: 1 running, 100 sleeping
     336load average: 0.00, 0.00, 0.18
     337Tasks: 683 total,   1 running, 682 sleeping,   0 stopped,   0 zombie
     338Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.9%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
     339Mem:  49311300k total, 15166288k used, 34145012k free,   910020k buffers
     340Swap:  1050168k total,        0k used,  1050168k free, 12125464k cached
     341
     342}}}
     343Ping statistic for PC to VM:
     344 - without iperf: "rtt min/avg/max/mdev = 0.083/0.168/0.193/0.032 ms"
     345 - with iperf: "rtt min/avg/max/mdev = 52.344/53.343/54.618/0.800 ms"
     346
     347Restarted iperf to run for the remainder of this test.
     348
     349== 2. View VLAN 1750 data plane monitoring ==
     350
     351In this section we verify the VLAN 1750 data plane monitoring which pings the rack's interface on VLAN 1750, and verify that packets are not being dropped.
     352
     353Leave first experiment IG-MON-4-exp2 running and add the !OpenFlow experiment IG-MON-4-exp3. The experiment IG-MON-4-exp3 includes 2 nodes in the rack exchanging traffic with 2 campus nodes also connected on the !OpenFlow network.  Two nodes were allocated on pc1:
     354
     355
     356'''EG-MON-4-exp3 statistics for PC1:'''
     357{{{
     358load average: 0.00, 0.30, 0.49
     359Tasks: 543 total,   1 running, 542 sleeping,   0 stopped,   0 zombie
     360Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
     361Mem:  49311300k total,  9917688k used, 39393612k free,   751168k buffers
     362Swap:  1050168k total,        0k used,  1050168k free,  7748796k cached
     363}}}
     364
     365'''EG-MON-4-exp3 statistics for Boss Node:'''
     366{{{
     367load averages:  0.05,  0.09,  0.08   
     368156 processes: 1 running, 154 sleeping, 1 zombie
     369CPU:  1.2% user,  0.0% nice,  3.5% system,  0.0% interrupt, 95.3% idle
     370Mem: 403M Active, 1230M Inact, 187M Wired, 44M Cache, 94M Buf, 135M Free
     371Swap: 2047M Total, 4120K Used, 2043M Free
     372
     373}}}
     374
     375'''EG-MON-4-exp3 statistics for FOAM Node:'''
     376{{{
     377load average: 0.15, 0.08, 0.06
     378Tasks:  67 total,   1 running,  63 sleeping,   0 stopped,   3 zombie
     379Cpu(s):  0.0%us,  0.3%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
     380Mem:    756268k total,   534816k used,   221452k free,   134888k buffers
     381Swap:   794620k total,     5428k used,   789192k free,   281560k cached
     382}}}
     383
     384'''EG-MON-4-exp3 statistics for !FlowVisor Node:'''
     385{{{
     386load average: 0.00, 0.01, 0.05
     387Tasks:  67 total,   1 running,  66 sleeping,   0 stopped,   0 zombie
     388Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
     389Mem:   4031252k total,   965816k used,  3065436k free,    92468k buffers
     390Swap:  4202492k total,        0k used,  4202492k free,   340976k cached
     391}}}
     392
     393
     394'''EG-MON-4-exp3 statistics for OPS Node:'''
     395{{{
     396load averages:  0.00,  0.00,  0.00
     397105 processes: 1 running, 104 sleeping
     398CPU:  0.0% user,  0.0% nice,  1.6% system,  0.0% interrupt, 98.4% idle
     399Mem: 162M Active, 876M Inact, 159M Wired, 6932K Cache, 96M Buf, 797M Free
     400Swap: 2047M Total, 712K Used, 2047M Free
     401
     402}}}
     403
     404Leave !OpenFlow experiment IG-MON-4-exp3 running and add the other !OpenFlow experiment IG-MON-4-exp4. Experiment includes 4 nodes exchanging pings withing the GPO Rack, all nodes in experiment were assigned on pc1.
     405
     406
     407'''EG-MON-4-exp4 statistics for Boss Node:'''
     408{{{
     409load averages:  0.03,  0.07,  0.07   
     410156 processes: 1 running, 154 sleeping, 1 zombie
     411CPU:  0.4% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.6% idle
     412Mem: 403M Active, 1234M Inact, 187M Wired, 44M Cache, 96M Buf, 131M Free
     413Swap: 2047M Total, 4104K Used, 2043M Free
     414}}}
     415
     416'''EG-MON-4-exp4 statistics for FOAM Node:'''
     417{{{
     418load average: 0.00, 0.04, 0.05
     419Tasks:  67 total,   1 running,  63 sleeping,   0 stopped,   3 zombie
     420Cpu(s):  0.3%us,  0.0%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
     421Mem:    756268k total,   535420k used,   220848k free,   134908k buffers
     422Swap:   794620k total,     5428k used,   789192k free,   281848k cached
     423}}}
     424
     425
     426'''EG-MON-4-exp4 statistics for !FlowVisor Node:'''
     427{{{
     428load average: 0.00, 0.01, 0.05
     429Tasks:  67 total,   1 running,  66 sleeping,   0 stopped,   0 zombie
     430Cpu(s):  0.0%us,  0.2%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
     431Mem:   4031252k total,   966080k used,  3065172k free,    92468k buffers
     432Swap:  4202492k total,        0k used,  4202492k free,   340992k cached
     433}}}
     434
     435
     436'''EG-MON-4-exp4 statistics for OPS Node:'''
     437{{{
     438load averages:  0.00,  0.00,  0.00 
     439105 processes: 1 running, 104 sleeping
    329440CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
    330 Mem: 160M Active, 852M Inact, 163M Wired, 7604K Cache, 95M Buf, 817M Free
    331 Swap: 2047M Total, 852K Used, 2046M Free
    332 
    333 }}}
    334 Ping statistic for PC to VM:
    335  - without iperf: "rtt min/avg/max/mdev = 0.115/0.129/0.169/0.017 ms"
    336  - with iperf: "rtt min/avg/max/mdev = 52.091/53.219/54.634/0.871 ms"
    337 
    338 Restarted iperf to run for the remainder of this test.
    339 
    340 == 2. View VLAN 1750 data plane monitoring ==
    341 
    342 In this section we verify the VLAN 1750 data plane monitoring which pings the rack's interface on VLAN 1750, and verify that packets are not being dropped..
    343 
    344 Leave first experiment IG-MON-4-exp2 running and add the !OpenFlow experiment IG-MON-4-exp3. Experiment includes 2 nodes in the rack exchanging traffic with 2 campus nodes also connected on the !OpenFlow network.  Two nodes were allocated on pc2:
    345 
    346 
    347 '''EG-MON-4-exp3 statistics for PC2:'''
    348 {{{
    349 load average: 0.58, 0.53, 0.35
    350 Tasks: 703 total,   1 running, 702 sleeping,   0 stopped,   0 zombie
    351 Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.9%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
    352 Mem:  49311300k total, 16904424k used, 32406876k free,   983744k buffers
    353 Swap:  1050168k total,        0k used,  1050168k free, 13605776k cached
    354 }}}
    355 
    356 '''EG-MON-4-exp3 statistics for Boss Node:'''
    357 {{{
    358 load averages:  0.11,  0.09,  0.08             
    359 165 processes: 1 running, 163 sleeping, 1 zombie
     441Mem: 162M Active, 876M Inact, 159M Wired, 6932K Cache, 96M Buf, 796M Free
     442Swap: 2047M Total, 712K Used, 2047M Free
     443
     444}}}
     445
     446
     447'''Statistics for pc1 Node:'''
     448{{{
     449load average: 0.01, 0.22, 0.37
     450Tasks: 595 total,   1 running, 594 sleeping,   0 stopped,   0 zombie
     451Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
     452Mem:  49311300k total, 11733656k used, 37577644k free,   818580k buffers
     453Swap:  1050168k total,        0k used,  1050168k free,  9252840k cached
     454}}}
     455
     456With all traffic from the above 4 experiments running, captured ping round trip time from boss node to each of the other nodes in the rack:
     457
     458 - ops: round-trip min/avg/max/stddev = 0.166/0.202/0.236/0.023 ms
     459 - pc1: round-trip min/avg/max/stddev = 0.222/0.257/0.354/0.039 ms
     460 - pc2: round-trip min/avg/max/stddev = 0.221/0.241/0.256/0.010 ms
     461 - pc3: round-trip min/avg/max/stddev = 0.212/0.228/0.267/0.014 ms
     462 - pc4: round-trip min/avg/max/stddev = 0.207/0.222/0.252/0.012 ms
     463 - pc5: round-trip min/avg/max/stddev = 0.217/0.232/0.248/0.012 ms
     464 - foam: round-trip min/avg/max/stddev = 0.154/2.193/20.119/5.976 ms
     465 - foam-priv: round-trip min/avg/max/stddev = 0.187/1.546/9.995/3.005 ms
     466 - flowvisor: round-trip min/avg/max/stddev = 0.167/0.208/0.385/0.063 ms
     467 - flowvisor-priv: round-trip min/avg/max/stddev = 0.185/0.223/0.381/0.057 ms
     468
     469Ping round trip time were collected after the 4 experiments (and traffic) were delete to provide comparison. Again captured from the boss to each of the other nodes in the rack:
     470
     471 - ops: round-trip min/avg/max/stddev = 0.168/1.190/9.990/2.933 ms
     472 - pc1: round-trip min/avg/max/stddev = 0.223/0.248/0.271/0.016 ms
     473 - pc2: round-trip min/avg/max/stddev = 0.239/0.259/0.307/0.018 ms
     474 - pc3: round-trip min/avg/max/stddev = 0.222/0.240/0.268/0.015 ms
     475 - pc4: round-trip min/avg/max/stddev = 0.235/0.247/0.266/0.009 ms
     476 - pc5: round-trip min/avg/max/stddev = 0.175/0.246/0.418/0.083 ms
     477 - foam: round-trip min/avg/max/stddev = 0.139/0.172/0.211/0.020 ms
     478 - foam-priv: round-trip min/avg/max/stddev = 0.212/0.236/0.272/0.018 ms
     479 - flowvisor: round-trip min/avg/max/stddev = 0.193/0.208/0.221/0.009 ms
     480 - flowvisor-priv: round-trip min/avg/max/stddev = 0.192/0.212/0.228/0.013 ms
     481
     482After the 4 experiment were deleted also collected top statistics:
     483
     484'''Boss Node Statistics:'''
     485{{{
     486load averages:  0.00,  0.07,  0.11   
     487160 processes: 2 running, 157 sleeping, 1 zombie
     488CPU:  0.4% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.6% idle
     489Mem: 484M Active, 1200M Inact, 223M Wired, 32M Cache, 95M Buf, 61M Free
     490Swap: 2047M Total, 2900K Used, 2044M Free
     491}}}
     492'''OPS Node Statistics:'''
     493{{{
     494load averages:  0.00,  0.03,  0.00   
     49599 processes:  1 running, 98 sleeping
    360496CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
    361 Mem: 429M Active, 1277M Inact, 177M Wired, 26M Cache, 93M Buf, 91M Free
    362 Swap: 2047M Total, 2628K Used, 2045M Free
    363 }}}
    364 
    365 '''EG-MON-4-exp3 statistics for FOAM Node:'''
    366 {{{
    367 load average: 0.00, 0.01, 0.05
    368 Tasks:  68 total,   1 running,  64 sleeping,   0 stopped,   3 zombie
    369 Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
    370 Mem:    756268k total,   609500k used,   146768k free,   149336k buffers
    371 Swap:   794620k total,     4444k used,   790176k free,   336988k cached
    372 }}}
    373 
    374 '''EG-MON-4-exp3 statistics for !FlowVisor Node:'''
    375 {{{
    376 load average: 0.00, 0.01, 0.05
    377 Tasks:  68 total,   1 running,  67 sleeping,   0 stopped,   0 zombie
    378 Cpu(s):  0.0%us,  0.2%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
    379 Mem:   4031252k total,   967324k used,  3063928k free,    92444k buffers
    380 Swap:  4202492k total,        0k used,  4202492k free,   366384k cached
    381 }}}
    382 
    383 
    384 '''EG-MON-4-exp3 statistics for OPS Node:'''
    385 {{{
    386 oad averages:  0.06,  0.04,  0.00   
    387 105 processes: 1 running, 104 sleeping
    388 CPU:  0.0% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.6% idle
    389 Mem: 165M Active, 854M Inact, 163M Wired, 7572K Cache, 96M Buf, 810M Free
    390 Swap: 2047M Total, 852K Used, 2046M Free
    391 }}}
    392 
    393 Leave !OpenFlow experiment IG-MON-4-exp3 running and add the other !OpenFlow experiment IG-MON-4-exp4. Experiment includes 4 nodes exchanging ping:
    394 
    395 ????START HERE?????
    396 
    397 '''EG-MON-4-exp4 statistics for Boss Node:'''
    398 {{{
    399 
    400 }}}
    401 The above shows CPU Load averages:
    402   * 1 minutes:  XX
    403   * 5 minutes:  XX
    404   * 15 minutes:  XX
    405 
    406 '''EG-MON-4-exp4 statistics for FOAM Node:'''
    407 {{{
    408 
    409 }}}
    410 The above shows CPU Load averages:
    411   * 1 minutes:  XX
    412   * 5 minutes:  XX
    413   * 15 minutes:  XX
    414 
    415 '''EG-MON-4-exp4 statistics for !FlowVisor Node:'''
    416 {{{
    417 
    418 }}}
    419 The above shows CPU Load averages:
    420   * 1 minutes:  XX
    421   * 5 minutes:  XX
    422   * 15 minutes:  XX
    423 
    424 '''EG-MON-4-exp4 statistics for OPS Node:'''
    425 {{{
    426 
    427 }}}
    428 The above shows CPU Load averages:
    429   * 1 minutes:  XX
    430   * 5 minutes:  XX
    431   * 15 minutes:  XX
    432 
    433 
    434 == 3. Verify CPU idle percentage on the server host and the FOAM VM are both nonzero ==
    435 
    436 
    437 
     497Mem: 155M Active, 911M Inact, 163M Wired, 6928K Cache, 95M Buf, 764M Free
     498Swap: 2047M Total, 712K Used, 2047M Free
     499
     500}}}
     501
     502'''FOAM Node Statistics:'''
     503{{{
     504load average: 0.00, 0.01, 0.05
     505Tasks:  67 total,   1 running,  63 sleeping,   0 stopped,   3 zombie
     506Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
     507Mem:    756268k total,   535296k used,   220972k free,   134996k buffers
     508Swap:   794620k total,     5428k used,   789192k free,   281964k cached
     509}}}
     510
     511'''!FlowVisor Node Statistics:'''
     512{{{
     513load average: 0.00, 0.01, 0.05
     514Tasks:  67 total,   1 running,  66 sleeping,   0 stopped,   0 zombie
     515Cpu(s):  0.2%us,  0.0%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
     516Mem:   4031252k total,   966080k used,  3065172k free,    92468k buffers
     517Swap:  4202492k total,        0k used,  4202492k free,   340992k cached
     518}}}
     519
     520'''PC1 Node Statistics:'''
     521{{{
     522load average: 0.06, 0.02, 0.03
     523Tasks: 491 total,   1 running, 490 sleeping,   0 stopped,   0 zombie
     524Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
     525Mem:  49311300k total,  8113708k used, 41197592k free,   686612k buffers
     526Swap:  1050168k total,        0k used,  1050168k free,  6244968k cached
     527}}}
     528
     529'''PC2 Node Statistics:'''
     530{{{
     531load average: 0.00, 0.04, 0.09
     532Tasks: 463 total,   1 running, 462 sleeping,   0 stopped,   0 zombie
     533Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
     534Mem:  49311300k total,  7008116k used, 42303184k free,   603528k buffers
     535Swap:  1050168k total,        0k used,  1050168k free,  5357824k cached
     536}}}
     537
     538'''PC4 Node Statistics:'''
     539{{{
     540load average: 0.06, 0.02, 0.00
     541Tasks: 403 total,   1 running, 402 sleeping,   0 stopped,   0 zombie
     542Cpu(s):  0.0%us,  5.9%sy,  0.0%ni, 94.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
     543Mem:  49311300k total,  1781088k used, 47530212k free,    57424k buffers
     544Swap:  1050168k total,        0k used,  1050168k free,  1353432k cached
     545}}}
     546
     547
     548== 3. Verify CPU idle percentage on the boss and FOAM/FV hosts are nonzero ==
     549
     550Reviewed all CPU statistics gathered in each of the previous steps and found that all "% idle" values are nonzero.
     551