wiki:GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-EXP-2

Version 42 (modified by lnevers@bbn.com, 12 years ago) (diff)

--

EG-EXP-2: InstaGENI Single Site Acceptance Test

This page captures status for the test case EG-EXP-2, which verifies basic operations of VMs and data flows within one rack. For overall status see the InstaGENI Acceptance Test Status page.

Test Status

This section captures the status for each step in the acceptance test plan.

Step State Date completed Ticket Comments
Step 1 Color(green,Pass)? instaticket:14Minor, multiple default images listed
Step 2 Color(green,Pass)? Customized Ubuntu image available
Step 3 Color(green,Pass)?
Step 4 Color(#98FB98,Pass: most criteria)? instaticket:19OpenVZ=Fedora15, cannot request other VM OS, modified test to use Fedora for VM.
Step 5 Color(green,Pass)?
Step 6 Color(green,Pass)?
Step 7 Color(green,Pass)?
Step 8 Color(green,Pass)?
Step 9 Color(green,Pass)?
Step 10 Color(green,Pass)?
Step 11 Color(green,Pass)? 2 raw-pc with custom Ubuntu 12.04
Step 12 Color(green,Pass)?
Step 13 Color(green,Pass)?
Step 14 Color(green,Pass)?
Step 15 Color(green,Pass)?
Step 16 Color(green,Pass)?
Step 17 Color(green,Pass)?
Step 18 Color(green,Pass)?
Step 19 Color(green,Pass)?
Step 20


State Legend Description
Color(green,Pass)? Test completed and met all criteria
Color(#98FB98,Pass: most criteria)? Test completed and met most criteria. Exceptions documented
Color(red,Fail)? Test completed and failed to meet criteria.
Color(yellow,Complete)? Test completed but will require re-execution due to expected changes
Color(orange,Blocked)? Blocked by ticketed issue(s).
Color(#63B8FF,In Progress)? Currently under test.


Prerequisites

Status for test prerequisites:

Prereq: InstaGENI makes available at least two Linux distributions and a FreeBSD image as stated in design document

Status: Done

Prereq: Two GPO customized Ubuntu image snapshots are available and have been manually uploaded by the rack administrator using available InstaGENI documentation. One Ubuntu image is for the VM and one Ubuntu image is for the physical node in this test.

Status: Done

Prereq: Traffic generation tools may be part of image or may be installed at experiment runtime.

Status: Done: Only using ping with mixed sizes and rates for now.

Prereq: Administrative accounts have been created for GPO staff on the Utah InstaGENI rack.

Status: Done

Prereq: GENI Experimenter1 and Experimenter2 accounts exist.

Status: Done. Two account already exist, lnevers@bbn.com and lnevers1@bbn.com, these will be used in all acceptance tests.

Prereq: Baseline Monitoring is in place for the entire Utah site, to ensure that any problems are quickly identified.

Status: Blocked on prerequisite tests completion.

Test Plan Steps

Step 1. As Experimenter1, request ListResources from Utah InstaGENI

As experimenter lnevers@bbn.com requested the list of available resources as follows:

 $ ./src/omni.py -a insta-utah listresources --available --api-version 2 -t GENI 3 --available -o

Step 2. Review advertisement RSpec for a list of OS images which can be loaded, and identify available resources

Used the output file from previous step to determine list of OS images available and available compute resources:

 $ egrep "node component|disk_image|available" rspec-boss-utah-geniracks-net-protogeni-xmlrpc-am-2-0.xml 

The following disk images were listed:

<disk_image description="FreeBSD 8.2 32-bit version" name="urn:publicid:IDN+utah.geniracks.net+image+emulab-ops:FBSD82-STD" os="FreeBSD" version="8.2"/>      
<disk_image default="true" description="Standard 32-bit Fedora 15 image" name="urn:publicid:IDN+utah.geniracks.net+image+emulab-ops:FEDORA15-STD" os="Fedora" version="15"/>      
<disk_image description="Standard 64-bit Ubuntu 11 image" name="urn:publicid:IDN+utah.geniracks.net+image+emulab-ops:UBUNTU11-64-STD" os="Linux" version="11.04"/>      
<disk_image default="true" description="Standard 32-bit Fedora 15 image" name="urn:publicid:IDN+utah.geniracks.net+image+emulab-ops:FEDORA15-STD" os="Fedora" version="15"/>      

Note 1: instaticket:14 written for duplicate default image in the Advertisement RSpec.

Step 3. Verify that the GPO Ubuntu customized image is available in the advertisement RSpec

Defined customized OS image as defined in the Custom OS InstaGENI notes page and in instaticket:20.

Images available are part of the listresources output which was collected as shown below:

$ ./src/omni.py -a https://boss.utah.geniracks.net/protogeni/xmlrpc/am/2.0 listresources --api-version 2 -t GENI 3 --available -o

Found that custom images along with available for PC1:

<node component_id="urn:publicid:IDN+utah.geniracks.net+node+pc1" component_manager_id="urn:publicid:IDN+utah.geniracks.net+authority+cm" component_name="pc1" exclusive="true">    
<disk_image description="Ubuntu 12.04 image " name="urn:publicid:IDN+utah.geniracks.net+image+gpo:LNUBUNTU1204" os="Linux" version="2.6.38.7-1.0"/>      
<available now="true"/>    

Step 4. Define a request RSpec for two VMs, each with a GPO Ubuntu image

Test is also to request a publically routable IP address and public TCP/UDP port mapping for the control interface on each node

Issue:

Using advertised Ubuntu or FreeBSD images is not supported for sliver_type emulab-openvz. This does not allow the execution of step 4 in the IG-EXP-2 InstaGENI Single Site test case, found at http://groups.geni.net/geni/wiki/GENIRacksHome/AcceptanceTests/InstageniAcceptanceTestsPlan#IG-EXP-2:InstaGENISingleSiteAcceptanceTest.

Two options available:

(a) Modify step 4 to assume default image is used (Fedora15).

(b) Modify step 4 to assume a raw-pc is used to load the custom image, which duplicates later step 12 in the procedure.

Choosing option (a) for the execution of step 4, which implies that RSpec is modified to use default OpenVZ image.

Step 5. Create the first slice

Created the slice as follows:

$ ./src/omni.py createslice ig-exp2-2vm  

Step 6. Create a sliver in the first slice, using the "modified" RSpec as defined in step 4

Created a 2 VM sliver, checked sliver status for completion, and determine assigned host from the sliverstatus sliver manifest:

$ ./src/omni.py createsliver -a insta-utah ig-exp2-2vm --api-version 2 -t GENI 3 ./instarspec/insta-2vm-v3.rspec 
$ ./src/omni.py sliverstatus -a insta-utah ig-exp2-2vm --api-version 2 -t GENI 3 -o
$ egrep "hostname|port" ig-exp2-2vm-sliverstatus-boss-utah-geniracks-net-protogeni-xmlrpc-am-2-0.json

Step 7. Log in to each of the systems, and send traffic to the other system sharing a VLAN

First host:

lnevers@arendia:~$ ssh pc5.utah.geniracks.net -p 30010
Last login: Fri May 25 19:30:25 2012 from arendia.gpolab.bbn.com
[lnevers@host1 ~]$ /sbin/ifconfig | egrep "inet addr"
          inet addr:172.17.5.1  Bcast:172.31.255.255  Mask:255.240.0.0
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet addr:10.10.1.1  Bcast:10.10.1.255  Mask:255.255.255.0
          inet addr:10.10.2.1  Bcast:10.10.2.255  Mask:255.255.255.0
[lnevers@host1 ~]$ ping 10.10.2.2 -c 5
PING 10.10.2.2 (10.10.2.2) 56(84) bytes of data.
64 bytes from 10.10.2.2: icmp_req=1 ttl=64 time=0.037 ms
64 bytes from 10.10.2.2: icmp_req=2 ttl=64 time=0.027 ms
64 bytes from 10.10.2.2: icmp_req=3 ttl=64 time=0.028 ms
64 bytes from 10.10.2.2: icmp_req=4 ttl=64 time=0.028 ms
64 bytes from 10.10.2.2: icmp_req=5 ttl=64 time=0.027 ms

--- 10.10.2.2 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.027/0.029/0.037/0.006 ms
[lnevers@host1 ~]$ 

Second host:

lnevers@arendia:~$ ssh pc5.utah.geniracks.net -p 30011
[lnevers@host2 ~]$ /sbin/ifconfig | egrep "inet addr"
          inet addr:172.17.5.2  Bcast:172.31.255.255  Mask:255.240.0.0
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet addr:10.10.1.2  Bcast:10.10.1.255  Mask:255.255.255.0
          inet addr:10.10.2.2  Bcast:10.10.2.255  Mask:255.255.255.0
[lnevers@host2 ~]$ ping 10.10.2.1 -c 5
PING 10.10.2.1 (10.10.2.1) 56(84) bytes of data.
64 bytes from 10.10.2.1: icmp_req=1 ttl=64 time=0.037 ms
64 bytes from 10.10.2.1: icmp_req=2 ttl=64 time=0.027 ms
64 bytes from 10.10.2.1: icmp_req=3 ttl=64 time=0.028 ms
64 bytes from 10.10.2.1: icmp_req=4 ttl=64 time=0.027 ms
64 bytes from 10.10.2.1: icmp_req=5 ttl=64 time=0.027 ms

--- 10.10.2.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.027/0.029/0.037/0.005 ms
[lnevers@host2 ~]$ 

Step 8. Using root privileges on one of the VMs load a Kernel module.

Loading a kernel module is expected to not work on shared OpenVZ nodes, testing will proceed past this step.

Step 9. Run a netcat listener and bind to port XYZ on each of the VMs in the Utah rack

Modified test to use iperf. Started iperf server on host2:

[lnevers@host2 ~]$ /usr/bin/iperf -s

Step 10. Send traffic to port XYZ on each of the VMs in the Utah rack over the control network from any commodity Internet host

Started iperf on host1 to send to host2, with the following results:

On host1:

[lnevers@host1 ~]$ /usr/bin/iperf -c 10.10.1.2
------------------------------------------------------------
Client connecting to 10.10.1.2, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 10.10.1.1 port 42950 connected with 10.10.1.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   116 MBytes  97.5 Mbits/sec

On host2:

[lnevers@host2 ~]$ /usr/bin/iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 10.10.1.2 port 5001 connected with 10.10.1.1 port 42950
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.2 sec   116 MBytes  95.7 Mbits/sec

Step 11. As Experimenter2, request ListResources from Utah InstaGENI

As experimenter lnevers1@bbn.com requested the list of available resources as follows:

 $ ./src/omni.py -a insta-utah listresources --available --api-version 2 -t GENI 3 --available -o

Step 12. Define a request RSpec for two physical nodes, both using the uploaded GPO Ubuntu images

Created a 2 raw-pc RSpec using the Customized OS image. Add RSpec link to add here.

Step 13. Create the second slice

The following command was used:

 $ ./src/omni.py createslice lnubuntu12b

Step 14. Create a sliver in the second slice, using the RSpec defined in step 12

The following commands were used to create the sliver and determine the host assignement:

 $ ./src/omni.py createsliver -a insta-utah --api-version 2 -t GENI 3 lnubuntu12b ./instarspec/insta-2rawpc-ubuntu12.rspec 
 $ ./src/omni.py sliverstatus -a insta-utah --api-version 2 -t GENI 3 lnubuntu12b -o
 $ egrep "hostname|port" lnubuntu12b-sliverstatus-boss-utah-geniracks-net-protogeni-xmlrpc-am-2-0.json

Step 15. Log in to each of the systems, and send traffic to the other system

Logged in to the first assigned host:

lnevers1@sendaria:~$ ssh pc4.utah.geniracks.net
Welcome to Ubuntu 12.04 LTS (GNU/Linux 2.6.38.7-1.0emulab x86_64)

 * Documentation:  https://help.ubuntu.com/

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

hostx:~% cat /etc/issue
Ubuntu 12.04 LTS \n \l

hostx:~% 

Logged in to the second host:

lnevers1@sendaria:~$ ssh pc1.utah.geniracks.netWelcome to Ubuntu 12.04 LTS (GNU/Linux 2.6.38.7-1.0emulab x86_64)

 * Documentation:  https://help.ubuntu.com/

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

hosty:~% cat /etc/issue
Ubuntu 12.04 LTS \n \l

hosty:~% 

Exchanged iperf traffic from hosty to hostx. Server statistics:

hostx:~% /usr/bin/iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 10.10.1.1 port 5001 connected with 10.10.1.2 port 36048
[  5] local 10.10.1.1 port 5001 connected with 10.10.1.2 port 36049
[  6] local 10.10.1.1 port 5001 connected with 10.10.1.2 port 36050
[  7] local 10.10.1.1 port 5001 connected with 10.10.1.2 port 36051
[  8] local 10.10.1.1 port 5001 connected with 10.10.1.2 port 36053
[  9] local 10.10.1.1 port 5001 connected with 10.10.1.2 port 36054
[ 10] local 10.10.1.1 port 5001 connected with 10.10.1.2 port 36052
[ 11] local 10.10.1.1 port 5001 connected with 10.10.1.2 port 36055
[ 12] local 10.10.1.1 port 5001 connected with 10.10.1.2 port 36056
[ 13] local 10.10.1.1 port 5001 connected with 10.10.1.2 port 36057
[ ID] Interval       Transfer     Bandwidth
[  7]  0.0-10.1 sec   163 MBytes   135 Mbits/sec
[  8]  0.0-10.1 sec   154 MBytes   128 Mbits/sec
[ 10]  0.0-10.1 sec   146 MBytes   121 Mbits/sec
[  5]  0.0-10.1 sec   105 MBytes  87.0 Mbits/sec
[ 11]  0.0-10.1 sec   106 MBytes  87.4 Mbits/sec
[ 12]  0.0-10.1 sec   102 MBytes  84.2 Mbits/sec
[  4]  0.0-10.1 sec  83.1 MBytes  68.8 Mbits/sec
[  9]  0.0-10.1 sec   111 MBytes  91.9 Mbits/sec
[  6]  0.0-10.1 sec  85.4 MBytes  70.6 Mbits/sec
[ 13]  0.0-10.1 sec  84.5 MBytes  69.9 Mbits/sec
[SUM]  0.0-10.1 sec  1.11 GBytes   941 Mbits/sec

Client statistics:

hosty:~% /usr/bin/iperf -c 10.10.1.1 -P 10
------------------------------------------------------------
Client connecting to 10.10.1.1, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[ 12] local 10.10.1.2 port 36057 connected with 10.10.1.1 port 5001
[  3] local 10.10.1.2 port 36048 connected with 10.10.1.1 port 5001
[  4] local 10.10.1.2 port 36049 connected with 10.10.1.1 port 5001
[  6] local 10.10.1.2 port 36050 connected with 10.10.1.1 port 5001
[  7] local 10.10.1.2 port 36051 connected with 10.10.1.1 port 5001
[  5] local 10.10.1.2 port 36052 connected with 10.10.1.1 port 5001
[  8] local 10.10.1.2 port 36053 connected with 10.10.1.1 port 5001
[ 11] local 10.10.1.2 port 36055 connected with 10.10.1.1 port 5001
[  9] local 10.10.1.2 port 36054 connected with 10.10.1.1 port 5001
[ 10] local 10.10.1.2 port 36056 connected with 10.10.1.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  83.1 MBytes  69.6 Mbits/sec
[  7]  0.0-10.0 sec   163 MBytes   136 Mbits/sec
[  9]  0.0-10.0 sec   111 MBytes  92.9 Mbits/sec
[  5]  0.0-10.0 sec   146 MBytes   122 Mbits/sec
[  8]  0.0-10.0 sec   154 MBytes   129 Mbits/sec
[ 11]  0.0-10.0 sec   106 MBytes  88.2 Mbits/sec
[  4]  0.0-10.0 sec   105 MBytes  87.7 Mbits/sec
[ 10]  0.0-10.1 sec   102 MBytes  84.8 Mbits/sec
[ 12]  0.0-10.1 sec  84.5 MBytes  70.4 Mbits/sec
[  6]  0.0-10.1 sec  85.4 MBytes  71.1 Mbits/sec
[SUM]  0.0-10.1 sec  1.11 GBytes   948 Mbits/sec

Step 16. Verify that experimenters 1 and 2 cannot use the control plane to access each other's resources

Verified network access from hostx to host2:

hostx:~% ifconfig eth0
eth0      Link encap:Ethernet  HWaddr e8:39:35:b1:ec:9c  
          inet addr:155.98.34.14  Bcast:155.98.34.255  Mask:255.255.255.0
          inet6 addr: fe80::ea39:35ff:feb1:ec9c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:952 errors:0 dropped:0 overruns:0 frame:0
          TX packets:669 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:91246 (91.2 KB)  TX bytes:84542 (84.5 KB)
          Interrupt:30 Memory:f4000000-f4012800 

hostx:~% ping -c 1 172.17.5.4
PING 172.17.5.4 (172.17.5.4) 56(84) bytes of data.

--- 172.17.5.4 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

hostx:~% 

Verified network access from host2 to hostx:

[lnevers@host2 ~]$ /sbin/ifconfig eth999
eth999    Link encap:Ethernet  HWaddr 00:00:AC:11:05:04  
          inet addr:172.17.5.4  Bcast:172.31.255.255  Mask:255.240.0.0
          inet6 addr: fe80::200:acff:fe11:504/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1977 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1873 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:122593 (119.7 KiB)  TX bytes:262672 (256.5 KiB)

[lnevers@host2 ~]$ ping -c 1 155.98.34.14
PING 155.98.34.14 (155.98.34.14) 56(84) bytes of data.

--- 155.98.34.14 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

[lnevers@host2 ~]$ 

Verified unauthenticated SSH access, as user lnevers1 tried to ssh to control interface for lnevers experiment:

hostx:~% ssh pc5.utah.geniracks.net -p 30267 -l lnevers
lnevers@pc5.utah.geniracks.net's password: 
Permission denied, please try again.
lnevers@pc5.utah.geniracks.net's password: 
Permission denied, please try again.
lnevers@pc5.utah.geniracks.net's password: 
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
hostx:~% 

Verified shared writable filesystem mount for each user. As user lnevers accessed shared area:

[lnevers@host2 ~]$ id
uid=20001(lnevers) gid=504(pgeni-gpolab-bbn) groups=504(pgeni-gpolab-bbn),0(root)
[lnevers@host2 ~]$ ls -l /proj/pgeni-gpolab-bbn-com/exp/
total 6
drwxrwx--- 10 geniuser pgeni-gpolab-bbn 512 May 31 07:57 ecgtest
drwxrwx--- 10 geniuser pgeni-gpolab-bbn 512 Jun  1 08:19 ig-exp2-2vm
drwxrwx--- 10 geniuser pgeni-gpolab-bbn 512 Jun  1 08:25 lnubuntu12b

As user lnevers1 accessed shared area:

hostx:~% id
uid=20001(lnevers1) gid=504(pgeni-gpolab-bbn) groups=504(pgeni-gpolab-bbn),0(root)
hostx:~% cd /proj/pgeni-gpolab-bbn-com/exp/
hostx:/proj/pgeni-gpolab-bbn-com/exp% ls -l
total 6
drwxrwx--- 10 geniuser pgeni-gpolab-bbn 512 May 31 07:57 ecgtest
drwxrwx--- 10 geniuser pgeni-gpolab-bbn 512 Jun  1 08:19 ig-exp2-2vm
drwxrwx--- 10 geniuser pgeni-gpolab-bbn 512 Jun  1 08:25 lnubuntu12b
hostx:/proj/pgeni-gpolab-bbn-com/exp% 

Step 17. Review system statistics and VM isolation and network isolation on data plane

Reviewed statistics on one of the VM nodes:

[lnevers@host1 ~]$ vmstat -s
     49311612 K total memory
      1622396 K used memory
       506288 K active memory
       646452 K inactive memory
     47689216 K free memory
       192144 K buffer memory
       805172 K swap cache
      1050168 K total swap
            0 K used swap
      1050168 K free swap
          292 non-nice user cpu ticks
            0 nice user cpu ticks
          114 system cpu ticks
     11139308 idle cpu ticks
          668 IO-wait cpu ticks
            0 IRQ cpu ticks
            0 softirq cpu ticks
            4 stolen cpu ticks
        25376 pages paged in
         1428 pages paged out
            0 pages swapped in
            0 pages swapped out
            0 interrupts
     63676068 CPU context switches
   1338560297 boot time
       823425 forks
[lnevers@host1 ~]$ top

top - 11:27:01 up  3:08,  1 user,  load average: 0.00, 0.00, 0.00
Tasks:  16 total,   1 running,  15 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  49311612k total,  1623180k used, 47688432k free,   192372k buffers
Swap:  1050168k total,        0k used,  1050168k free,   805400k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND           
    1 root      20   0 19236 1496 1248 S  0.0  0.0   0:00.01 init               
    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd/3         
    3 root      20   0     0    0    0 S  0.0  0.0   0:00.00 khelper/3          
   35 root      20   0 15664 1212  644 S  0.0  0.0   0:00.01 crond              
  314 root      20   0  179m 1436 1048 S  0.0  0.0   0:00.03 rsyslogd           
  327 root      20   0 53216 1088  548 S  0.0  0.0   0:00.29 watchquagga        
  348 root      20   0 75096 1232  544 S  0.0  0.0   0:00.00 sshd               
  357 root      20   0  4152  468  344 S  0.0  0.0   0:00.00 pubsubd            
  407 root      20   0  135m 7620 1232 S  0.0  0.0   0:00.02 watchdog           
  544 root      20   0  135m 6948  960 S  0.0  0.0   0:00.00 rc.progagent       
  547 geniuser  20   0 20440 1464 1140 S  0.0  0.0   0:00.00 program-agent      
  563 root      20   0 43428  768  300 S  0.0  0.0   0:00.00 linktest           
  593 root      20   0  116m 4016 3120 S  0.0  0.0   0:00.01 sshd               
  595 lnevers   20   0  116m 1844  940 S  0.0  0.0   0:00.00 sshd               
  596 lnevers   20   0 16276 2160 1232 S  0.0  0.0   0:00.01 csh                
  664 lnevers   20   0 14940 1136  904 R  0.0  0.0   0:00.00 top                
[lnevers@host1 ~]$ procinfo
Linux 2.6.32-042stab049.6.emulab.1 (root@vhost-1) (gcc 4.6.0 20110603 (Red Hat 4.6.0-10)) #1 12CPU [host1]

Memory:      Total        Used        Free      Shared     Buffers      
Mem:      49311612     1779116    47532496           0      195832
Swap:      1050168           0     1050168

Bootup: Fri Jun  1 08:18:17 2012    Load average: 0.12 0.06 0.02 1/19 682


user  :       0:00:10.64 0.0  page in :        0
nice  :       0:00:00.00 0.0  page out:        0
system:       0:00:02.02 0.0  swap in :        0
idle  :   1d  8:11:46.75 83.1  swap out:        0
steal :       0:00:00.06 0.0
uptime:       3:13:41.57         context : 63864389	interrupts:        0

On the dedicated node:

hosty:~% vmstat -s
     49419336 K total memory
      1157500 K used memory
        87184 K active memory
        97064 K inactive memory
     48261836 K free memory
        24744 K buffer memory
       135156 K swap cache
      1050172 K total swap
            0 K used swap
      1050172 K free swap
          891 non-nice user cpu ticks
            0 nice user cpu ticks
         1913 system cpu ticks
     15472943 idle cpu ticks
         3951 IO-wait cpu ticks
            0 IRQ cpu ticks
           30 softirq cpu ticks
            0 stolen cpu ticks
       130541 pages paged in
       110892 pages paged out
            0 pages swapped in
            0 pages swapped out
       876929 interrupts
       390560 CPU context switches
   1338561263 boot time
         2388 forks
hosty:~% top

top - 12:09:44 up  3:35,  2 users,  load average: 0.08, 0.06, 0.05
Tasks: 125 total,   1 running, 124 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  49419336k total,  1157820k used, 48261516k free,    24756k buffers
Swap:  1050172k total,        0k used,  1050172k free,   135264k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                              
    1 root      20   0 24448 2412 1344 S    0  0.0   0:04.40 init                                                                  
    2 root      20   0     0    0    0 S    0  0.0   0:00.00 kthreadd                                                              
    3 root      20   0     0    0    0 S    0  0.0   0:00.10 ksoftirqd/0                                                           
    4 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/0:0                                                           
    5 root      20   0     0    0    0 S    0  0.0   0:00.34 kworker/u:0                                                           
    6 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/0                                                           
    7 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/1                                                           
    8 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/1:0                                                           
    9 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/1                                                           
   10 root      20   0     0    0    0 S    0  0.0   0:00.43 kworker/0:1                                                           
   11 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/2                                                           
   12 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/2:0                                                           
   13 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/2                                                           
   14 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/3                                                           
   16 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/3                                                           
   17 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/4                                                           
   18 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/4:0                                                           
   19 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/4                                                           
   20 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/5                                                           
   21 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/5:0                                                           
   22 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/5                                                           
   23 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/6                                                           
   24 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/6:0                                                           
   25 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/6                                                           
   26 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/7                                                           
   27 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/7:0                                                           
   28 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/7                                                           
   29 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/8                                                           
   30 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/8:0                                                           
   31 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/8                                                           
   32 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/9                                                           
   33 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/9:0                                                           
   34 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/9                                                           
   35 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/10                                                          
   36 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/10:0                                                          
   37 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/10                                                          
   38 root      RT   0     0    0    0 S    0  0.0   0:00.00 migration/11                                                          
   39 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/11:0                                                          
   40 root      20   0     0    0    0 S    0  0.0   0:00.00 ksoftirqd/11                                                          
   41 root       0 -20     0    0    0 S    0  0.0   0:00.00 cpuset                                                                
   42 root       0 -20     0    0    0 S    0  0.0   0:00.00 khelper                                                               
   43 root       0 -20     0    0    0 S    0  0.0   0:00.00 netns                                                                 
   44 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/u:1                                                           
   45 root      20   0     0    0    0 S    0  0.0   0:00.02 sync_supers                                                           
   46 root      20   0     0    0    0 S    0  0.0   0:00.00 bdi-default                                                           
   47 root       0 -20     0    0    0 S    0  0.0   0:00.00 kintegrityd                                                           
   48 root       0 -20     0    0    0 S    0  0.0   0:00.00 kblockd                                     
hosty:~% procinfo
Memory:        Total        Used        Free     Buffers                       
RAM:        49419336     1159808    48259528       25264                       
Swap:        1050172           0     1050172                                   

Bootup: Fri Jun  1 08:34:23 2012   Load average: 0.30 0.12 0.07 1/134 2557     

user  :      00:00:09.94   0.0%  page in :           131061                    
nice  :      00:00:00.00   0.0%  page out:           155300                    
system:      00:00:19.54   0.0%  page act:            36316                    
IOwait:      00:00:43.91   0.0%  page dea:                0                    
hw irq:      00:00:00.00   0.0%  page flt:           901566                    
sw irq:      00:00:00.31   0.0%  swap in :                0                    
idle  :   1d 19:16:12.01 100.0%  swap out:                0                    
uptime:      03:36:29.88         context :           395778                    

irq   0:         91  timer               irq  73:     146679  eth1-2           
irq   1:          2  i8042               irq  74:      31512  eth1-3           
irq   4:        257  serial              irq  75:      18875  eth1-4           
irq   8:          1  rtc0                irq  76:      15677  eth1-5           
irq   9:          0  acpi                irq  77:      55817  eth1-6           
irq  12:          4  i8042               irq  78:      38014  eth1-7           
irq  17:         32  uhci_hcd:usb6       irq  98:       2667  eth0-0           
irq  20:          0  ehci_hcd:usb1, uh   irq  99:      12860  eth0-1           
irq  22:          0  uhci_hcd:usb4       irq 100:        323  eth0-2           
irq  23:          0  uhci_hcd:usb3, uh   irq 101:        447  eth0-3           
irq  64:          0  dmar0               irq 102:        698  eth0-4           
irq  67:      15591  hpsa0               irq 103:      12655  eth0-5           
irq  71:      14552  eth1-0              irq 104:        331  eth0-6           
irq  72:      45503  eth1-1              irq 105:        173  eth0-7           

sda             6190r           10560w                                         

eth0        TX 2.12MiB       RX 1.24MiB       eth3        TX 0.00B         RX 0.00B        
eth1        TX 3.49GiB       RX 26.92MiB      lo          TX 8.25KiB       RX 8.25KiB      
eth2        TX 0.00B         RX 0.00B                                          

Step 18. Verify that each VM has a distinct MAC address for that interface

On the first host:

hosty:~% ifconfig -a|grep HW
eth0      Link encap:Ethernet  HWaddr e8:39:35:b1:0f:94  
eth1      Link encap:Ethernet  HWaddr e8:39:35:b1:0f:96  
eth2      Link encap:Ethernet  HWaddr e8:39:35:b1:0f:64  
eth3      Link encap:Ethernet  HWaddr e8:39:35:b1:0f:66  

On the second host:

hostx:~% ifconfig -a|grep HW
eth0      Link encap:Ethernet  HWaddr e8:39:35:b1:ec:9c  
eth1      Link encap:Ethernet  HWaddr e8:39:35:b1:ec:9e  
eth2      Link encap:Ethernet  HWaddr e8:39:35:b1:ec:d0  
eth3      Link encap:Ethernet  HWaddr e8:39:35:b1:ec:d2  

Step 19. Verify that VMs' MAC addresses are learned on the data plane switch

Successfully exchanged traffic over the dataplane, as shown by ARP tables.

ARP table on hosty:

hosty:~% arp -a
ops.utah.geniracks.net (155.98.34.5) at 00:00:9b:62:24:e0 [ether] on eth0
? (155.98.34.1) at 00:d0:bc:f4:14:f8 [ether] on eth0
boss.utah.geniracks.net (155.98.34.4) at 00:00:9b:62:24:df [ether] on eth0
hostX-lan0 (10.10.1.1) at e8:39:35:b1:ec:9e [ether] on eth1
hosty:~% 

ARP table on hostx:

hostx:~% arp -a
ops.utah.geniracks.net (155.98.34.5) at 00:00:9b:62:24:e0 [ether] on eth0
? (155.98.34.1) at 00:d0:bc:f4:14:f8 [ether] on eth0
boss.utah.geniracks.net (155.98.34.4) at 00:00:9b:62:24:df [ether] on eth0
hostY-lan0 (10.10.1.2) at e8:39:35:b1:0f:96 [ether] on eth1
hostx:~% 

Step 20. Stop traffic and delete slivers

As user lnevers@bbn.com issued the following:

$ ./src/omni.py deletesliver -a insta-utah ig-exp2-2vm --api-version 2 

As user lnevers1@bbn.com issued the following:

$ ./src/omni.py deletesliver -a insta-utah lnubuntu12b --api-version 2

Verified that resources are released by checking listresources details. Listresources details show that the dedicated nodes are available, and that the VM slot counts are restored:

$ ./src/omni.py -a insta-utah listresources --available --api-version 2 -t GENI 3 -o
$ egrep "node comp|avail|type_slots" rspec-boss-utah-geniracks-net-protogeni-xmlrpc-am-2-0.xml


XXXX - waiting for dedicated nodes....