= EG-EXP-2: InstaGENI Single Site Acceptance Test = This page captures status for the test case EG-EXP-2, which verifies basic operations of VMs and data flows within one rack. For overall status see the [wiki:GENIRacksHome/InstageniRacks/AcceptanceTestStatus InstaGENI Acceptance Test Status] page. = Test Status = This section captures the status for each step in the acceptance test plan. || '''Step''' || '''State''' || '''Date completed''' ||''' Ticket ''' || '''Comments''' || || Step 1 ||[[Color(green,Pass)]] || || instaticket:14||Minor, multiple default images listed|| || Step 2 ||[[Color(green,Pass)]] || || || Customized Ubuntu image available|| || Step 3 ||[[Color(green,Pass)]] || || || || || Step 4 ||[[Color(#98FB98,Pass: most criteria)]] || ||instaticket:19||OpenVZ=Fedora15, cannot request other VM OS, suggestion: modify test to use Fedora for VM. || || Step 5 ||[[Color(green,Pass)]] || || || || || Step 6 ||[[Color(green,Pass)]] || || || || || Step 7 ||[[Color(green,Pass)]] || || || || || Step 8 ||[[Color(green,Pass)]] || || || || || Step 9 ||[[Color(green,Pass)]] || || || || || Step 10 ||[[Color(green,Pass)]] || || || || || Step 11 ||[[Color(green,Pass)]] || || ||2 raw-pc with custom OS - ok || || Step 12 ||[[Color(green,Pass)]] || || || || || Step 13 ||[[Color(green,Pass)]] || || || || || Step 14 ||[[Color(green,Pass)]] || || || || || Step 15 ||[[Color(green,Pass)]] || || || || || Step 16 || || || || || || Step 17 || || || || || || Step 18 || || || || || || Step 19 || || || || || || Step 20 || || || || || [[BR]] || '''State Legend''' || '''Description''' || || [[Color(green,Pass)]] || Test completed and met all criteria || || [[Color(#98FB98,Pass: most criteria)]] || Test completed and met most criteria. Exceptions documented || || [[Color(red,Fail)]] || Test completed and failed to meet criteria. || || [[Color(yellow,Complete)]] || Test completed but will require re-execution due to expected changes || || [[Color(orange,Blocked)]] || Blocked by ticketed issue(s). || || [[Color(#63B8FF,In Progress)]] || Currently under test. || [[BR]] = Prerequisites = Status for test prerequisites: __Prereq:__ InstaGENI makes available at least two Linux distributions and a FreeBSD image as stated in design document __Status:__ Done __Prereq:__ Two GPO customized Ubuntu image snapshots are available and have been manually uploaded by the rack administrator using available InstaGENI documentation. One Ubuntu image is for the VM and one Ubuntu image is for the physical node in this test. __Status:__ Done __Prereq:__ Traffic generation tools may be part of image or may be installed at experiment runtime. __Status:__ Done: Only using ping with mixed sizes and rates for now. __Prereq:__ Administrative accounts have been created for GPO staff on the Utah InstaGENI rack. __Status:__ Done __Prereq:__ GENI Experimenter1 and Experimenter2 accounts exist. __Status:__ Done. Two account already exist, lnevers@bbn.com and lnevers1@bbn.com, these will be used in all acceptance tests. __Prereq:__ Baseline Monitoring is in place for the entire Utah site, to ensure that any problems are quickly identified. __Status:__ Blocked on prerequisite tests completion. = Test Plan Steps = == Step 1. As Experimenter1, request !ListResources from Utah InstaGENI == As experimenter lnevers@bbn.com requested the list of available resources as follows: {{{ $ ./src/omni.py -a insta-utah listresources --available --api-version 2 -t GENI 3 --available -o }}} == Step 2. Review advertisement RSpec for a list of OS images which can be loaded, and identify available resources == Used the output file from previous step to determine list of OS images available and available compute resources: {{{ $ egrep "node component|disk_image|available" rspec-boss-utah-geniracks-net-protogeni-xmlrpc-am-2-0.xml }}} The following disk images were listed: {{{ }}} Note 1: instaticket:14 written for duplicate default image in the Advertisement RSpec. [[BR]] == Step 3. Verify that the GPO Ubuntu customized image is available in the advertisement RSpec == Defined customized OS image as defined in the [http://groups.geni.net/syseng/wiki/GENIRacksProjects/InstageniRackNotes/CustomOS Custom OS] InstaGENI notes page and in instaticket:20. Images available are part of the listresources output which was collected as shown below: {{{ $ ./src/omni.py -a https://boss.utah.geniracks.net/protogeni/xmlrpc/am/2.0 listresources --api-version 2 -t GENI 3 --available -o }}} Found that custom images along with available for PC1: {{{ }}} == Step 4. Define a request RSpec for two VMs, each with a GPO Ubuntu image == Test is also to request a publically routable IP address and public TCP/UDP port mapping for the control interface on each node '''Issue:''' Using advertised Ubuntu or FreeBSD images is not supported for sliver_type emulab-openvz. This does not allow the execution of step 4 in the IG-EXP-2 InstaGENI Single Site test case, found at http://groups.geni.net/geni/wiki/GENIRacksHome/AcceptanceTests/InstageniAcceptanceTestsPlan#IG-EXP-2:InstaGENISingleSiteAcceptanceTest. Two options available: (a) Modify step 4 to assume default image is used (Fedora15). (b) Modify step 4 to assume a raw-pc is used to load the custom image, which duplicates later step 12 in the procedure. Choosing option (a) for the execution of step 4, which implies that RSpec is modified to use default OpenVZ image. == Step 5. Create the first slice == Created the slice as follows: {{{ $ ./src/omni.py createslice ig-exp2-2vm }}} == Step 6. Create a sliver in the first slice, using the "modified" RSpec as defined in step 4 == Created a 2 VM sliver, checked sliver status for completion, and determine assigned host from the sliverstatus sliver manifest: {{{ $ ./src/omni.py createsliver -a insta-utah ig-exp2-2vm --api-version 2 -t GENI 3 ./instarspec/insta-2vm-v3.rspec $ ./src/omni.py sliverstatus -a insta-utah ig-exp2-2vm --api-version 2 -t GENI 3 -o $ egrep "hostname|port" ig-exp2-2vm-sliverstatus-boss-utah-geniracks-net-protogeni-xmlrpc-am-2-0.json }}} == Step 7. Log in to each of the systems, and send traffic to the other system sharing a VLAN == First host: {{{ lnevers@arendia:~$ ssh pc5.utah.geniracks.net -p 30010 Last login: Fri May 25 19:30:25 2012 from arendia.gpolab.bbn.com [lnevers@host1 ~]$ /sbin/ifconfig | egrep "inet addr" inet addr:172.17.5.1 Bcast:172.31.255.255 Mask:255.240.0.0 inet addr:127.0.0.1 Mask:255.0.0.0 inet addr:10.10.1.1 Bcast:10.10.1.255 Mask:255.255.255.0 inet addr:10.10.2.1 Bcast:10.10.2.255 Mask:255.255.255.0 [lnevers@host1 ~]$ ping 10.10.2.2 -c 5 PING 10.10.2.2 (10.10.2.2) 56(84) bytes of data. 64 bytes from 10.10.2.2: icmp_req=1 ttl=64 time=0.037 ms 64 bytes from 10.10.2.2: icmp_req=2 ttl=64 time=0.027 ms 64 bytes from 10.10.2.2: icmp_req=3 ttl=64 time=0.028 ms 64 bytes from 10.10.2.2: icmp_req=4 ttl=64 time=0.028 ms 64 bytes from 10.10.2.2: icmp_req=5 ttl=64 time=0.027 ms --- 10.10.2.2 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 3999ms rtt min/avg/max/mdev = 0.027/0.029/0.037/0.006 ms [lnevers@host1 ~]$ }}} Second host: {{{ lnevers@arendia:~$ ssh pc5.utah.geniracks.net -p 30011 [lnevers@host2 ~]$ /sbin/ifconfig | egrep "inet addr" inet addr:172.17.5.2 Bcast:172.31.255.255 Mask:255.240.0.0 inet addr:127.0.0.1 Mask:255.0.0.0 inet addr:10.10.1.2 Bcast:10.10.1.255 Mask:255.255.255.0 inet addr:10.10.2.2 Bcast:10.10.2.255 Mask:255.255.255.0 [lnevers@host2 ~]$ ping 10.10.2.1 -c 5 PING 10.10.2.1 (10.10.2.1) 56(84) bytes of data. 64 bytes from 10.10.2.1: icmp_req=1 ttl=64 time=0.037 ms 64 bytes from 10.10.2.1: icmp_req=2 ttl=64 time=0.027 ms 64 bytes from 10.10.2.1: icmp_req=3 ttl=64 time=0.028 ms 64 bytes from 10.10.2.1: icmp_req=4 ttl=64 time=0.027 ms 64 bytes from 10.10.2.1: icmp_req=5 ttl=64 time=0.027 ms --- 10.10.2.1 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 3999ms rtt min/avg/max/mdev = 0.027/0.029/0.037/0.005 ms [lnevers@host2 ~]$ }}} == Step 8. Using root privileges on one of the VMs load a Kernel module. == Loading a kernel module is expected to not work on shared OpenVZ nodes, testing will proceed past this step. == Step 9. Run a netcat listener and bind to port XYZ on each of the VMs in the Utah rack == Modified test to use iperf. Started iperf server on host2: {{{ [lnevers@host2 ~]$ /usr/bin/iperf -s }}} == Step 10. Send traffic to port XYZ on each of the VMs in the Utah rack over the control network from any commodity Internet host == Started iperf on host1 to send to host2, with the following results: On host1: {{{ [lnevers@host1 ~]$ /usr/bin/iperf -c 10.10.1.2 ------------------------------------------------------------ Client connecting to 10.10.1.2, TCP port 5001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.10.1.1 port 42950 connected with 10.10.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 116 MBytes 97.5 Mbits/sec }}} On host2: {{{ [lnevers@host2 ~]$ /usr/bin/iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ [ 4] local 10.10.1.2 port 5001 connected with 10.10.1.1 port 42950 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.2 sec 116 MBytes 95.7 Mbits/sec }}} == Step 11. As Experimenter2, request !ListResources from Utah InstaGENI == As experimenter lnevers1@bbn.com requested the list of available resources as follows: {{{ $ ./src/omni.py -a insta-utah listresources --available --api-version 2 -t GENI 3 --available -o }}} == Step 12. Define a request RSpec for two physical nodes, both using the uploaded GPO Ubuntu images == Created a 2 raw-pc RSpec using the Customized OS image. Add RSpec link to add here. == Step 13. Create the second slice == The following command was used: {{{ $ ./src/omni.py createslice lnubuntu12b }}} == Step 14. Create a sliver in the second slice, using the RSpec defined in step 12 == The following commands were used to create the sliver and determine the host assignement: {{{ $ ./src/omni.py createsliver -a insta-utah --api-version 2 -t GENI 3 lnubuntu12b ./instarspec/insta-2rawpc-ubuntu12.rspec $ ./src/omni.py sliverstatus -a insta-utah --api-version 2 -t GENI 3 lnubuntu12b -o $ egrep "hostname|port" lnubuntu12b-sliverstatus-boss-utah-geniracks-net-protogeni-xmlrpc-am-2-0.json }}} == Step 15. Log in to each of the systems, and send traffic to the other system == Logged in to the first assigned host: {{{ lnevers1@sendaria:~$ ssh pc4.utah.geniracks.net Welcome to Ubuntu 12.04 LTS (GNU/Linux 2.6.38.7-1.0emulab x86_64) * Documentation: https://help.ubuntu.com/ The programs included with the Ubuntu system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. hostx:~% cat /etc/issue Ubuntu 12.04 LTS \n \l hostx:~% }}} Logged in to the second host: {{{ lnevers1@sendaria:~$ ssh pc1.utah.geniracks.netWelcome to Ubuntu 12.04 LTS (GNU/Linux 2.6.38.7-1.0emulab x86_64) * Documentation: https://help.ubuntu.com/ The programs included with the Ubuntu system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. hosty:~% cat /etc/issue Ubuntu 12.04 LTS \n \l hosty:~% }}} == Step 16. Verify that experimenters 1 and 2 cannot use the control plane to access each other's resources == Verified network access from hostx to host2: {{{ hostx:~% ifconfig eth0 eth0 Link encap:Ethernet HWaddr e8:39:35:b1:ec:9c inet addr:155.98.34.14 Bcast:155.98.34.255 Mask:255.255.255.0 inet6 addr: fe80::ea39:35ff:feb1:ec9c/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:952 errors:0 dropped:0 overruns:0 frame:0 TX packets:669 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:91246 (91.2 KB) TX bytes:84542 (84.5 KB) Interrupt:30 Memory:f4000000-f4012800 hostx:~% ping -c 1 172.17.5.4 PING 172.17.5.4 (172.17.5.4) 56(84) bytes of data. --- 172.17.5.4 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms hostx:~% }}} Verified network access from host2 to hostx: {{{ [lnevers@host2 ~]$ /sbin/ifconfig eth999 eth999 Link encap:Ethernet HWaddr 00:00:AC:11:05:04 inet addr:172.17.5.4 Bcast:172.31.255.255 Mask:255.240.0.0 inet6 addr: fe80::200:acff:fe11:504/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1977 errors:0 dropped:0 overruns:0 frame:0 TX packets:1873 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:122593 (119.7 KiB) TX bytes:262672 (256.5 KiB) [lnevers@host2 ~]$ ping -c 1 155.98.34.14 PING 155.98.34.14 (155.98.34.14) 56(84) bytes of data. --- 155.98.34.14 ping statistics --- 1 packets transmitted, 0 received, 100% packet loss, time 0ms [lnevers@host2 ~]$ }}} Verified unauthenticated SSH access, as user lnevers1 tried to ssh to control interface for lnevers experiment: {{{ hostx:~% ssh pc5.utah.geniracks.net -p 30267 -l lnevers lnevers@pc5.utah.geniracks.net's password: Permission denied, please try again. lnevers@pc5.utah.geniracks.net's password: Permission denied, please try again. lnevers@pc5.utah.geniracks.net's password: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). hostx:~% }}} Verified shared writable filesystem mount for each user. As user lnevers accessed shared area: {{{ [lnevers@host2 ~]$ id uid=20001(lnevers) gid=504(pgeni-gpolab-bbn) groups=504(pgeni-gpolab-bbn),0(root) [lnevers@host2 ~]$ ls -l /proj/pgeni-gpolab-bbn-com/exp/ total 6 drwxrwx--- 10 geniuser pgeni-gpolab-bbn 512 May 31 07:57 ecgtest drwxrwx--- 10 geniuser pgeni-gpolab-bbn 512 Jun 1 08:19 ig-exp2-2vm drwxrwx--- 10 geniuser pgeni-gpolab-bbn 512 Jun 1 08:25 lnubuntu12b }}} As user lnevers1 accessed shared area: {{{ hostx:~% id uid=20001(lnevers1) gid=504(pgeni-gpolab-bbn) groups=504(pgeni-gpolab-bbn),0(root) hostx:~% cd /proj/pgeni-gpolab-bbn-com/exp/ hostx:/proj/pgeni-gpolab-bbn-com/exp% ls -l total 6 drwxrwx--- 10 geniuser pgeni-gpolab-bbn 512 May 31 07:57 ecgtest drwxrwx--- 10 geniuser pgeni-gpolab-bbn 512 Jun 1 08:19 ig-exp2-2vm drwxrwx--- 10 geniuser pgeni-gpolab-bbn 512 Jun 1 08:25 lnubuntu12b hostx:/proj/pgeni-gpolab-bbn-com/exp% }}} == Step 17. Review system statistics and VM isolation and network isolation on data plane == == Step 18. Verify that each VM has a distinct MAC address for that interface == On the first host: {{{ hosty:~% ifconfig -a|grep HW eth0 Link encap:Ethernet HWaddr e8:39:35:b1:0f:94 eth1 Link encap:Ethernet HWaddr e8:39:35:b1:0f:96 eth2 Link encap:Ethernet HWaddr e8:39:35:b1:0f:64 eth3 Link encap:Ethernet HWaddr e8:39:35:b1:0f:66 }}} On the second host: {{{ hostx:~% ifconfig -a|grep HW eth0 Link encap:Ethernet HWaddr e8:39:35:b1:ec:9c eth1 Link encap:Ethernet HWaddr e8:39:35:b1:ec:9e eth2 Link encap:Ethernet HWaddr e8:39:35:b1:ec:d0 eth3 Link encap:Ethernet HWaddr e8:39:35:b1:ec:d2 }}} == Step 19. Verify that VMs' MAC addresses are learned on the data plane switch == Successfully exchanged traffic over the dataplane, as shown by ARP tables. ARP table on hosty: {{{ hosty:~% arp -a ops.utah.geniracks.net (155.98.34.5) at 00:00:9b:62:24:e0 [ether] on eth0 ? (155.98.34.1) at 00:d0:bc:f4:14:f8 [ether] on eth0 boss.utah.geniracks.net (155.98.34.4) at 00:00:9b:62:24:df [ether] on eth0 hostX-lan0 (10.10.1.1) at e8:39:35:b1:ec:9e [ether] on eth1 hosty:~% }}} ARP table on hostx: {{{ hostx:~% arp -a ops.utah.geniracks.net (155.98.34.5) at 00:00:9b:62:24:e0 [ether] on eth0 ? (155.98.34.1) at 00:d0:bc:f4:14:f8 [ether] on eth0 boss.utah.geniracks.net (155.98.34.4) at 00:00:9b:62:24:df [ether] on eth0 hostY-lan0 (10.10.1.2) at e8:39:35:b1:0f:96 [ether] on eth1 hostx:~% }}} == Step 20. Stop traffic and delete slivers ==