= EG-EXP-4: ExoGENI Multi-site Acceptance Test = This page captures status for the test case EG-EXP-4, which verifies the ability to support basic operations of VMs and data flows between two racks. For overall status see the [wiki:GENIRacksHome/ExogeniRacks/AcceptanceTestStatus ExoGENI Acceptance Test Status] page. ''Last update: 03/11/13'' = Test Status = This section captures the status for each step in the acceptance test plan. || '''Step''' || '''State''' || '''Date completed''' ||''' Ticket ''' || '''Comments''' || || Step 1 ||[[Color(yellow,Complete)]]|| || ||Run with ExoSM || || Step 2 ||[[Color(yellow,Complete)]]|| || || || || Step 3 ||[[Color(yellow,Complete)]]|| || || || || Step 4 ||[[Color(yellow,Complete)]]|| || || || || Step 5 ||[[Color(yellow,Complete)]]|| || || || || Step 6 ||[[Color(yellow,Complete)]]|| || || || || Step 7 ||[[Color(yellow,Complete)]]|| || ||Run with ExoSM|| || Step 8 ||[[Color(yellow,Complete)]]|| || |||| || Step 9 ||[[Color(yellow,Complete)]]|| || || || || Step 10 ||[[Color(yellow,Complete)]]|| || || || || Step 11 ||[[Color(yellow,Complete)]]|| || || || || Step 12 ||[[Color(yellow,Complete)]]|| || || || || Step 13 ||[[Color(yellow,Complete)]]|| || || || || Step 14 ||[[Color(orange,Blocked)]]|| || ||monitoring does not support bare metal nodes, no VM data at this time|| || Step 15 ||[[Color(yellow,Complete)]]|| || || || || Step 16 ||[[Color(orange,Blocked)]]|| || ||monitoring does not support bare metal nodes|| || Step 17 ||[[Color(yellow,Complete)]]|| || || || [[BR]] || '''State Legend''' || '''Description''' || || [[Color(green,Pass)]] || Test completed and met all criteria || || [[Color(#98FB98,Pass: most criteria)]] || Test completed and met most criteria. Exceptions documented || || [[Color(red,Fail)]] || Test completed and failed to meet criteria. || || [[Color(yellow,Complete)]] || Test completed but will require re-execution due to expected changes || || [[Color(orange,Blocked)]] || Blocked by ticketed issue(s). || || [[Color(#63B8FF,In Progress)]] || Currently under test. || [[BR]] = Test Plan Steps = This test case is modified to use ExoSM to request resources across sites. The omni commands shown in this test use nicknames nick_names for aggregates, which are: {{{ eg-sm=,https://geni.renci.org:11443/orca/xmlrpc }}} is used in place of individual site's aggregates: {{{ eg-gpo=,https://bbn-hn.exogeni.net:11443/orca/xmlrpc eg-renci=,https://rci-hn.exogeni.net:11443/orca/xmlrpc }}} == Step 1. As Experimenter1, Request !ListResources RENCI ExoGENI == Using the credentials lnevers@bbn.com, request listresources from ExoSM, to determine which resources can be requested for the first experiment: {{{ $ omni.py -a eg-sm listresources -o }}} The above command generates a file named ''rspec-geni-renci-org-11443-orca.xml''. == Step 2. Review !ListResources output == Reviewed content of output file named ''rspec-geni-renci-org-11443-orca.xml'' and determined site information for VM request. == Step 3. Define a request RSpec == Define a request RSpec for a VM at BBN ExoGENI, a VM at RENCI ExoGENI and an unbound exclusive non-!OpenFlow VLAN to connect the 2 endpoints. RSpec created for this experiment is [http://groups.geni.net/geni/browser/trunk/GENIRacks/ExoGENI/Spiral4/Rspecs/AcceptanceTests/EG-EXP-4/EG-EXP-4-exp1.rspec EG-EXP-4-exp1.rspec] == Step 4. Create the first slice. == Using the following command create a slice for the first experiment: {{{ $ omni.py createslice EG-EXP-4-exp1 INFO:omni:Loading config file /home/lnevers/.gcf/omni_config INFO:omni:Using control framework pg INFO:omni:Created slice with Name EG-EXP-4-exp1, URN urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+EG-EXP-4-exp1, Expiration 2013-03-12 15:46:18+00:00 INFO:omni: ------------------------------------------------------------ INFO:omni: Completed createslice: Options as run: framework: pg Args: createslice EG-EXP-4-exp1 Result Summary: Created slice with Name EG-EXP-4-exp1, URN urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+EG-EXP-4-exp1, Expiration 2013-03-12 15:46:18+00:00 INFO:omni: =========================================================== }}} == Step 5. Create a sliver == Using the ExoSM and the RSpecs defined above create a sliver with one VM at BBN and one VM at RENCI: {{{ $ omni.py createsliver -a eg-sm EG-EXP-4-exp1 EG-EXP-4-exp1.rspec }}} Verify that sliver status is ready: {{{ $ omni.py createsliver -a eg-sm EG-EXP-4-exp1 EG-EXP-4-exp1.rspec INFO:omni:Loading config file /home/lnevers/.gcf/omni_config INFO:omni:Using control framework pg INFO:omni:Substituting AM nickname eg-sm with URL https://geni.renci.org:11443/orca/xmlrpc, URN unspecified_AM_URN INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+EG-EXP-4-exp1 expires on 2013-03-12 15:46:18 UTC INFO:omni:Substituting AM nickname eg-sm with URL https://geni.renci.org:11443/orca/xmlrpc, URN unspecified_AM_URN INFO:omni:Substituting AM nickname eg-sm with URL https://geni.renci.org:11443/orca/xmlrpc, URN unspecified_AM_URN INFO:omni:Creating sliver(s) from rspec file EG-EXP-4-exp1.rspec for slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+EG-EXP-4-exp1 INFO:omni:Got return from CreateSliver for slice EG-EXP-4-exp1 at https://geni.renci.org:11443/orca/xmlrpc: INFO:omni: INFO:omni: INFO:omni: unknown unknown 0 0 10000000 10000000 unknown l2sc ethernet hop2 unknown unknown 0 0 10000000 10000000 unknown l2sc ethernet hop3 unknown unknown 0 0 10000000 10000000 unknown l2sc ethernet hop4 unknown unknown 0 0 10000000 10000000 unknown l2sc ethernet null INFO:omni: ------------------------------------------------------------ INFO:omni: Completed createsliver: Options as run: aggregate: ['eg-sm'] framework: pg Args: createsliver EG-EXP-4-exp1 EG-EXP-4-exp1.rspec Result Summary: Got Reserved resources RSpec from geni-renci-org-11443-orca INFO:omni: ============================================================ }}} When sliverstatus reports an overall sliverStatus of ''ready'', determined login information for allocated VMs: {{{ $ readyToLogin.py -a eg-sm EG-EXP-4-exp1 <...> User root logins to geni1 using: xterm -e ssh -i /home/lnevers/.ssh/id_rsa root@192.1.242.18 & User root logins to geni2 using: xterm -e ssh -i /home/lnevers/.ssh/id_rsa root@152.54.14.12 & }}} == Step 6. Log in to each of the systems, and send traffic to the other system, leave traffic running == Connect to the RENCI VM and send traffic to the BBN VM: {{{ $ ssh root@152.54.14.34 root@debian:~# ifconfig|egrep 172.16 inet addr:172.16.7.2 Bcast:172.16.7.255 Mask:255.255.255.0 root@debian:~# ping 172.16.7.1 PING 172.16.7.1 (172.16.7.1) 56(84) bytes of data. 64 bytes from 172.16.7.1: icmp_req=1 ttl=64 time=48.2 ms 64 bytes from 172.16.7.1: icmp_req=2 ttl=64 time=18.6 ms 64 bytes from 172.16.7.1: icmp_req=3 ttl=64 time=17.9 ms }}} Connect to the BBN VM and send traffic to the RENCI VM: {{{ $ ssh root@192.1.242.12 root@debian:~# ifconfig |egrep 172.16 inet addr:172.16.7.1 Bcast:172.16.7.255 Mask:255.255.255.0 root@debian:~# ping 172.16.7.2 PING 172.16.7.2 (172.16.7.2) 56(84) bytes of data. 64 bytes from 172.16.7.2: icmp_req=1 ttl=64 time=24.4 ms 64 bytes from 172.16.7.2: icmp_req=2 ttl=64 time=17.9 ms 64 bytes from 172.16.7.2: icmp_req=3 ttl=64 time=17.8 ms }}} == Step 7. As Experimenter2, Request !ListResources from RENCI ExoGENI == Using the credentials lnevers2@bbn.com, request listresources from ExoSM, to determine which resources can be requested: {{{ $ omni.py -a eg-sm listresources -o }}} == Step 8. Define a request RSpec == Define a request RSpec for one VM and one bare metal node each with two interfaces in the BBN ExoGENI rack, two VMs at RENCI, and two VLANs to connect the BBN ExoGENI to the RENCI ExoGENI. RSpec created for this topology is [http://groups.geni.net/geni/browser/trunk/GENIRacks/ExoGENI/Spiral4/Rspecs/AcceptanceTests/EG-EXP-4/EG-EXP-4-exp2.rspec EG-EXP-4-exp2.rspec] and it defines the following mapping for network, interfaces, and network addresses: - BBN Bare Metal if0 172.16.2.1 connects to BBN VM 2 if0 172.16.2.2 - BBN Bare Metal if1 172.16.3.1 connects to RCI VM 1 if0 172.16/.3.2 - BBN VM2 if1 172.16.4.1 connects to RCI VM 2 if0 172.16.4.2 == Step 9. Create a second slice == Using the following command create a slice for the second experiment: {{{ $ omni.py createslice EG-EXP-4-exp2 }}} == Step 10. In the second slice, create a sliver at the RENCI ExoGENI aggregate using the RSpecs defined above == Using the ExoSM and the RSpecs defined in step 8 create a sliver: {{{ $ omni.py createsliver -a eg-sm EG-EXP-4-exp2 EG-EXP-4-exp2.rspec }}} Verify that sliver status is ready: {{{ $ omni.py sliverstatus -a eg-sm EG-EXP-4-exp2 }}} Determined which nodes (VMs and bare metal) are allocated to this sliver: {{{ $ omni.py listresources -a eg-sm EG-EXP-4-exp2 -o $ egrep hostname EG-EXP-4-exp2-rspec-geni-renci-org-11443-orca.xml }}} Also can determine login information as follows: {{{ $ examples/readyToLogin.py -a eg-sm EG-EXP-4-exp2 }}} == Step 11. Log in to each of the end-point systems, and send traffic to the other end-point system which shares the same VLAN == Logged in to BBN VM and pinged each of the remotes: {{{ $ ssh root@192.1.242.13 root@debian:~# ifconfig eth0 Link encap:Ethernet HWaddr 02:16:3e:0c:60:0f inet addr:10.103.0.10 Bcast:10.103.0.255 Mask:255.255.255.0 inet6 addr: fe80::16:3eff:fe0c:600f/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:278 errors:0 dropped:0 overruns:0 frame:0 TX packets:248 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:39892 (38.9 KiB) TX bytes:35307 (34.4 KiB) eth1 Link encap:Ethernet HWaddr 52:54:00:bf:8e:ad inet addr:172.16.4.1 Bcast:172.16.4.255 Mask:255.255.255.0 inet6 addr: fe80::5054:ff:febf:8ead/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:17 errors:0 dropped:0 overruns:0 frame:0 TX packets:5 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1005 (1005.0 B) TX bytes:398 (398.0 B) eth2 Link encap:Ethernet HWaddr 52:54:00:d7:20:a4 inet addr:172.16.2.2 Bcast:172.16.2.255 Mask:255.255.255.0 inet6 addr: fe80::5054:ff:fed7:20a4/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:30 errors:0 dropped:0 overruns:0 frame:0 TX packets:5 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2060 (2.0 KiB) TX bytes:398 (398.0 B) root@debian:~# ping 172.16.4.2 -c 3 PING 172.16.4.2 (172.16.4.2) 56(84) bytes of data. 64 bytes from 172.16.4.2: icmp_req=1 ttl=64 time=58.9 ms 64 bytes from 172.16.4.2: icmp_req=2 ttl=64 time=18.0 ms 64 bytes from 172.16.4.2: icmp_req=3 ttl=64 time=17.9 ms --- 172.16.4.2 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 17.916/31.659/58.970/19.311 ms root@debian:~# ping 172.16.2.1 -c 3 PING 172.16.2.1 (172.16.2.1) 56(84) bytes of data. 64 bytes from 172.16.2.1: icmp_req=1 ttl=64 time=11.5 ms 64 bytes from 172.16.2.1: icmp_req=2 ttl=64 time=0.442 ms 64 bytes from 172.16.2.1: icmp_req=3 ttl=64 time=0.399 ms --- 172.16.2.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2000ms rtt min/avg/max/mdev = 0.399/4.135/11.566/5.254 ms }}} Log in to BBN bare metal and pinged each of the remotes: {{{ $ ssh root@192.1.242.110 [root@bbn-w4 ~]# ifconfig eth0 Link encap:Ethernet HWaddr 5C:F3:FC:BA:51:D4 inet addr:10.100.0.14 Bcast:10.100.0.255 Mask:255.255.255.0 inet6 addr: fe80::5ef3:fcff:feba:51d4/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:211 errors:0 dropped:0 overruns:0 frame:0 TX packets:1595 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:34898 (34.0 KiB) TX bytes:205446 (200.6 KiB) Interrupt:28 Memory:92000000-92012800 eth2 Link encap:Ethernet HWaddr 00:07:43:11:A4:60 inet6 addr: fe80::207:43ff:fe11:a460/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:285 errors:0 dropped:0 overruns:0 frame:0 TX packets:21 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:22953 (22.4 KiB) TX bytes:1774 (1.7 KiB) Interrupt:24 eth2.24 Link encap:Ethernet HWaddr 00:07:43:11:A4:60 inet addr:172.16.2.1 Bcast:172.16.2.255 Mask:255.255.255.0 inet6 addr: fe80::207:43ff:fe11:a460/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:15 errors:0 dropped:0 overruns:0 frame:0 TX packets:10 errors:0 dropped:1 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:930 (930.0 b) TX bytes:756 (756.0 b) eth2.2603 Link encap:Ethernet HWaddr 00:07:43:11:A4:60 inet addr:172.16.3.1 Bcast:172.16.3.255 Mask:255.255.255.0 inet6 addr: fe80::207:43ff:fe11:a460/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:10 errors:0 dropped:0 overruns:0 frame:0 TX packets:5 errors:0 dropped:1 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:590 (590.0 b) TX bytes:378 (378.0 b) [root@bbn-w4 ~]# ping -c 3 172.16.3.2 PING 172.16.3.2 (172.16.3.2) 56(84) bytes of data. 64 bytes from 172.16.3.2: icmp_seq=1 ttl=64 time=46.8 ms 64 bytes from 172.16.3.2: icmp_seq=2 ttl=64 time=17.6 ms 64 bytes from 172.16.3.2: icmp_seq=3 ttl=64 time=17.6 ms --- 172.16.3.2 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2021ms rtt min/avg/max/mdev = 17.688/27.402/46.827/13.736 ms [root@bbn-w4 ~]# ping -c 3 172.16.2.2 PING 172.16.2.2 (172.16.2.2) 56(84) bytes of data. 64 bytes from 172.16.2.2: icmp_seq=1 ttl=64 time=4.96 ms 64 bytes from 172.16.2.2: icmp_seq=2 ttl=64 time=0.444 ms 64 bytes from 172.16.2.2: icmp_seq=3 ttl=64 time=0.404 ms --- 172.16.2.2 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.404/1.937/4.964/2.140 ms [root@bbn-w4 ~]# }}} == Step 12. Verify traffic handling per experiment, VM isolation, and MAC address assignment == Verified MAC address assignment for the BBN VM: {{{ lnevers@sendaria:~/gcf-1.6.2$ ssh root@192.1.242.13 root@debian:~# ifconfig|grep HWaddr eth0 Link encap:Ethernet HWaddr 02:16:3e:0c:60:0f eth1 Link encap:Ethernet HWaddr 52:54:00:bf:8e:ad eth2 Link encap:Ethernet HWaddr 52:54:00:d7:20:a4 root@debian:~# }}} Verified MAC address assignment for the BBN bare metal: {{{ Connection to 192.1.242.13 closed. lnevers@sendaria:~/gcf-1.6.2$ ssh root@192.1.242.110 [root@bbn-w4 ~]# ifconfig|grep HWaddr eth0 Link encap:Ethernet HWaddr 5C:F3:FC:BA:51:D4 eth2 Link encap:Ethernet HWaddr 00:07:43:11:A4:60 eth2.24 Link encap:Ethernet HWaddr 00:07:43:11:A4:60 eth2.2603 Link encap:Ethernet HWaddr 00:07:43:11:A4:60 }}} == Step 13. Construct and send a non-IP Ethernet packet over the data plane interface. == On two ExoGENI hosts in the BBN rack, installed and compiled the the GPO package !PingPlus, which is available at the [http://software.geni.net/local-sw/ GPO Software Repository] to generate a non-IP Ethernet packet. Determined interface information on each host. Host1: {{{ root@debian:~/pingPlus-0.2# ifconfig eth1 eth1 Link encap:Ethernet HWaddr 52:54:00:9c:3f:8a inet addr:172.16.1.1 Bcast:172.16.255.255 Mask:255.255.0.0 inet6 addr: fe80::5054:ff:fe9c:3f8a/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:16 errors:0 dropped:0 overruns:0 frame:0 TX packets:5 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1128 (1.1 KiB) TX bytes:398 (398.0 B) }}} Host2: {{{ eth1 Link encap:Ethernet HWaddr 52:54:00:0e:e5:a1 inet addr:172.16.1.2 Bcast:172.16.255.255 Mask:255.255.0.0 inet6 addr: fe80::5054:ff:fe0e:e5a1/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:14 errors:0 dropped:0 overruns:0 frame:0 TX packets:5 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:916 (916.0 B) TX bytes:398 (398.0 B) }}} Started the pingPlusListener on host1: {{{ root@debian:~/pingPlus-0.2# ./pingPlusListener 10111 }}} After installing tcpdump package, started tcpdump on host1: {{{ root@debian:~# tcpdump ether host 52:54:00:0e:e5:a1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes }}} On host 2 sent 10 packets with !pingPlus to host1: {{{ root@debian:~/pingPlus-0.2# ./pingPlus 52:54:00:9c:3f:8a eth0 10111 5 RQ:'1844+6426' to 52:54:0:9c:3f:8a. RL:1844+6426=8270 from 2:16:3e:3c:ee:1f. RQ:'725+1578' to 52:54:0:9c:3f:8a. RL:1844+6426=8270 from 2:16:3e:3c:ee:1f. RQ:'6819+4568' to 52:54:0:9c:3f:8a. RL:1844+6426=8270 from 2:16:3e:3c:ee:1f. RQ:'2740+9308' to 52:54:0:9c:3f:8a. RL:1844+6426=8270 from 2:16:3e:3c:ee:1f. RQ:'340+8734' to 52:54:0:9c:3f:8a. RL:1844+6426=8270 from 2:16:3e:3c:ee:1f. }}} Packets were received on host1: {{{ root@debian:~/pingPlus-0.2# ./pingPlusListener 10111 RQ:1844+6426 from 2:16:3e:78:b9:b2. RL:1844+6426=8270. RQ:725+1578 from 2:16:3e:78:b9:b2. RL:725+1578=2303. RQ:6819+4568 from 2:16:3e:78:b9:b2. RL:6819+4568=11387. RQ:2740+9308 from 2:16:3e:78:b9:b2. RL:2740+9308=12048. RQ:340+8734 from 2:16:3e:78:b9:b2. RL:340+8734=9074. }}} Tcpdump captured the content of the Ethernet packet: {{{ root@debian:~# tcpdump ether host 02:16:3e:78:b9:b2 .... 16:31:49.363091 02:16:3e:78:b9:b2 (oui Unknown) > 52:54:00:9c:3f:8a (oui Unknown), ethertype Unknown (0x277f), length 56: 0x0000: 5251 3a31 3633 302b 3733 3832 0000 0000 RQ:1630+7382.... 0x0010: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0x0020: 0000 0000 0000 0000 0000 .......... .... }}} == Step 14. Review baseline monitoring statistics == Current monitoring does not support bare metal nodes and no resource detail is available for the slice. Captures will be added when data is available. == Step 15. Run test for at least 4 hours == Current test run is 1 hour, will increase in future test run. == Step 16. Review baseline monitoring statistics == Current monitoring does not support bare metal nodes and no resource detail is available for the slice. Captures will be added when data is available. == Step 17. Stop traffic and delete slivers == As experimenter1, delete the sliver: {{{ $ omni.py deletesliver -a eg-sm EG-EXP-4-exp1 }}} As experimenter2 delete the sliver: {{{ $ omni.py deletesliver -a eg-sm EG-EXP-4-exp2 }}}