= IG-EXP-4: InstaGENI Multi-site Acceptance Test =
This page captures status for the test case IG-EXP-4, which verifies the ability to support basic operations of VMs and data flows between two racks. For overall status see the [wiki:GENIRacksHome/InstageniRacks/AcceptanceTestStatus InstaGENI Acceptance Test Status] page.
''Last Updates: 2013/01/07''
= Test Status =
This section captures the status for each step in the acceptance test plan.
|| '''Step''' || '''State''' || '''Date completed''' ||''' Ticket ''' || '''Comments''' ||
|| Step 1 ||[[Color(green,Pass)]] || || || ||
|| Step 2 ||[[Color(green,Pass)]] || || || ||
|| Step 3 ||[[Color(green,Pass)]] || || || ||
|| Step 4 ||[[Color(green,Pass)]] || || || ||
|| Step 5 ||[[Color(green,Pass)]] || || || ||
|| Step 6 ||[[Color(green,Pass)]] || || || ||
|| Step 7 ||[[Color(green,Pass)]] || || || ||
|| Step 8 ||[[Color(green,Pass)]] || || || ||
|| Step 9 ||[[Color(green,Pass)]] || || || ||
|| Step 10 ||[[Color(green,Pass)]] || || || ||
|| Step 11 ||[[Color(green,Pass)]] || || || ||
|| Step 12 ||[[Color(green,Pass)]] || || || ||
|| Step 13 ||[[Color(green,Pass)]] || || || ||
|| Step 14 ||[[Color(green,Pass)]] || || || ||
|| Step 15 ||[[Color(green,Pass)]] || || || ||
|| Step 16 ||[[Color(green,Pass)]] || || || ||
|| Step 17 ||[[Color(green,Pass)]] || || || ||
|| Step 18 ||[[Color(green,Pass)]] || || instaticket:74|| ||
|| Step 19 ||[[Color(green,Pass)]] || || || ||
[[BR]]
|| '''State Legend''' || '''Description''' ||
|| [[Color(green,Pass)]] || Test completed and met all criteria ||
|| [[Color(#98FB98,Pass: most criteria)]] || Test completed and met most criteria. Exceptions documented ||
|| [[Color(red,Fail)]] || Test completed and failed to meet criteria. ||
|| [[Color(yellow,Complete)]] || Test completed but will require re-execution due to expected changes ||
|| [[Color(orange,Blocked)]] || Blocked by ticketed issue(s). ||
|| [[Color(#63B8FF,In Progress)]] || Currently under test. ||
[[BR]]
= Test Plan Steps =
The tests described in this page are executed at Utah and GPO InstaGENI racks, as planned. Two user credentials are used to execute the experiments, lnevers@bbn.com and lnever2@bbn.com. Additionally the following aggregate manager nicknames are used:
{{{
ig-utah=,https://utah.geniracks.net/protogeni/xmlrpc/am/2.0
ig-gpo=,https://instageni.gpolab.bbn.com/protogeni/xmlrpc/am/2.0
}}}
== 1. As Experimenter1, Request !ListResources from GPO InstaGENI ==
As experimenter 1, lnevers@bbn.com, collected listresources from GPO InstaGENI rack:
{{{
$ omni.py listresources -a ig-gpo -o
}}}
== 2. Request !ListResources from Utah InstaGENI. ==
Collected listresources from Utah InstaGENI rack:
{{{
$ omni.py listresources -a ig-utah -o
}}}
== 3. Review !ListResources output from both AMs ==
Reviewed listresources output from each aggregated to choose resources.
== 4. Define a request RSpec for a VM at GPO InstaGENI ==
The Experiment 1 RSpec for the topology in this test case is used for both aggregates, the file generated is [http://groups.geni.net/geni/browser/trunk/GENIRacks/InstaGENI/Spiral4/RSpecs/AcceptanceTests/IG-EXP-4/IG-EXP-4-exp1.rspec IG-EXP-4-exp1.rspec].
== 5. Define a request RSpec for a VM at Utah InstaGENI ==
The same [http://groups.geni.net/geni/browser/trunk/GENIRacks/InstaGENI/Spiral4/RSpecs/AcceptanceTests/IG-EXP-4/IG-EXP-4-exp1.rspec IG-EXP-4-exp1.rspec] is used at both aggregates and it includes an unbound exclusive non-!OpenFlow VLAN to connect the 2 endpoints.
== 6. Create the first slice ==
Created the slice:
{{{
$ omni.py createslice IG-EXP-4-exp1
}}}
== 7. Create a sliver at each InstaGENI aggregate ==
Using the RSpec defined above, create a sliver at each InstaGENI aggregate. First at GPO aggregate:
{{{
$ omni.py createsliver IG-EXP-4-exp1 -a ig-gpo ./IG-EXP-4-exp1.rspec
INFO:omni:Loading config file /home/lnevers/.gcf/omni_config
INFO:omni:Using control framework pg
INFO:omni:Substituting AM nickname ig-gpo with URL http://instageni.gpolab.bbn.com/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN
INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-EXP-4-exp1 expires on 2013-01-08 17:08:18 UTC
INFO:omni:Substituting AM nickname ig-gpo with URL http://instageni.gpolab.bbn.com/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN
INFO:omni:Substituting AM nickname ig-gpo with URL http://instageni.gpolab.bbn.com/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN
INFO:omni:Creating sliver(s) from rspec file ./IG-EXP-4-exp1.rspec for slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-EXP-4-exp1
INFO:omni:Got return from CreateSliver for slice IG-EXP-4-exp1 at http://instageni.gpolab.bbn.com/protogeni/xmlrpc/am/2.0:
INFO:omni:
INFO:omni:
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed createsliver:
Options as run:
aggregate: ['ig-gpo']
framework: pg
Args: createsliver IG-EXP-4-exp1 ./IG-EXP-4-exp1.rspec
Result Summary: Got Reserved resources RSpec from instageni-gpolab-bbn-com-protogeniv2
INFO:omni: ============================================================
}}}
Then at Utah aggregate:
{{{
$ omni.py createsliver IG-EXP-4-exp1 -a ig-utah ./IG-EXP-4-exp1.rspec
INFO:omni:Loading config file /home/lnevers/.gcf/omni_config
INFO:omni:Using control framework pg
INFO:omni:Substituting AM nickname ig-utah with URL http://utah.geniracks.net/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN
INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-EXP-4-exp1 expires on 2013-01-08 17:08:18 UTC
INFO:omni:Substituting AM nickname ig-utah with URL http://utah.geniracks.net/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN
INFO:omni:Substituting AM nickname ig-utah with URL http://utah.geniracks.net/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN
INFO:omni:Creating sliver(s) from rspec file ./IG-EXP-4-exp1.rspec for slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-EXP-4-exp1
INFO:omni:Got return from CreateSliver for slice IG-EXP-4-exp1 at http://utah.geniracks.net/protogeni/xmlrpc/am/2.0:
INFO:omni:
INFO:omni:
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed createsliver:
Options as run:
aggregate: ['ig-utah']
framework: pg
Args: createsliver IG-EXP-4-exp1 ./IG-EXP-4-exp1.rspec
Result Summary: Got Reserved resources RSpec from utah-geniracks-net-protogeniv2
INFO:omni: ============================================================
}}}
=== 8. Log in to each of the systems ==
Determine login information, log in to each system, and send traffic to the other system, leave traffic running.
{{{
$ readyToLogin.py IG-EXP-4-exp1 -a ig-gpo
<...>
VM-2's geni_status is: ready (am_status:ready)
User lnevers logins to VM-2 using:
xterm -e ssh -p 31290 -i /home/lnevers/.ssh/id_rsa lnevers@pc2.instageni.gpolab.bbn.com &
$ readyToLogin.py IG-EXP-4-exp1 -a ig-utah
<...>
VM-3's geni_status is: ready (am_status:ready)
User lnevers logins to VM-3 using:
xterm -e ssh -p 31546 -i /home/lnevers/.ssh/id_rsa lnevers@pc1.utah.geniracks.net &
}}}
On the IG Utah VM:
{{{
[lnevers@VM-3 ~]$ ping 192.168.4.2
PING 192.168.4.2 (192.168.4.2) 56(84) bytes of data.
64 bytes from 192.168.4.2: icmp_req=1 ttl=64 time=0.029 ms
64 bytes from 192.168.4.2: icmp_req=2 ttl=64 time=0.021 ms
}}}
On the GPO VM:
{{{
[lnevers@VM-2 ~]$ ping 192.168.4.1
PING 192.168.4.1 (192.168.4.1) 56(84) bytes of data.
64 bytes from 192.168.4.1: icmp_req=1 ttl=64 time=0.028 ms
64 bytes from 192.168.4.1: icmp_req=2 ttl=64 time=0.020 ms
}}}
Leave traffic running.
== 9. As Experimenter2, Request !ListResources from GPO and Utah InstaGENI. ==
As experimenter 2, lnevers2@bbn.com, collected listresources from both GPO InstaGENI racks:
{{{
$ omni.py listresources -a ig-gpo -o
$ omni.py listresources -a ig-utah -o
}}}
== 10. Define a request RSpec ==
Defined on RSpec to be used at both GPO and Utah aggregates. Rspec included one VM and one bare metal node in the GPO InstaGENI rack. Each GPO resource should have two logical interfaces and a 3rd VLAN for the local connection. Also, added two VMs at Utah and two VLANs to connect the GPO InstaGENI to the Utah InstaGENI. The experiment is captured in the file [http://groups.geni.net/geni/browser/trunk/GENIRacks/InstaGENI/Spiral4/RSpecs/AcceptanceTests/IG-EXP-4/IG-EXP-4-exp2.rspec IG-EXP-4-exp2.rspec].
== 11. Create a second slice. ==
Created the slice:
{{{
$ omni.py createslice IG-EXP-4-exp2
}}}
== 12. In the second slice, create a sliver at each InstaGENI aggregate ==
Using the RSpecs [http://groups.geni.net/geni/browser/trunk/GENIRacks/InstaGENI/Spiral4/RSpecs/AcceptanceTests/IG-EXP-4/IG-EXP-4-exp2.rspec IG-EXP-4-exp2.rspec] created a sliver at each GPO and Utah rack. First at the GPO aggregate:
{{{
lnevers2@arendia:~/gcf-2.2-rc3$ omni.py createsliver IG-EXP-4-exp2 ./IG-EXP-4-exp2.rspec -a ig-gpo
INFO:omni:Loading config file /home/lnevers2/.gcf/omni_config
INFO:omni:Using control framework pg
INFO:omni:Substituting AM nickname ig-gpo with URL http://instageni.gpolab.bbn.com/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN
INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-EXP-4-exp2 expires on 2013-01-08 17:23:16 UTC
INFO:omni:Substituting AM nickname ig-gpo with URL http://instageni.gpolab.bbn.com/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN
INFO:omni:Substituting AM nickname ig-gpo with URL http://instageni.gpolab.bbn.com/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN
INFO:omni:Creating sliver(s) from rspec file ./IG-EXP-4-exp2.rspec for slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-EXP-4-exp2
INFO:omni:Got return from CreateSliver for slice IG-EXP-4-exp2 at http://instageni.gpolab.bbn.com/protogeni/xmlrpc/am/2.0:
INFO:omni:
INFO:omni:
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed createsliver:
Options as run:
aggregate: ['ig-gpo']
framework: pg
Args: createsliver IG-EXP-4-exp2 ./IG-EXP-4-exp2.rspec
Result Summary: Got Reserved resources RSpec from instageni-gpolab-bbn-com-protogeniv2
INFO:omni: ============================================================
}}}
Then at the Utah aggregate:
{{{
lnevers2@arendia:~/gcf-2.2-rc3$ omni.py createsliver IG-EXP-4-exp2 ./IG-EXP-4-exp2.rspec -a ig-utah
INFO:omni:Loading config file /home/lnevers2/.gcf/omni_config
INFO:omni:Using control framework pg
INFO:omni:Substituting AM nickname ig-utah with URL http://utah.geniracks.net/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN
INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-EXP-4-exp2 expires on 2013-01-08 17:23:16 UTC
INFO:omni:Substituting AM nickname ig-utah with URL http://utah.geniracks.net/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN
INFO:omni:Substituting AM nickname ig-utah with URL http://utah.geniracks.net/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN
INFO:omni:Creating sliver(s) from rspec file ./IG-EXP-4-exp2.rspec for slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-EXP-4-exp2
INFO:omni:Got return from CreateSliver for slice IG-EXP-4-exp2 at http://utah.geniracks.net/protogeni/xmlrpc/am/2.0:
INFO:omni:
INFO:omni:
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed createsliver:
Options as run:
aggregate: ['ig-utah']
framework: pg
Args: createsliver IG-EXP-4-exp2 ./IG-EXP-4-exp2.rspec
Result Summary: Got Reserved resources RSpec from utah-geniracks-net-protogeniv2
INFO:omni: ============================================================
}}}
== 13. Log in to each of the end-point systems ==
Determine login information, login and send traffic to the other end-point system which shares the same VLAN.
{{{
$ readyToLogin.py IG-EXP-4-exp2 -a ig-gpo
<...>
PC's geni_status is: ready (am_status:ready)
User lnevers2 logins to PC using:
xterm -e ssh -i /home/lnevers2/.ssh/geni_key lnevers2@pc5.instageni.gpolab.bbn.com &
VM-1's geni_status is: ready (am_status:ready)
User lnevers2 logins to VM-1 using:
xterm -e ssh -p 31546 -i /home/lnevers2/.ssh/geni_key lnevers2@pc2.instageni.gpolab.bbn.com &
$ readyToLogin.py IG-EXP-4-exp2 -a ig-utah
<...>
VM-4's geni_status is: changing (am_status:changing)
User lnevers2 logins to VM-4 using:
xterm -e ssh -p 32058 -i /home/lnevers2/.ssh/geni_key lnevers2@pc5.utah.geniracks.net &
VM-5's geni_status is: changing (am_status:changing)
User lnevers2 logins to VM-5 using:
xterm -e ssh -p 32059 -i /home/lnevers2/.ssh/geni_key lnevers2@pc5.utah.geniracks.net &
}}}
Logged into each of the nodes and exchange traffic among nodes. Traffic exchange between GPO PC and GPO VM:
{{{
[lnevers2@pc ~]$ ping 10.10.1.1
PING 10.10.1.1 (10.10.1.1) 56(84) bytes of data.
64 bytes from 10.10.1.1: icmp_req=1 ttl=64 time=0.526 ms
64 bytes from 10.10.1.1: icmp_req=2 ttl=64 time=0.129 ms
}}}
== 14. Verify traffic handling per experiment ==
Verified that traffic handling was as expected per experiment. Experiment 1 nodes could not interact with experiment 2 nodes.
For example, GPO PC from experiment 2 could not interact with GPO VM 2 for experiment 1:
{{{
[lnevers2@pc ~]$ ping 192.139.4.2
PING 192.139.4.2 (192.139.4.2) 56(84) bytes of data.
From 38.104.187.117 icmp_seq=2 Destination Host Unreachable
From 38.104.187.117 icmp_seq=4 Destination Host Unreachable
}}}
Verified VM isolation and MAC address assignment. Each device had unique MAC addresses, some examples follow.
GPO PC in Experiment 2:
{{{
[lnevers2@pc ~]$ /sbin/ifconfig -a|grep HWaddr
eth0 Link encap:Ethernet HWaddr 10:60:4B:9C:47:68
eth1 Link encap:Ethernet HWaddr 10:60:4B:9C:47:6A
eth2 Link encap:Ethernet HWaddr 10:60:4B:9C:47:94
eth3 Link encap:Ethernet HWaddr 10:60:4B:9C:47:96
gre-tunnel23 Link encap:UNSPEC HWaddr C0-01-F2-90-7E-08-00-3B-00-00-00-00-00-00-00-00
gre0 Link encap:UNSPEC HWaddr 00-00-00-00-7E-08-00-3B-00-00-00-00-00-00-00-00
}}}
GPO VM-5 in Experiment 2:
{{{
[lnevers2@VM-5 ~]$ /sbin/ifconfig -a|grep HWaddr
eth999 Link encap:Ethernet HWaddr 02:64:B3:45:41:B5
gre0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-03-00-00-00-00-00-00-00-00
gre3 Link encap:Ethernet HWaddr 00:18:51:74:D9:A9
}}}
Utah VM-3 in Experiment 1:
{{{
[lnevers@VM-3 ~]$ /sbin/ifconfig -a|grep HWaddr
eth999 Link encap:Ethernet HWaddr 02:4B:3B:8D:B1:81
gre0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-03-00-00-00-00-00-00-00-00
gre2 Link encap:Ethernet HWaddr 00:18:51:D9:91:05
}}}
GPO VM-2 in Experiment 1:
{{{
[lnevers@VM-2 ~]$ /sbin/ifconfig -a|grep HWaddr
eth999 Link encap:Ethernet HWaddr 02:9D:1B:D2:16:9C
gre0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-03-00-00-00-00-00-00-00-00
gre2 Link encap:Ethernet HWaddr 00:18:51:E6:45:86
}}}
== 15. Construct and send a non-IP ethernet packet over the data plane interface ==
Used the GPO package pingPlus to send raw ethernet packets between the two nodes in Experiment 1.
On the GPO Node started the listener:
{{{
[lnevers2@pc ~/pingPlus-0.2]$ /sbin/ifconfig | grep eth1
eth1 Link encap:Ethernet HWaddr 10:60:4B:9C:47:6A
[lnevers2@pc ~/pingPlus-0.2]$ sudo ./pingPlusListener 101111
}}}
On the other GPO Node send a raw ether packet to listener:
{{{
[lnevers2@VM-3 ~/pingPlus-0.2]$ /sbin/ifconfig|grep mv4.3
mv4.3 Link encap:Ethernet HWaddr 02:B5:90:6E:43:D7
[lnevers2@VM-3 ~/pingPlus-0.2]$ sudo ./pingPlus 10:60:4B:9C:47:6A mv4.3 101111
RQ:'5064+6544' to 10:60:4b:9c:47:6a.
}}}
The listener showed the packet being received from the mac address of the sender:
{{{
[lnevers2@pc ~/pingPlus-0.2]$ sudo ./pingPlusListener 101111
RQ:5064+6544 from 2:b5:90:6e:43:d7.
RL:5064+6544=11608.
}}}
== 16. Review baseline monitoring statistics ==
Reviewed monitoring statistics for the two slices at the [https://gmoc-db.grnoc.iu.edu/protected-openid/index.pl GMOC Monitoring site], which showed the Slivers in this test:
[[Image(IG-EXP-4-slices.jpg)]]
== 17. Run test for at least 4 hours ==
Completed numerous times.
== 18. Review baseline monitoring statistics ==
Statistics show information that is up to current definition with known issues which prevent the Aggregate Slivers and Resources from being shown, this issue is investigated in instaticket:74. Issue does not block this test.
== 19. Stop traffic and delete slivers ==
As Experiementer 2, deleted slivers at both GPO and Utah aggregates:
{{{
$ omni.py deletesliver IG-EXP-4-exp2 -a ig-gpo
$ omni.py deletesliver IG-EXP-4-exp2 -a ig-utah
}}}
As Experimenter 1, deleted slivers at both GPO and Utah aggregates:
{{{
$ omni.py deletesliver IG-EXP-4-exp1 -a ig-gpo
$ omni.py deletesliver IG-EXP-4-exp1 -a ig-utah
}}}
Verified that sliver resources were released.