= EG-EXP-6: ExoGENI and Meso-scale Multi-site OpenFlow Acceptance Test = This page captures status for the test case EG-EXP-6, which verifies ExoGENI rack interoperability with other meso-scale GENI sites. For overall status see the [wiki:GENIRacksHome/ExogeniRacks/AcceptanceTestStatus ExoGENI Acceptance Test Status] page. = Test Status = This section captures the status for each step in the acceptance test plan. || '''Step''' || '''State''' || ''' Ticket ''' || '''Comments''' || || Step 1 ||[[Color(yellow,Complete)]]|| || || || Step 2 ||[[Color(yellow,Complete)]]|| || || || Step 3 ||[[Color(yellow,Complete)]]|| || || || Step 4 ||[[Color(yellow,Complete)]]|| || || || Step 5 ||[[Color(yellow,Complete)]]|| || || || Step 6 ||[[Color(yellow,Complete)]]|| || || || Step 7 ||[[Color(yellow,Complete)]]|| || || || Step 8 ||[[Color(yellow,Complete)]]|| || || || Step 9 ||[[Color(yellow,Complete)]]|| exoticket:94 || Exception while reserving nodes on VLAN 1750 || || Step 10 ||[[Color(yellow,Complete)]]|| || Had to replace bare metal with VM, see step for detail || || Step 11 || || || || || Step 12 || || || || || || Step 13 || || || || || || Step 14 || || || || || || Step 15 || || || || || || Step 16 || || || || || || Step 17 || || || || || || Step 18 || || || || || || Step 19 || || || || || || Step 20 || || || || || || Step 21 || || || || || || Step 22 || || || || || || Step 23 || || || || || || Step 24 || || || || || || Step 25 || || || || || || Step 26 || || || || || || Step 27 || || || || || || Step 28 || || || || || || Step 29 || || || || || || Step 30 || || || || || || Step 31 || || || || || || Step 32 || || || || || || Step 33 || || || || || || Step 34 || || || || || || Step 35 || || || || || || Step 36 || || || || || || Step 37 || || || || || [[BR]] || '''State Legend''' || '''Description''' || || [[Color(green,Pass)]] || Test completed and met all criteria || || [[Color(#98FB98,Pass: most criteria)]] || Test completed and met most criteria. Exceptions documented || || [[Color(red,Fail)]] || Test completed and failed to meet criteria. || || [[Color(yellow,Complete)]] || Test completed but will require re-execution due to expected changes || || [[Color(orange,Blocked)]] || Blocked by ticketed issue(s). || || [[Color(#63B8FF,In Progress)]] || Currently under test. || [[BR]] = Test Plan Steps = This test was August 7, 2012. The pgeni.gpolab.bbn.com slice authority is used for the credentials and the following aggregate manager nick_names were defined in the omni_config: {{{ #ExoGENI Compute and OF Aggregates Managers exobbn=,https://bbn-hn.exogeni.net:11443/orca/xmlrpc exorci=,https://rci-hn.exogeni.net:11443/orca/xmlrpc exosm=,https://geni.renci.org:11443/orca/xmlrpc of-exobbn=,https://bbn-hn.exogeni.net:3626/foam/gapi/1 of-exorci=,https://rci-hn.exogeni.net:3626/foam/gapi/1 #Meso-scale Compute and OF Aggregates Managers of-bbn=,https://foam.gpolab.bbn.com:3626/foam/gapi/1 of-clemson=,https://foam.clemson.edu:3626/foam/gapi/1 of-i2=,https://foam.net.internet2.edu:3626/foam/gapi/1 of-rutgers=,https://nox.orbit-lab.org:3626/foam/gapi/1 plc-bbn=,http://myplc.gpolab.bbn.com:12346/ plc-clemson=,http://myplc.clemson.edu:12346/ }}} == 1. As Experimenter1, request !ListResources from BBN ExoGENI, RENCI ExoGENI, and FOAM at NLR Site. == Requested listresources from each of the FOAM aggregates: {{{ $ omni.py listresources -a of-exobbn -o $ omni.py listresources -a of-bbn -o $ omni.py listresources -a of-nlr -o $ omni.py listresources -a of-i2 -o $ omni.py listresources -a of-exorci -o }}} == 2. Review !ListResources output from all AMs == Requested listresources from each of the GENI AM aggregates: {{{ $ omni.py listresources -a plc-bbn $ omni.py listresources -a plc-clemson $ omni.py listresources -a pg2 $ omni.py listresources -a exobbn $ omni.py listresources -a exorci }}} == 3. Define a request RSpec for a VM at the BBN ExoGENI. == Defined an RSpec for one VM on the shared VLAN 1750 in the BBN ExoGENI rack: [EG-EXP-6-exp1-exobbn.rspec] == 4. Define a request RSpec for a VM at the RENCI ExoGENI. == Define an RSpec for one VM on the shared VLAN 1750 in the RENCI ExoGENI rack: [EG-EXP-6-exp1-exorci.rspec] == 5. Define request RSpecs for !OpenFlow resources from BBN FOAM to access GENI !OpenFlow core resources. == Defined an !OpenFlow RSpec for the FOAM controllers in each rack: - BBN Rack RSpec: [EG-EXP-6-exp1-openflow-exobbn.rspec] - RENCI Rack RSpec: [EG-EXP-6-exp1-openflow-exorci.rspec] The BBN ExoGENI rack connects to the GENI backbone via the BBN GPO Lab OpenFlow switch named poblano. In order for this scenario to work OpenFlow must be configured also on the poblano switch to allow the BBN ExoGENI rack OF traffic to get to the OF GENI core. The following OpenFlow RSpec was defined for the BBN site switch poblano: [EG-EXP-6-exp1-openflow-bbn.rspec] == 6. Define request RSpecs for !OpenFlow core resources at NLR FOAM. == Defined an !OpenFlow RSpec for the NLR FOAM controller: [EG-EXP-6-exp1-openflow-nlr.rspec] == 7. Create the first slice == Created the first slice: {{{ $ omni.py createslice EG-EXP-6-exp1 }}} == 8. Create a sliver in the first slice at each AM, using the RSpecs defined above. == {{{ $omni.py -a exobbn createsliver EG-EXP-6-exp1 EG-EXP-6-exp1-exobbn.rspec $omni.py -a of-bbn createsliver EG-EXP-6-exp1 EG-EXP-6-exp1-openflow-bbn.rspec $omni.py -a of-nlr createsliver EG-EXP-6-exp1 EG-EXP-6-exp1-openflow-nlr.rspec $omni.py -a exorci createsliver EG-EXP-6-exp1 EG-EXP-6-exp1-exorci.rspec }}} Here are of the Rspecs used: [EG-EXP-6-exp1-exobbn.rspec], [EG-EXP-6-exp1-openflow-bbn.rspec], [EG-EXP-6-exp1-openflow-nlr.rspec], and [EG-EXP-6-exp1-exorci.rspec]. == 9. Log in to each of the systems, verify IP address assignment. Send traffic to the other system, leave traffic running. == Determine the status of the OpenFlow slivers, check the sliverstatus for each to make sure that they have been approved. Note that 'geni_status' is 'ready' when the sliver is approved. If the OpenFlow sliver is waiting for approval the 'geni_status' will be 'configuring: {{{ $ omni.py -a of-bbn sliverstatus EG-EXP-6-exp1 $ omni.py -a of-nlr sliverstatus EG-EXP-6-exp1 }}} Determine compute resources allocated to each sliver in the ExoGENI racks. First make sure the sliverstatus is ready: {{{ $ omni.py -a exobbn sliverstatus EG-EXP-6-exp1 $ omni.py -a exorci sliverstatus EG-EXP-6-exp1 }}} Once the slivers are ready get the list of hosts allocated with Omni: {{{ $ omni.py -a exobbn listresources EG-EXP-6-exp1 -o $ omni.py -a exorci listresources EG-EXP-6-exp1 -o $ egep hostname EG-EXP-6-exp1-rspec-bbn-hn-exogeni-net-11443-orca.xml $ egrep hostname EG-EXP-6-exp1-rspec-rci-hn-exogeni-net-11443-orca.xml }}} == 10. As Experimenter2, define a request RSpec for one VM and one bare metal node at BBN ExoGENI. == Defined an RSpec for one VM and one bare metal node on the shared VLAN 1750 in the BBN ExoGENI rack [EG-EXP-6-exp2-exobbn.rspec]. Attached as a reference, even though it could not be used for this test case as described. Two options were available because the Bare Metal node is only available via the ExoSM: a. Replace the BBN SM reservation with a reservation via the ExoSM, but this will cause the next step to fail. The RENCI SM reservation will report that a sliver by that name already exists, due to the ExoSM request. To get around this problem, one option is to replace the bare metal node with a VM, so that the requests could be submitted as expected to each of the local SMs. b. Replace the BBN SM reservation with a reservation via the ExoSM, but create a new slice and sliver name for the RCI portion. So the BBN resources are reserved via the ExoSM, and the RENCI resources are reserved via the local RENCI SM. Option ''a'' was chosen for this test run, and the new RSpec used is [[EG-EXP-6-exp2-exobbn-mod.rspec]. == 11. Define a request RSpec for two VMs on the same worker node at RENCI ExoGENI. == Defined an RSpec for two VMs on the shared VLAN 1750 in the RENCI rack: [EG-EXP-6-exp2-exobbn.rspec] == 12. Define request RSpecs for !OpenFlow resources from GPO FOAM to access GENI !OpenFlow core resources. == The experiment uses the flows defined by the previous experiment, as defined in [EG-EXP-6-exp1-openflow-bbn.rspec] == 13. Define request RSpecs for !OpenFlow core resources at NLR FOAM == The experiment uses the flows defined by the previous experiment, as defined in [EG-EXP-6-exp1-openflow-nlr.rspec] == 14. Create a second slice == {{{ $ omni.py createslice EG-EXP-6-exp2 }}} == 15. Create a sliver in the second slice at each AM, using the RSpecs defined above == Because of the issues described in step 10, had to request to VMs on the shared VLAN 1750 at the BBN SM. The request: {{{ $ omni.py -a exobbn createsliver EG-EXP-6-exp2 EG-EXP-6-exp2-exobbn-mod.rspec }}} Created a sliver at RENCI ExoGENIrack aggregate: {{{ $ omni.py -a exorci createsliver EG-EXP-6-exp2 ./exorspec/EG-EXP-6/EG-EXP-6-exp2-exorci.rspec }}} == 16. Log in to each of the systems in the slice, and send traffic to each other systems; leave traffic running Determine compute resources allocated to each sliver in the ExoGENI racks. First make sure the sliverstatus is ready: {{{ $ omni.py -a exobbn sliverstatus EG-EXP-6-exp2 $ omni.py -a exorci sliverstatus EG-EXP-6-exp2 }}} Once the slivers are ready get the list of hosts allocated with Omni: {{{ $ omni.py -a exobbn listresources EG-EXP-6-exp2 -o $ omni.py -a exorci listresources EG-EXP-6-exp2 -o $ egep hostname EG-EXP-6-exp2-rspec-bbn-hn-exogeni-net-11443-orca.xml $ egrep hostname EG-EXP-6-exp2-rspec-rci-hn-exogeni-net-11443-orca.xml }}} == 17. As Experimenter3, request !ListResources from BBN ExoGENI, GPO FOAM, and FOAM at Meso-scale Site == For the initial run only had 2 user account available, so the Experimenter3 steps are executed with the same credentials as experiment 1. Also the WAPG resource at Rutgers is available via the Emulab PG aggregate. Executed the following to get resources for each aggregate: {{{ $ omni.py listresources -a exobbn -o $ omni.py listresources -a of-bbn -o $ omni.py listresources -a of-i2 -o $ omni.py listresources -a of-rutgers -o $ omni.py listresources -a pg2 --api-version 2 -t GENI 3 -o }}} == 18. Review !ListResources output from all AMs == Reviewed content of advertisement RSpecs for each of the aggregates polled in the previous step. == 19. Define a request RSpec for a VM at the BBN ExoGENI == Defined RSpec for one VM on the shared VLAN 1750 in the BBN ExoGENI rack: [EG-EXP-6-exp3-exobbn.rspec] == 20. Define a request RSpec for a compute resource at the GPO Meso-scale site == Defined RSpec for one PG Resource at the GPO PG site: [EG-EXP-6-exp3-bbn-pgeni.rspec] == 21. Define a request RSpec for a compute resource at a Meso-scale site == Defined RSpec for one WAPG Resource at the Rutgers site [EG-EXP-6-exp3-rutgers-wapg.rspec] == 22. Define request RSpecs for !OpenFlow resources to allow connection from OF BBN ExoGENI to Meso-scale OF sites GPO (NLR) and Rutgers Sites (I2) == Defined the !OpenFlow RSpecs for the sites below, note that the BBN OF RSpec overlaps with the one in experiment 1 and is being used here: - Rutgers [EG-EXP-6-exp3-openflow-rutgers.rspec] - BBN [EG-EXP-6-exp1-openflow-bbn.rspec] - Internet2 [EG-EXP-6-exp3-openflow-i2.rspec] == 23. Create a third slice == {{{ $ omni.py createslice EG-EXP-6-exp3 }}} == 24. Create a sliver that connects the Internet2 Meso-scale !OpenFlow site to the BBN ExoGENI Site, and the GPO Meso-scale site == As stated in step 22, the BBN OpenFlow flows were requested as part of experiment1 and are not re-executed here. The remaining requests are as follows: {{{ $ omni.py -a exobbn createsliver EG-EXP-6-exp3 EG-EXP-6-exp3-exobbn.rspec $ omni.py -a pgeni createsliver EG-EXP-6-exp3 EG-EXP-6-exp3-bbn-pgeni.rspec $ omni.py -a of-i2 createsliver EG-EXP-6-exp3 EG-EXP-6-exp3-openflow-i2.rspec $ omni.py -a pg2 createsliver EG-EXP-6-exp3 --api-version 2 -t GENI 3 EG-EXP-6-exp3-rutgers-wapg.rspec $ omni.py -a of-rutgers createsliver EG-EXP-6-exp3 EG-EXP-6-exp3-openflow-rutgers.rspec }}} == 25. Log in to each of the compute resources in the slice == Determine compute resources allocated to the sliver in the BBN ExoGENI rack. First make sure the sliverstatus is ready and once the sliver is ready get the host allocated with Omni: {{{ $ omni.py -a exobbn sliverstatus EG-EXP-6-exp3 $ omni.py -a exobbn listresources EG-EXP-6-exp3 -o $ egep hostname EG-EXP-6-exp3-rspec-bbn-hn-exogeni-net-11443-orca.xml < }}} Determine the resources allocated to the WAPG Rutgers node: {{{ $ omni.py -a pg2 sliverstatus EG-EXP-6-exp3 --api-version 2 -t GENI 3 }}} Determine the resources allocated to the GPO PG node: {{{ $ omni.py -a pgeni sliverstatus EG-EXP-6-exp3 $ omni.py -a pgeni listresources EG-EXP-6-exp3 -o $ egrep "hostname" EG-EXP-6-exp3-rspec-pgeni-gpolab-bbn-com.xml Login to each node in the slice, onfigure data plane network interfaces on any non-ExoGENI resources as necessary, and send traffic to each of the other systems; leave traffic running. == 26. Verify that all three experiments continue to run == Verify that each experiment is running without impacting each other's traffic and verify that data is exchanged over the path along which data is supposed to flow == 27. Review baseline, GMOC, and monitoring statistics == == 28. As site administrator, identify all controllers that the BBN ExoGENI !OpenFlow switch is connected to == == 29. As Experimenter3, verify that traffic only flows on the network resources assigned to slivers as specified by the controller == == 35. Run test for at least 4 hours == == 36. Review monitoring statistics and checks as above == == 37. Stop traffic and delete slivers ==