= EG-EXP-3: ExoGENI Single Site 100 VM Test = This page captures status for the test case EG-EXP-3, which verifies the ability to support 100 VM in one ExoGENI rack. For overall status see the [wiki:GENIRacksHome/ExogeniRacks/AcceptanceTestStatus ExoGENI Acceptance Test Status] page. ''Last update: 12/06/12'' Due to the rack configuration a set of scenarios are being tested to capture scaling to 100 VMs, the following table captures scenario tested: || ''' Test Scenario''' ||'''Results''' || '''Notes''' || ||Test 1: 10 slivers with 10 VMs each||[[Color(#63B8FF,In Progress)]] || 89VMs ok, 11 VMs failed, exoticket:134 || ||Test 2: 4 slivers w/20VMS +2 slivers w/10 VMs||[[Color(#63B8FF,In Progress)]] || 88VMs ok, 12 VMs failed, exoticket:134 || ||Test 3: 20 slivers with 5 VMs each ||[[Color(#63B8FF,In Progress)]] || 90VMs ok, 10 VMs failed, exoticket:134 || ||Test 4: 100 Sliver with 1 VM ||[[Color(#63B8FF,In Progress)]] ||41 VMs ok, BBN SM failed, exoticket:135 || ||Test 5: 2 slivers with 50 VMs each ||[[Color(#63B8FF,In Progress)]] || Allowed with current rack configuration, but untested || = Test Status = This section captures the status for each step in the acceptance test plan. || '''Step''' || '''State''' || ''' Tickets''' || '''Comments''' || || Step 1 ||[[Color(yellow,Complete)]] || || Rack 50/50 resource allocation forces use of ExoSM and BBN SM || || Step 2 ||[[Color(yellow,Complete)]] || || || || Step 3 ||[[Color(yellow,Complete)]] || || || || Step 4 ||[[Color(yellow,Complete)]] || || || || Step 5 ||[[Color(#63B8FF,In Progress)]]|| ||Step modified to cover 2 slivers with 50 VMs each|| || Step 6 ||[[Color(yellow,Complete)]] || || || || Step 7 ||[[Color(#63B8FF,In Progress)]]|| ||Not executed in initial test runs|| || Step 8 ||[[Color(#63B8FF,In Progress)]]|| ||Not executed in initial test runs|| || Step 9 ||[[Color(#63B8FF,In Progress)]]|| ||Not executed in initial test runs|| || Step 10 ||[[Color(yellow,Complete)]] || || || || Step 11 ||[[Color(yellow,Complete)]] || || || || Step 12 ||[[Color(#63B8FF,In Progress)]]|| ||Not executed in initial test runs|| || Step 13 ||[[Color(yellow,Complete)]] || || || || Step 14 ||[[Color(#63B8FF,In Progress)]]|| ||exoticket:134|| || Step 15 ||[[Color(#63B8FF,In Progress)]]|| ||exoticket:134|| || Step 16 ||[[Color(#63B8FF,In Progress)]]|| ||exoticket:134|| || Step 17 ||[[Color(#63B8FF,In Progress)]]|| ||exoticket:135|| || Step 18 || || || || [[BR]] || '''State Legend''' || '''Description''' || || [[Color(green,Pass)]] || Test completed and met all criteria || || [[Color(#98FB98,Pass: most criteria)]] || Test completed and met most criteria. Exceptions documented || || [[Color(red,Fail)]] || Test completed and failed to meet criteria. || || [[Color(yellow,Complete)]] || Test completed but will require re-execution due to expected changes || || [[Color(orange,Blocked)]] || Blocked by ticketed issue(s). || || [[Color(#63B8FF,In Progress)]] || Currently under test. || [[BR]] = Test Plan Steps = The Omni configuration file used in this test case includes the following nick_names for ExoGENI Aggregates: {{{ eg-gpo=,https://bbn-hn.exogeni.net:11443/orca/xmlrpc eg-renci=,https://rci-hn.exogeni.net:11443/orca/xmlrpc eg-sm=,https://geni.renci.org:11443/orca/xmlrpc }}} Testing uses '''' for the resource requests, which generates a 1 core VM with 128 MBytes of RAM and 3 GBytes of disk space. For a full list of available resource types and associated cores, RAM and Disk space allocation see the following [https://wiki.exogeni.net/doku.php?id=public:experimenters:resource_types:start resource type table] at the ExoGENI wiki site. == Step 1. Request !ListResources from BBN ExoGENI. == The following command is executed to collect resource from the BBN ExoGENI rack: {{{ $ omni.py -a eg-bbn listresources -o }}} The rack resource allocation is 50% of resources for the local rack SM and 50% for the ExoSM, therefore one must collect listresources on both to get the complete list of resources available within an ExoGENI Rack. {{{ $ omni.py -a eg-sm listresources -o }}} '''Note:''' Some of the tests completed on this page were executed on the RENCI ExoGENI rack. == Step 2. Review !ListResources output, and identify available resources. == The ExoGENI RSpec provides an indication of the potentially available VMs in the "type_slots" element and for the . The listresources output file generated from previous step for the ExoSM named rspec-geni-renci-org-11443-orca.xml shows: {{{ }}} The listresources output file for the BBN SM named rspec-bbn-hn-exogeni-net-11443-orca.xml shows: {{{ }}} == Step 3. Write a RSpec that requests 100 VMs evenly distributed across the worker nodes using the default image. == This step cannot be executed for 100 VM in one sliver due to the current rack resource allocation, which allocates 50% of the resources to the local aggregate manager (BBN SM) and 50% of the resources to the global aggregate manager (ExoSM). The test case step was modified to create RSpecs which are used at each local and global aggregate manager. == Step 4. Create a slice. == The slice is created as follows: {{{ $ omni.py createslice EG-EXP-3 }}} == Step 5. Create a sliver in the slice, using the RSpec defined in step 3. == This step cannot be executed for 100 VM in one sliver, due to the resource allocation which is 50/50 between the local BBN SM and the global ExoSM. Based on the allocation, the test case description was modified to create two slivers, where each has 50 VMs in the BBN Rack, with one request to the BBN SM and one request to the ExoSM. '''NOTE:''' Rspecs with 50 VMs have not yes been tested, but other scenarios have been tested and are covered in this page. To set up a sliver in the BBN rack that is created via the ExoSM, this type of command is used: {{{ $ omni.py -a eg-sm createsliver EG-EXP-3 EG-EXP-3.rspec }}} To set up a sliver in the BBN rack that is created via the BBN SM: {{{ $ omni.py -a eg-gpo createsliver EG-EXP-3 EG-EXP-3.rspec }}} '''Note:''' The same sliver name and Rspec can be used concurrently via BBN Sm and ExoSM, and that is the approach used in all tests described in this page. == Step 6. Log into several of the VMs, and send traffic to several other systems. == Was able to login to some nodes after the slivers were ready. == Step 7. Step up traffic rates to verify VMs continue to operate with realistic traffic loads. == == Step 8. Review system statistics and VM isolation (does not include network isolation) == == Step 9. Review monitoring statistics and check for resource status for CPU, disk, memory utilization, interface counters, uptime, process counts, and active user counts. == == Step 10. Verify that several VMs running on the same worker node have a distinct MAC address for their interface. == Allocated VMs have unique MAC addresses. == Step 11. Verify for several VMs running on the same worker node, that their MAC addresses are learned on the data plane switch. == The successful exchange of traffic on the dataplane interface shows that are learned on data plane switch. == Step 12. Review monitoring statistics and check for resource status for CPU, disk, memory utilization, interface counters, uptime, process counts, and active user counts. == This step is replaced by iperf for ExoGENI test cases. Not yet completed during scalability tests. == Step 13. Stop traffic and delete sliver. == Slivers are deleted as follows throughout testing: {{{ $ omni.py deletesliver -a eg-sm EG-EXP-3 $ omni.py deletesliver -a eg-bbn EG-EXP-3 }}} == Step 14. Re-execute procedure in steps 1-13 with changes required for Test 1: 10 slivers with 10 VMs each == This experiment was setup at the RENCI ExoGENI rack using the RSpec [http://groups.geni.net/geni/browser/trunk/GENIRacks/ExoGENI/Spiral4/Rspecs/AcceptanceTests/EG-EXP-3/exo-10vm-rci.rspec exo-10vm-rci.rspec] for both RENCI SM and ExoSM. The following slices were created: {{{ $ omni.py createslice ln-10vm1 $ omni.py createslice ln-10vm2 $ omni.py createslice ln-10vm3 $ omni.py createslice ln-10vm4 $ omni.py createslice ln-10vm5 $ omni.py createslice ln-10vm6 $ omni.py createslice ln-10vm7 $ omni.py createslice ln-10vm8 $ omni.py createslice ln-10vm9 $ omni.py createslice ln-10vm10 }}} The slivers were created at the RENCI SM: {{{ $ omni.py createsliver -a eg-renci ln-10vm1 exo-10vm-rci.rspec $ omni.py createsliver -a eg-renci ln-10vm2 exo-10vm-rci.rspec $ omni.py createsliver -a eg-renci ln-10vm3 exo-10vm-rci.rspec $ omni.py createsliver -a eg-renci ln-10vm4 exo-10vm-rci.rspec $ omni.py createsliver -a eg-renci ln-10vm5 exo-10vm-rci.rspec }}} The following slivers were created at the ExoSM: {{{ $ omni.py createsliver -a eg-sm ln-10vm6 exo-10vm-rci.rspec $ omni.py createsliver -a eg-sm ln-10vm7 exo-10vm-rci.rspec $ omni.py createsliver -a eg-sm ln-10vm8 exo-10vm-rci.rspec $ omni.py createsliver -a eg-sm ln-10vm9 exo-10vm-rci.rspec $ omni.py createsliver -a eg-sm ln-10vm10 exo-10vm-rci.rspec }}} '''Results:''' 89VMs ok, 11 VMs failed, exoticket:134 In initial run no traffic was exchanged, and logins were only tried on a few nodes. Slivers were deleted upon completion. == Step 15. Re-execute procedure in steps 1-13 with changes required for Test 2: 4 slivers w/20VMS +2 slivers w/10 VMs == In initial run no traffic was exchanged, and logins were only tried on a few nodes. Slivers were deleted upon completion == Step 16. Re-execute procedure in steps 1-13 with changes required for Test 3: 20 slivers with 5 VMs each == In initial run no traffic was exchanged, and logins were only tried on a few nodes. Slivers were deleted upon completion. == Step 17. Re-execute procedure in steps 1-13 with changes required for Test 4: 100 Sliver with 1 VM == In initial run no traffic was exchanged, and logins were only tried on a few nodes. Slivers were deleted upon completion. == Step 18. Re-execute procedure in steps 1-13 with changes required for Test 6: 2 slivers with 50 VMs each == Not yet tested!