IG-EXP-3: InstaGENI Single Site 100 VM Test

This page captures status for the test case EG-EXP-3, which verifies the ability to support 100 VM in one rack. For overall status see the InstaGENI Acceptance Test Status page.

Last update: 08/21/12

Due to the current rack configuration a set of scenarios are being tested to capture the findings of each 100VM scenario, the following table is added for each scenario tested:

Test Scenario	Results	Notes
Scenario 1: 1 Slice with 100 VMs	Color(red,Fail)?	​instaticket:32, Not allowed with current rack configuration
Scenario 2: 2 Slices with 50 VMs each	Color(red,Fail)?	​instaticket:32, Not allowed with current rack configuration
Scenario 3: 4 Slices with 25 VMS each	Color(red,Fail)?	​instaticket:32, Not allowed with current rack configuration
Scenario 7: 5 slices with 20 VMs each	Color(green,Pass)?	Allocation:pc3=50 VMs, pc5=40 VMs, pc1=10 VMs
Scenario 6: 10 Slices with 10 VMs each	Color(green,Pass)?	Allocation:pc3=90 VMs, pc5=10 VMs
Scenario 8: 20 slices with 5 VMs each	Color(green,Pass)?	Allocation:pc5=30 VMs, pc3=40 VMs, pc1=30 VMs
Scenario 4: 50 Slices with 2 VMs each	Color(green,Pass)?	Allocation:pc3=59 VMs, pc5=42 VMs
Scenario 5: 100 Slices with 1 VM each	Color(green,Pass)?	Allocation:pc3=61 VMs, pc5=39 VMs

Test Status

This section captures the status for each step in the acceptance test plan.

Step	State	Date completed	Ticket	Comments
Step 1	Color(green,Pass)?
Step 2	Color(green,Pass)?
Step 3	Color(green,Pass)?
Step 4	Color(green,Pass)?
Step 5	Color(red,Fail)?		​instaticket:32	Cannot create 1 exp w/100 VMs with current rack configuration
Step 6	Color(orange,Blocked)?			Cannot execute due to step 5
Step 7	Color(orange,Blocked)?			Cannot execute due to step 5
Step 8	Color(orange,Blocked)?			Cannot execute due to step 5
Step 9	Color(orange,Blocked)?			Cannot execute due to step 5
Step 10	Color(orange,Blocked)?			Cannot execute due to step 5
Step 11	Color(orange,Blocked)?			Cannot execute due to step 5
Step 12	Color(orange,Blocked)?			Cannot execute due to step 5
Step 13	Color(red,Fail)?		​instaticket:32	Cannot create 2 exp w/50 VMs with current rack configuration
Step 14	Color(red,Fail)?		​instaticket:32	Cannot create 4 exp w/25 VMs with current rack configuration
Step 15	Color(green,Pass)?
Step 16	Color(green,Pass)?

State Legend	Description
Color(green,Pass)?	Test completed and met all criteria
Color(#98FB98,Pass: most criteria)?	Test completed and met most criteria. Exceptions documented
Color(red,Fail)?	Test completed and failed to meet criteria.
Color(yellow,Complete)?	Test completed but will require re-execution due to expected changes
Color(orange,Blocked)?	Blocked by ticketed issue(s).
Color(#63B8FF,In Progress)?	Currently under test.

Test Plan Steps

Experimenter1 account uses credentials for ​lnevers@bbn.com from the GPO PG.

The following nick_name is used in the omni_config:

ig-utah=,http://utah.geniracks.net/protogeni/xmlrpc/am

Step 1. As Experimenter1, request ListResources from Utah InstaGENI

Issue the following:

$ omni.py -a ig-utah listresources --api-version 2 -t GENI 3 --available -o

Step 2. Review ListResources output, and identify available resources

The RSpec show information for each non-exclusive node about the number of VMs that are possible. For example the excerpt below :

<node component_id="urn:publicid:IDN+utah.geniracks.net+node+pc5"      
      component_manager_id="urn:publicid:IDN+utah.geniracks.net+authority+cm" 
      component_name="pc5" exclusive="false"> 

      <hardware_type name="pcvm">
          <emulab:node_type type_slots="99"/>
      </hardware_type>

The above states that the node pc5, has 99 VM slots available.

Step 3. Write the Scenario 1 RSpec that requests 100 VMs evenly distributed across the experiment nodes using the default image

Created a 100 VM grid scenario.

Step 4. Create a slice

 $ omni.py -a ig-utah createslice 100vmgrid

Step 5. Create a sliver in the slice, using the RSpec defined in step 3

Created the 100 VM sliver with the following command:

 $ omni.py -a ig-utah createsliver 100vmgrid insta-100vm-grid.rspec  --api-version 2 -t GENI 3

FAILED: See ​instaticket:32

The Request RSpec for the 100VM sliver can be found ​here

Step 6. Log into several of the VMs, and send traffic to several other systems

FAILED: See ​instaticket:32

Step 7. Step up traffic rates to verify VMs continue to operate with realistic traffic loads

FAILED: See ​instaticket:32

Step 8. Review system statistics and VM isolation (does not include network isolation)

FAILED: See ​instaticket:32

Step 9. Verify that several VMs running on the same experiment node have a distinct MAC address for their interface

FAILED: See ​instaticket:32

Step 10. Verify for several VMs running on the same experiment node, that their MAC addresses are learned on the data plane switch

FAILED: See ​instaticket:32

Step 11. Review monitoring statistics and check for resource status for CPU, disk, memory utilization, interface counters, uptime, process counts, and active user counts

FAILED: See ​instaticket:32

Step 12. Stop traffic and delete sliver

FAILED: See ​instaticket:32

Step 13. Re-execute the procedure described in steps 1-12 with changes required for Scenario 2 (2 Slices with 50 VMs each)

Commands used:

 $ omni.py -a ig-utah createslice 2exp-50vm
 $ omni.py -a ig-utah createsliver 2exp-50vm insta-50vm-grid.rspec --api-version 2 -t GENI 3

First sliver completed, checked for node allocation:

$ omni.py -a ig-utah sliverstatus 2exp-50vm --api-version 2 -t GENI 3 -o

Create a second experiment with 50 VMs:

FAILED: See ​instaticket:32

The Request RSpec for the 50VM sliver can be found ​here

Step 14. Re-execute the procedure described in steps 1-12 with changes required for Scenario 3 (4 Slices with 25 VMS each)

With the current rack configuration, this scenario is not allowed, only 3 slices set up:

 $ omni.py -a ig-utah createslice 4exp-25vm
 $ omni.py -a ig-utah createsliver 4exp-25vm --api-version 2 -t GENI 3 ./instarspec/insta-25vm-grid.rspec 
 $ omni.py -a ig-utah createslice 4exp-25vma
 $ omni.py -a ig-utah createsliver 4exp-25vma --api-version 2 -t GENI 3 ./instarspec/insta-25vm-grid.rspec 
 ASSIGN FAILED:
  *** 25 nodes of type pcvm requested, but only 20 available nodes of type pcvm found
*** Type precheck failed!

The above experiments resulted in the following allocation:

• pc3=30 VMs
• pc5=30 VMS
• pc1=5 VMs
• pc4=5 VMs
• pc2=5 VMs

The Request RSpec for the 25VM slivers can be found ​here

Step 15. Re-execute the procedure described in steps 1-12 with changes required for Scenario 4 (50 Slices with 2 VMs each)

Successfully execute this scenario in 06/01/12 and was able to login to several of the VMs and exchange traffic. Final allocation distribution was applied as follows on the two pcshared nodes:

• 58 VMs on pc3
• 42 VMs on pc5

The Request RSpec for the 2VM slivers can be found ​here

08/21/12 test run

Re-executed procedure on 08/21/12 and was once again able to reserve 100 VMS (50 slices with 2 VMs each). Final allocation distribution was applied as follows on the three pcshared nodes:

• 38 VMs on pc5
• 34 VMs on pc3
• 28 VMs on pc1.

Extended the test to 150 WMs could be reserved with 75 slivers, and added 25 more slices to the 08/21/12 run. The resulting allocation:

• 42 VMs on pc5
• 50 VMs on pc3
• 58 VMs on pc1.

Ran the GCF script readyToLogin.py to determine access information for the 150 VMs allocated:

j=1 
while [ $j -le 75 ]; 
do 
./examples/readyToLogin.py 2vmslice$j -a ig-utah 
sleep 2; 
j=`expr $j + 1`
done

The readyToLogin script is part of the GCF examples directory and provides login information for active slivers, which looks as follows:

 $ ./examples/readyToLogin.py 2vmslice1 -a ig-utah -q
   ================================================================================
   Aggregate [http://utah.geniracks.net/protogeni/xmlrpc/am] has a ProtoGENI sliver.


   pc3.utah.geniracks.net's geni_status is: ready
  Login using:
	xterm -e ssh -i ~/.ssh/id_rsa lnevers@pc3.utah.geniracks.net -p 30778 &

   pc3.utah.geniracks.net's geni_status is: ready
   Login using:
	xterm -e ssh -i ~/.ssh/id_rsa lnevers@pc3.utah.geniracks.net -p 30779 &

  ================================================================================

Using the information from the readyToLogin script output logged into all 150 VMs. Was only able to login to 112 of the reserved nodes, the remaining 38 did not failed at the ssh login.

Step 16. Re-execute the procedure described in steps 1-12 with changes required for Scenario 5 (100 Slices with 1 VM each)

Successfully execute this scenario and was able to login to several of the VMs. Final allocation distribution was applied as follows on the two pcshared nodes:

• 39 VMs on pc5
• 61 VMs on pc3

The Request RSpec for the 1 VM slivers can be found ​here

Additional Scenarios

Scenario 7: 5 slices with 20 VMs each

Successfully execute this scenario and was able to login to several of the VMs. Final allocation distribution was applied as follows on the two pcshared nodes:

• 30 VMs on pc5
• 40 VMs on pc3
• 30 VMs on pc1

The Request RSpec for the 5 VMs on 1 LAN slivers can be found ​here

NOTE:

An additional scenario was executed, "Scenario 7: 5 slices with 20 VMs each" which was successfully completed. The Request RSpec for the 20 VM slivers can be found ​here