wiki:GENIRacksHome/InstageniRacks/SiteConfirmationTests

Version 29 (modified by lnevers@bbn.com, 10 years ago) (diff)

--

InstaGENI New Site Confirmation Tests

Upon completion of the GENI Rack Site Installation Checklist for new InstaGENI sites, the GPO will also complete a series of InstaGENI (IG) Site Confirmation Tests (CT) to validate Meso-scale specific features for the newly deployed site. These confirmation tests verify experimenter access to network and compute resources at the newly installed InstaGENI site, the ability to monitor the site, and some basic administrative capabilities. The experiments defined for the Confirmation Tests assume that Omni tools are used for all AM API operations. Also some tests may replace the GPO InstaGENI rack with the Utah InstaGENI rack if needed. See the InstaGENI New Site Confirmation Tests Status page for details about sites that have been tested.

IG-CT-1 - Access to New Site VM resources

The InstaGENI Confirmation Test IG-CT-1 creates a sliver at the New Site InstaGENI aggregate that includes 4 VMs all in the same rack. The VMs are used to exchange traffic. Resources are released upon completion of the test. The test topology used:

Following is the sequence of operations for this test at the new InstaGENI Rack:

  1. Get AM API version, verify that ProtoGENI_version is same or newer than existing racks.
  2. List resources at the New Site AM to determine available resources for use in this experiment. Write resource RSpec requesting 4 VMs.
  3. Create a slice named IG-CT-1, and then create sliver at New Site aggregate requesting the 4 VMs.
  4. Login to each of the compute resources allocated to the sliver and exchange ping traffic to verify connectivity.
  5. Renew slice and sliver. Verify expiration changes.
  6. Delete sliver and verify that resources are released by checking listresources details.

Note: This test originally used OpenVZ VM, but after the introduction of Xen VM support all VMs were modified to Xen.

IG-CT-2 - Access to New Site Raw PC and VM resources

This confirmation test creates a sliver at the Aggregate Manager requesting a one raw pc node and one VM at the new InstaGENI site. The VM and raw pc resources are used to exchange traffic. Slivers are deleted upon completion. The test topology used:

Following is the sequence of AM operations for this test at the new InstaGENI Rack:

  1. List resources at the Aggregate Manager to determine VM and raw pc available resources for experiment. Write RSpec.
  2. Create slice, and create sliver at site aggregate requesting 1 VM and 1 raw pc at the site.
  3. Login to each of the resources allocated to the sliver and exchange traffic.
  4. Delete sliver and verify that resources are released by checking listresources available details.

IG-CT-3 - Multiple sites experiment

This confirmation test creates a sliver at the New InstaGENI Site and one sliver at the GPO requesting 1 VM at each aggregate. The VMs exchange traffic and slivers are deleted upon completions. The test topology used:

Depending on the site, there may also be regional networks in the path from the site to Internet2 or NLR included in the test.

Following is the sequence of AM operations for this test at each InstaGENI Rack:

  1. List resources at the Aggregate Managers to determine available resources for experiment at GPO InstaGENI and at New InstaGENI Site. Write RSpecs.
  2. Create slice and create slivers at each site aggregate requesting 1 VM.
  3. Login to each of the resources allocated to the slivers and exchange traffic.
  4. Delete slivers and verify that resources are released by checking listresources available details.

IG-CT-4 - Multiple sites OpenFlow experiment and interoperability

This confirmation test creates a sliver at the New InstaGENI Site with one VM on shared VLAN 1750. This New Site sliver inter-operates with OpenFlow slivers that include: a Meso-scale site, an ExoGENI site, and a PG Utah OpenFlow node. Each of the VMs exchange traffic, and slivers are deleted upon completion. The test topology used is shown here:

Depending on the site, there may also be regional networks in the path from the site to the OpenFlow core networks on Internet2 or NLR included in the test. This note applies to all multi-site topologies used in the Site Confirmation Tests.

Following is the sequence of AM operations for this test at each InstaGENI Rack:

  1. List resources at the Aggregate Managers to determine available resources at GPO InstaGENI, New Site InstaGENI, Meso-scale site, ExoGENI Site, and at the PG Utah site. Write resources RSpecs requesting shared nodes at InstaGENI and PG Utah aggregates. Write compute resource RSpec to request nodes at Meso-scale site.
  2. List resources at the intermediate core FOAM aggregates and at Meso-scale aggregate to determine available resources. Write remaining RSpecs.
  3. Create slice and create slivers at core aggregates, ExoGENI, and Meso-scale FOAM aggregates.
  4. Create slivers at the GPO InstaGENI, PG Utah, and New Site InstaGENI aggregate requesting 1 shared VLAN VM at each site.
  5. Create compute resource sliver at Meso-scale site.
  6. Login to each of the compute resources at New Site InstaGENI, GPO InstaGENI, Emulab, ExoGENI, and Meso-scale sites and exchange traffic with each of the other remotes.
  7. Delete slivers and verify that resources are released by checking listresources available details.

Note: This test originally used OpenVZ VM, after the introduction of Xen VM support all VMs were modified to Xen. In order to collect most reliable performance results, eventually the hosts were eventually modified to dedicated hosts (raw-pc).

IG-CT-5 - Experiment Monitoring

While each of the above experiments are being run, GMOC monitoring is reviewed to ensure accuracy. Monitoring tests will verify the following show up at the GMOC monitoring site for the new Site:

  1. New Site compute resources aggregate show up in the list of aggregates and provides the following details:
    1a. Aggregate Name, Type, Last Update, Version, POP, and Organization.
    1b. List of active slivers.
    1c. List of Resources.
    1d. Aggregate measurement for CPU utilization, Disk Utilization, Network Statistics.
  2. New Site FOAM resources aggregate show up in the list of aggregates and provides the following details:
    2a. Aggregate Name, Type, Last Update, Version, POP, and Organization.
    2b. List of active slivers.
    2c. List of Resources.
    2d. Aggregate measurement for CPU utilization, Disk Utilization, Network Statistics, OF Datapath and Sliver Statistics.

Note that monitoring tests verify the New Site's ability to report status monitoring data to GMOC and to validate monitoring features as they exist at the time of site deployment. More details about available monitoring features will be linked to this page as they develop.

IG-CT-6 - Administrative Tests

Confirmation Tests are also executed for administrative tasks by GPO staff. These tests assume that the GPO staff have administrative accounts on the newly deployed rack. GPO staff will report to the site about test results and existing GPO accounts.

As a GPO Admin user, the following administrative tasks are verified:

  1. Access each rack node (control node and experiment nodes) via control network interface with SSH and execute a sudo command.
  2. Access infrastructure devices such as switches, remote PDU, and other components via control network interface with SSH.
  3. Access remote console for all rack hosts.
  4. Check FOAM version installed (admin:get-version)
  5. Check FOAM configuration (config:get-value) show that certain parameters are set, such as site.admin.email, geni.site-tag, email.from, others.
  6. Execute FOAM commands to show sliver details
  7. Check FlowVisor version installed
  8. Verify FlowVisor list of devices, active slices.
  9. Verify FlowVisor setting (fvctl getConfig <some_param>)
  10. Verify alerts for the compute resource Aggregate Manager are being reported to the GPO Tango GENI Nagios monitoring and that all alerts have status OK.
  11. Verify alerts for the FOAM Aggregate Manager are being reported to the GPO Tango GENI Nagios monitoring and that all alerts have status OK.

Email help@geni.net for GENI support or email me with feedback on this page!

Attachments (6)

Download all attachments as: .zip