wiki:GENIRacksHome/ExogeniRacks/SiteConfirmationTests

ExoGENI New Site Confirmation Tests

Upon completion of the GENI Rack Site Installation Checklist for new ExoGENI sites, the GPO will also complete a series of ExoGENI (EG) Site Confirmation Tests (CT) to validate Meso-scale specific features for the newly deployed site. These confirmation tests verify experimenter access to network and compute resources at the newly installed ExoGENI site, the ability to monitor the site, and some basic administrative functions. The experiments defined for the Confirmation Tests make two assumptions: GPO personnel have been white-listed on the site rack, and Omni tools are used for all AM API operations. See the ExoGENI New Site Confirmation Test Status page for details about sites that have been tested.

EG-CT-1 - Access to New Site VM resources

The ExoGENI Confirmation Test EG-CT-1 creates a sliver at the New Site ExoGENI aggregate that includes 4 VMs all in the same rack. The VMs are used to exchange traffic. Resources are released upon completion of the test. The test topology used:

Following is the sequence of operations for this test at the new ExoGENI Rack:

  1. Get AM API version, verify that orca_version is same or newer than existing racks.
  2. List resources at the New Site local SM to determine available resources for use in this experiment. Write resource RSpec requesting 4 VMs.
  3. Create slice EG-CT-1, and then create sliver at New Site aggregate requesting the 4 VMs.
  4. Wait for the sliver to reach the "ready" state. Immediately after it reaches the ready state, determine allocated hostname. Using using a DNS server outside the aggregate, use dig command to check whether if the authoritative name servers are answering for the given domain.
  5. Login to each of the compute resources allocated to the sliver and exchange ping traffic to verify connectivity.
  6. Renew slice and sliver. Verify expiration changes.
  7. Delete sliver and verify that resources are released by checking listresources details.

EG-CT-2 - Access to New Site bare metal and VM resources

This confirmation test creates a sliver at the ExoSM requesting one bare metal node and one VM at the new ExoGENI site. The VM and bare metal resources are used to exchange traffic. Slivers are deleted upon completions. The test topology used:

Following is the sequence of AM operations for this test at the new ExoGENI Rack:

  1. List resources at the ExoSM to determine the availability of VMs and bare metal nodes at the New Site. Write RSpec.
  2. Create slice and create sliver at ExoSM requesting 1 VM and 1 bare metal node at the New ExoGENI site.
  3. Login to resources allocated to the slivers and exchange traffic.
  4. Delete sliver, and verify that resources are released by checking listresources available details.

EG-CT-3 - Multiple sites experiment

This confirmation test creates a sliver at the New ExoGENI Site and one sliver at the GPO ExoGENI requesting one VM at each site. (If the test is run to validate new software on the GPO ExoGENI site, the RENCI ExoGENI site is the second site in the test.) All VMs exchange traffic and slivers are deleted upon completion. The test topology used is shown here:

Depending on the site, there may also be regional networks in the path from the site to Internet2 or NLR included in the test.

Following is the sequence of AM operations for this test at each ExoGENI Rack:

  1. List resources at the ExoSM to determine available resources for experiment at GPO ExoGENI and at New ExoGENI Site. Write RSpecs.
  2. Create slice, and create slivers at ExoSM requesting resources.
  3. Login to resources allocated to the slivers and exchange traffic.
  4. Delete sliver and verify that resources are released by checking listresources available details.

EG-CT-4 - Multiple sites OpenFlow experiment and interoperability

Note: For sites not supporting Meso-scale, network stitching tests will be executed that use OpenFlow. See the GENI Network Stitching Confirmation Tests page for details.

This confirmation creates a sliver at the New ExoGENI Site requesting one VM on shared VLAN 1750. The New Site sliver inter-operates with OpenFlow slivers that include: a Meso-scale site, an InstaGENI site, and an Emulab OpenFlow node. Each of the VMs exchange traffic, and slivers are deleted upon completions. The test topology used is shown here:

Depending on the site, there may also be regional networks in the path from the site to the OpenFlow core networks on Internet2 or NLR included in the test. This note applies to all multi-site topologies used in the Site Confirmation Tests.

Following is the sequence of AM operations for this test at each ExoGENI Rack:

  1. List resources at the ExoSM to determine available resources at GPO ExoGENI and New Site ExoGENI racks. Do the same for the AMs at the Meso-scale site, InstaGENI site, and at the PG Utah site. Write compute resources RSpecs requesting nodes on VLAN 1750 for each aggregate.
  2. List resources at the intermediate core FOAM aggregates and at Meso-scale aggregate to determine available resources. Write remaining RSpecs.
  3. Create slice, and create slivers at core aggregates, at PG Utah, InstaGENI, and Meso-scale FOAM aggregates.
  4. Create compute resource sliver at Meso-scale site.
  5. Login to compute resources at New Site ExoGENI, GPO ExoGENI, PG Utah, InstaGENI, and Meso-scale sites and exchange traffic with each of the other remotes.
  6. Delete slivers and verify that resources are released by checking listresources details.

EG-CT-5 - Experiment Monitoring

While each of the above experiments are being run, GMOC monitoring is reviewed to ensure accuracy. Monitoring tests verify the following show up at the GMOC monitoring site for the new Site:

  1. New Site compute resources aggregate show up in the list of aggregates and provides the following details:
    1a. Aggregate Name, Type, Last Update, Version, POP, and Organization.
    1b. List of active slivers.
    1c. List of Resources.
    1d. Aggregate measurement for CPU utilization, Disk Utilization, Network Statistics.
  2. New Site FOAM resources aggregate show up in the list of aggregates and provides the following details:
    2a. Aggregate Name, Type, Last Update, Version, POP, and Organization.
    2b. List of active slivers.
    2c. List of Resources.
    2d. Aggregate measurement for CPU utilization, Disk Utilization, Network Statistics, OF Datapath and Sliver Statistics.

Note that monitoring tests verify the New Site's ability to report status monitoring data to GMOC and to validate monitoring features as they exist at the time of site deployment. More details about available monitoring features will be linked to this page as they develop.

EG-CT-6 - Administrative Tests

Confirmation Tests are executed for administrative task by GPO staff. These tests assume that the GPO staff has administrative accounts on the newly deployed rack. GPO staff will report to the site about test results and existing GPO accounts.

As a GPO Admin user, the following administrative tasks are verified:

  1. Access to each rack node (head node and worker nodes) via control network interface with SSH and execute a sudo command.
  2. Access infrastructure devices such as switches, remote PDU, and other components via control network interface with SSH.
  3. Access remote console for all rack hosts.
  4. Check FOAM version installed (admin:get-version)
  5. Check FOAM configuration (config:get-value) show that certain parameters are set, such as site.admin.email, geni.site-tag, email.from, others.
  6. Execute FOAM commands to show sliver details
  7. Check FlowVisor version installed
  8. Verify FlowVisor list of devices, active slices.
  9. Verify FlowVisor setting (fvctl getConfig <some_param>)
  10. Verify alerts for the compute resource Aggregate Manager are being reported to the GPO Tango GENI Nagios monitoring and that all alerts have status OK.
  11. Verify alerts for the FOAM Aggregate Manager are being reported to the GPO Tango GENI Nagios monitoring and that all alerts have status OK.


Email help@geni.net for GENI support or email me with feedback on this page!
Last modified 10 years ago Last modified on 07/28/14 16:26:34

Attachments (4)

Download all attachments as: .zip