Changes between Initial Version and Version 1 of GeniNetworkStitchingTestPlan

Timestamp:

05/03/13 10:17:57 (11 years ago)

Author:

lnevers@bbn.com

Comment:

--

GeniNetworkStitchingTestPlan

@@ +1 @@
+[[PageOutline]]
+
+= GENI Network Stitching Test Plan =
+
+This page provides an outline of the testing for the GENI Network Stitching that is currently planned in the [http://geni.maxgigapop.net/twiki/bin/view/GENI/NetworkStitching Network Stitching Architecture] discussion page.   For testing described in this plan, it is assumed that the GCF tool stitcher, can be used to execute tests using credentials from the [https://www.pgeni.gpolab.bbn.com GPO PG Clearinghouse].  Upon completion of this test plan, some tests will be re-executed using [http://groups.geni.net/geni/wiki/GeniClearinghouse GENI Clearinghouse] credentials via the [https://panther.gpolab.bbn.com GENI Clearinghouse Portal].  If support for stitching is available, Flack may also be used to execute some tests in this plan. Overall goals of this plan are:
+    - Verify that experimenters can get the resources they request.
+    - Verify that operators can tell what resources are in use, and determine who is using the resources.
+    - Verify that the aggregates present an accurate picture of stitching resources available.
+    - Validate representative network stitching topologies and collect sample measurements for stitching those topologies
+
+
+= Stitching Functional Tests =
+
+The functional tests in this plan are based on the ''GENI Network Stitching Architecture'' [http://geni.maxgigapop.net/twiki/pub/GENI/NetworkStitching/geni-network-stitching-architecture-overview.pdf overview] and [http://geni.maxgigapop.net/twiki/pub/GENI/NetworkStitching/geni-network-stitching-architecture-example.pdf example] documents.  Additionally, the [http://geni.maxgigapop.net/twiki/pub/GENI/NetworkStitching/geni-stitching-component-details-example.pdf GENI Network Stitching Component Design] document was referenced in defining the functional network stitching tests. See the [wiki:GENINetworkStitchingTestPlan#References Reference] section for the full list of documents used to define this plan.
+
+The GENI Network Stitching architecture uses six major components (Stitching Resource Element, Common Stitching Topology Schema, GENI Stitching Topology Service, GENI Stitching Path Computation Function, GENI Stitching Workflow Function, and  GENI AM API Network Stitching Extensions) to provide stitching operations. The functional tests do not focus on the individual components, instead they focus on accessing stitching features to create inter-aggregate VLAN network connections via GENI AM API at available substrate aggregates managers.  GPO and MAX developers are testing stitching software components in parallel.
+
+== GENI AM API Functions ==
+
+The following GENI AM API functions are to be verified in managing stitching resources using GENI AM API V3. It is expected that GCF tool (stitcher.py) will negotiate to V2 for aggregates that do not support V3.
+      * List Resources for aggregate 
+      * Getversion 
+      * Create Sliver (allocate, provion, perform operational action start/stop/restart)
+      * List resources for sliver (describe)
+      * Sliver status  
+      * Renew sliver 
+      * Delete sliver
+      * Shutdown Sliver
+
+
+== GENI RSpec Support ==
+
+GENI RSpec support tests will validate that each of the RSpec (Advertisement, Manifest, and Request) provide complete and accurate information.
+
+'''Stitching Extensions:''' [[BR]]
+
+    - Request RSpec with stitching extension -> to verify that extension is used
+    - Request RSpec without stitching extension -> to verify that one is created
+    - Stitching Path support:
+      * Point-to-point path only
+      * Multiple point-to-point 
+
+'''Requests RSpec validation:''' [[BR]]
+
+   - RSpec Versions support (V3)
+   - Routing profile options to include or exclude:
+     * hop_include_list
+     * hop_exclude_list
+     * VLAN tags (exclude only)
+  - Verify '''only''' bandwidth and VLAN are negotiable.
+
+
+'''RSpec Input Validation:'' [[BR]]
+
+  - Request link with only 1 component_manager tag
+  - Request link with 2 interface_ref elements but no component_manager elements
+  - Request link with 2 local interface_ref elements and 1 remote interface_ref element
+  - Request link with an interface_ref whose client_id is not the ID for any interface in the RSpec
+  - Request stitching extension with a path that has 0 hops
+  - Request capacity that is below minimum and above maximum reservable Capacity
+  - Invalid Path options (not sure what this is and how to do it ?)
+  - Invalid Routing profiles (not sure what this is and how to do it ?)
+  - Include/exclude invalid VLANs
+
+ 
+'''Advertisement RSpecs:''' [[BR]]
+
+Advertisement RSpecs will be reviewed to verify that they accurately capture:
+ - Stitching elements (nodes, ports, links, capacity)
+ - VLAN IDs and their availability
+ - VLAN Bandwidth capability
+ 
+
+== Negative and Boundary tests ==
+
+This is a starting list of scenarios that will validate negative and boundary conditions:
+ - At ION, InstaGENI and ExoGENI, allocate resources that are not advertised, verify failure.
+ - At ION, InstaGENI and ExoGENI, allocate VLAN tags that are already in use, verify failure.
+ - Kill an ION circuit manually, verify recovery and logging of event.
+ - Request a VLAN that is already in use, verify handling.
+ - Create a request race condition where two slices (Slice1 and Slice2) request the same resources (AM1 <->VLAN1<->AM2), but Slice1 gets VLAN1 at AM1 and Slice2 gets VLAN1 at AM2. Verify results tools handle the results and properly handle resources.
+ - Psuedo Loop Scenario:  Request PG Utah to ION to IG GPO. Then request a 2nd interface at PG Utah node to ION to same interface on same node at IG GPO.  If that fails, then request a 2nd interface on that node at IG GPO - that should work.
+
+= Operator Functions =
+  
+== VLAN Delegation ==
+
+Verify ability of operator to:
+
+  - Allocate non-OF VLAN for use by GENI Aggregates responsible for stitching path. 
+  - Verify Aggregate Advertisement RSpec reflects the non-OF VLAN allocation
+
+== Configuration ==
+
+  - Add/Modify/Delete the VLANs available for the aggregate, verify Advertisement reflects change for each add/modify/delete operation.
+  - Modify the endpoints for a VLAN, verify Advertisement reflects change.
+
+
+== Bandwidth allocation ==
+
+Verify ability of operator to:
+
+  - Allocate bandwidth for their ION connector that can be used for stitching
+  - Verify Aggregate Advertisement RSpec reflects bandwidth allocation defined by operator
+
+(Note: allocation does not require bandwidth enforcement at this point in time.)
+ 
+== Monitoring == 
+
+Verify ability to access data that allows GENI to track the following over time:
+
+ - existing slices/experiments
+ - mapping slices/experiments to resources/slivers
+ - mapping slices to sliver to user
+ - map circuit ID to sliver or slice (for I2)
+ - map VLAN ID on a given port to a sliver or slice. The VLAN ID for a given connection might be different and on different ports in the aggregate. 
+ - Stitching traffic statistics 
+
+= Experimenter Functions =
+
+Experimenter functions focus on verification of basic experiment scenario with the following priorities:
+ - GENI Rack to GENI Rack stitching
+ - Stitching with representative regional and core networks
+ - Stitching GENI Rack to non-rack resources.
+ 
+== Experiment Resources Scenarios ==
+
+Verify the ability to create, use, and release resources for the following:
+ - Single slice with two endpoints using single stitched VLAN
+   * 1 slice includes: (Aggr1<->VLAN1<->Aggr2)
+ - Single slice with two endpoints using multiple stitched VLANs 
+   * 1 slice includes: (Aggr1<->VLAN1<->Aggr2) + (Aggr1<->VLAN2<->Aggr2) + (Aggr1<->VLAN3<->Aggr2) 
+ - Multiple slices each with two endpoints using single stitched VLAN
+   * slice1 includes: (Aggr1<->VLAN1<->Aggr2)
+   * slice2 includes: (Aggr1<->VLAN2<->Aggr2)
+ - Multiple slices each with two endpoints using multiple stitched VLANs
+   * slice1 includes: (Aggr1<->VLAN1<->Aggr2) + (Aggr1<->VLAN2<->Aggr2)
+   * slice2 includes: (Aggr1<->VLAN3<->Aggr2) + (Aggr1<->VLAN4<->Aggr2)
+
+== Experiment Measurements ==
+
+A set of GENI Network Stitching (GNS) test topologies have been selected to validate stitching experiment scenarios before GEC17.
+In addition to the procedure described in each test case below, benchmark measurements are to be collected in each test case. Some end-points exist in multiple test cases, so their measurements will be not be duplicated.  Measurements will characterize the following areas:
+  - VLAN stitching setup timing
+  - VLAN stitching tear down timing
+  - Timing for an experiment to change data paths from one stitched VLAN to another
+  - End-host to end-host data flow set up timing
+  - End-host to end-host delay
+  - Throughput between end-hosts.
+ 
+
+== GNS-T1 - Topology 1 - Utah InstaGENI to GPO InstaGENI ==
+
+Verify stitching path for a scenario that has the end-points Utah and GPO InstaGENI via Internet2 ION. This scenario will verify experimenter functions and some operator functions as described below:
+
+    1. As Experimenter, use GENI AM API client to submit create sliver for initial Request RSpec. No aggregate needs to be specified, aggregates are determined from the Request RSpec. 
+    2. Verify that request RSpec is expanded after Path Computation into one RSpec for each Stitching Path Aggregates (IG Utah, PG Utah, ION, and IG GPO).
+    3. Verify that workflow agent negotiates VLAN with Utah InstaGENI using the RSpec from step 2 and generates a manifest for IG Utah resources.
+    4. Verify that workflow agent negotiates VLAN with Utah ProtoGENI using the RSpec from step 2 and generates a manifest for PG Utah resources.
+    5. Verify that workflow agent negotiates VLAN with GPO InstaGENI using the GPO RSpec from step 2 and generates a manifest for GPO resources.
+    6. Verify that workflow agent negotiates VLAN with ION using the ION RSpec from step 2 and generates a manifest for ION resources.
+    7. Verify that the final "combined" RSpec reflects the allocate stitching and non-stitching resources.
+    8. Log in to compute resources at each Utah InstaGENI and GPO InstaGENI, gather delay and throughput measurements. Leave traffic exchange running between the end-hosts.
+    9. As Operator, verify that nodes, ports, links, capacity can be determined as allocated for this experiment.
+    10. As Experimenter, leave resources running with traffic being exchanged. Allow sliver to expire, verify that sliver resources/traffic have been deleted.
+    11. As Operator, verify that expired resource resources (nodes, ports, links, capacity) are released properly and made available and part of the Advertisement RSpec.
+   
+Following is a diagram of the planned test topology GNS-T1:
+
+[[Image(GNS-T1.jpg)]]
+
+
+''Note: NOX stitching resources may be delegated statically and no GENI aggregate providing stitching path is available during the experiment''
+
+
+== GNS-T2 - Topology 2 - InstaGENI Utah to GPO ExoGENI ==
+
+Verify stitching negotiation for a scenario that has the following end-points Utah InstaGENI and GPO ExoGENI via Internet2 ION. It is expected that workflow outlined in [wiki:GENINetworkStitchingTestPlan#GNS-T1-Topology1-UtahInstaGENItoGPOInstaGENI GNS-T1] also applies in this topology.  The following steps are to be completed.
+
+    1. As Experimenter, use GENI AM API client to submit create sliver for initial Request RSpec. No aggregate needs to be specified, aggregates are determined from the Request RSpec.
+    2. Verify that request RSpec is expanded after Path Computation into one RSpec for each Stitching Path Aggregates (IG Utah, PG Utah, ION, and EG GPO).
+    3. Verify that workflow agent negotiates VLAN with Utah InstaGENI using the RSpec from step 2 and generates a manifest for IG Utah resources.
+    4. Verify that workflow agent negotiates VLAN with Utah ProtoGENI using the RSpec from step 2 and generates a manifest for PG Utah resources.
+    5. Verify that workflow agent negotiates VLAN with GPO ExoGENI using the GPO RSpec from step 2 and generates a manifest for GPO resources.
+    6. Verify that workflow agent negotiates VLAN with ION using the ION RSpec from step 2 and generates a manifest for ION resources.
+    7. Verify that the final "combined" RSpec reflects the allocate stitching and non-stitching resources.
+    8. Log in to compute resources at each Utah InstaGENI and GPO ExoGENI, gather delay and throughput measurements. Leave traffic exchange running between the end-hosts.
+    9. As Operator, verify that nodes, ports, links, capacity can be determined as allocated for this experiment.
+    10. As Experimenter, release resources by deleting sliver(s) at each aggregate where compute and VLAN resources were allocated. 
+    11. As Operator, verify that released resources (nodes, ports, links, capacity) are made available and part of the Advertisement RSpec.
+
+Following is a diagram of the planned test topology GNS-T2:
+
+[[Image(GNS-T2.jpg)]]
+
+
+''Note: NOX stitching resources may be delegated statically and no GENI aggregate providing stitching path is available during the experiment''
+
+''Note: ExoGENI Stitching support may not be available initially, if this is the case, GPO IG will replace GPO EG''
+
+== GNS-T3 - Topology 3 - InstaGENI Utah to GPO ExoGENI to LONI and MAX ==
+
+Verify scenario that includes stitched VLAN connections form the GPO ExoGENI to each of the other remotes: Utah InstaGENI, non-rack endpoints at LONI and MAX.  Other VLAN stitching scenario may also be tried, if deemed of interest. The following steps are to be completed.
+
+    1. As Experimenter, use GENI AM API client to submit initial Request RSpec.
+    2. Verify that request RSpec is expanded after Path Computation into one RSpec for each Stitching Path Aggregates (IG Utah, PG Utha, ION, EG GPO, LONI, and MAX).
+    3. Verify that workflow agent negotiates VLAN with Utah InstaGENI using the RSpec from step 2 and generates a manifest for IG Utah resources.
+    4. Verify that workflow agent negotiates VLAN with Utah ProtoGENI using the RSpec from step 2 and generates a manifest for PG Utah resources.
+    5. Verify that workflow agent negotiates VLAN with GPO ExoGENI using the GPO RSpec from step 2 and generates a manifest for GPO resources.
+    6. Verify that workflow agent negotiates VLAN with ION using the ION RSpec from step 2 and generates a manifest for ION resources.
+    7. Verify that workflow agent negotiates VLAN with LONI using the LONI RSpec from step 2 and generates a manifest for LONI resources.
+    8. Verify that workflow agent negotiates VLAN with MAX using the MAX RSpec from step 2 and generates a manifest for MAX resources.
+    9. Verify that the final "negotiated" RSpec is use by the agent to allocate stitching and non-stitching resources. 
+    10. Log in to compute resources at each Utah InstaGENI, GPO ExoGENI, LONI and MAX, gather delay and throughput measurements to new endpoints. Leave traffic exchange running between the end-hosts.
+    11. As Operator, verify that nodes, ports, links, capacity can be determined as allocated for this experiment.
+    12. As Experimenter, leave resources running with traffic being exchange. Allow sliver to expire, verify that sliver resources/traffic have been deleted.
+    13. As Operator, verify that expired resource resources (nodes, ports, links, capacity) are released properly and made available and part of the Advertisement RSpec.
+
+
+Following is a diagram of the planned test topology GNS-T3:
+
+[[Image(GNS-T3.jpg)]]
+
+
+''Note: NOX stitching resources may be delegated statically and no GENI aggregate providing stitching path is available during the experiment''
+
+''Note: If the LSU GENI rack is available while testing occurs it will be used as the end-point for LONI.''
+
+''Note: If the University of Maryland rack is available while testing occurs it will be used as the end-point for MAX.''
+
+''Note: ExoGENI Stitching support may not be available initially, if this is the case, GPO IG will replace GPO EG''
+
+== GNS-T4 - Topology 4 - InstaGENI Utah to GPO ExoGENI to CENIC ==
+
+Verify stitching for a scenario that includes InstaGENI Utah, ExoGENI GPO and non-rack endpoints CENIC. The following steps are to be completed.
+
+    1. As Experimenter, use GENI AM API client to submit initial Request RSpec.
+    2. Verify that request RSpec is expanded after Path Computation into one RSpec for each Stitching Path Aggregates (IG Utah, PG Utah, ION, EG GPO, and CENIC).
+    3. Verify that workflow agent negotiates VLAN with Utah InstaGENI using the RSpec from step 2 and generates a manifest for IG Utah resources.
+    4. Verify that workflow agent negotiates VLAN with Utah ProtoGENI using the RSpec from step 2 and generates a manifest for PG Utah resources.
+    5. Verify that workflow agent negotiates VLAN with GPO ExoGENI using the GPO RSpec from step 2 and generates a manifest for GPO resources.
+    6. Verify that workflow agent negotiates VLAN with ION using the ION RSpec from step 2 and generates a manifest for ION resources.
+    7. Verify that workflow agent negotiates VLAN with CENIC using the CENIC RSpec from step 2 and generates a manifest for CENIC resources.
+    8. Verify that the final "combined" RSpec reflects the allocate stitching and non-stitching resources.
+    9. Log in to compute resources at each Utah InstaGENI, GPO ExoGENI and CENIC, gather delay and throughput measurements to new end-points. Leave traffic exchange running between the end-hosts.
+    10. As Operator, verify that nodes, ports, links, capacity can be determined as allocated for this experiment.
+    11. As Experimenter, release resources by deleting sliver(s) at each aggregate where compute and VLAN resources were allocated.
+    12. As Operator, verify that released resources (nodes, ports, links, capacity) are made available and part of the Advertisement RSpec.
+ 
+
+
+Following is a diagram of the planned test topology GNS-T4:
+
+[[Image(GNS-T4.jpg)]]
+
+''Note: NOX stitching resources may be delegated statically and no GENI aggregate providing stitching path is available during the experiment''
+
+''Note: ExoGENI Stitching support may not be available initially, if this is the case, GPO IG will replace GPO EG''
+
+== GNS-T5 - Topology 5 - InstaGENI Utah to GPO ExoGENI to NYSERNet ==
+
+NYSERNet includes separate path for OF and non-OF at site, and would be a new and untested scenario.  A network path is stitched to a non-existing NYSERNet endpoint.  The following steps are to be completed.
+
+    1. As Experimenter, use GENI AM API client to submit initial Request RSpec.
+    2. Verify that request RSpec is expanded after Path Computation into one RSpec for each Stitching Path Aggregates (IG Utah, PG Utah, ION, EG GPO, and NYSERNet).
+    3. Verify that workflow agent negotiates VLAN with Utah InstaGENI using the RSpec from step 2 and generates a manifest for IG Utah resources.
+    4. Verify that workflow agent negotiates VLAN with Utah ProtoGENI using the RSpec from step 2 and generates a manifest for PG Utah resources.
+    5. Verify that workflow agent negotiates VLAN with GPO ExoGENI using the GPO RSpec from step 2 and generates a manifest for GPO resources.
+    6. Verify that workflow agent negotiates VLAN with ION using the ION RSpec from step 2 and generates a manifest for ION resources.
+    7. Verify that workflow agent negotiates VLAN with NYSERNet using the NYSERNet RSpec from step 2 and generates a manifest for NYSERNet resources.
+    8. Verify that the final "combined" RSpec reflects the allocate stitching and non-stitching resources. 
+    9. Log in to compute resources at each Utah InstaGENI, GPO ExoGENI and NYSERNet, gather delay and throughput measurements for new end-point. Leave traffic exchange running between the end-hosts.
+    10. As Operator, verify that nodes, ports, links, capacity can be determined as allocated for this experiment.
+    11. As Experimenter, release resources by deleting sliver(s) at each aggregate where compute and VLAN resources were allocated.
+    12. As Operator, verify that released resources (nodes, ports, links, capacity) are made available and part of the Advertisement RSpec.
+
+
+Following is a diagram of the planned test topology GNS-T5:
+
+[[Image(GNS-T5.jpg)]]
+
+''Note: NOX and NYSERNet stitching resources may be delegated statically and no GENI aggregate providing stitching path are available during the experiment ''
+
+''Note: ExoGENI Stitching support may not be available initially, if this is the case, GPO IG will replace GPO EG''
+
+
+
+== GNS-T6 - Topology 6 - Utah PG to GPO InstaGENI ==
+
+Verify stitching path for a scenario that has the end-points Utah PG and GPO InstaGENI via Internet2 ION. This scenario will verify experimenter function in the same way as described in [wiki:GENINetworkStitchingTestPlan#GNS-T1-Topology1-UtahInstaGENItoGPOInstaGENI GNS-T1]. The Stitching path is the same as [wiki:GENINetworkStitchingTestPlan#GNS-T1-Topology1-UtahInstaGENItoGPOInstaGENI GNS-T1] but without the InstaGENI aggregate. 
+
+
+= Post GEC17 testing =
+
+This section captures testing that should occur, but is expected to take place after GEC17.
+
+== OpenFlow Over Stitched VLAN Paths ==
+
+Topologies using !OpenFlow over stitched VLAN paths will be tested after GEC17 for various !OpenFlow scenarios that include a mix of aggregate end-points. It is expected that early scenarios will include a 2-step approach where the experimenter requests the VLANs to be stitched first and then the experimenter uses FOAM to create flows that use the VLANs, as they do now.  If this 2-step approach is available in an aggreate before GEC17, we will include it in the earlier tests.
+
+== Multi-point support ==
+
+Internet2 AL2S has the potential to provide additional services, such as multi-point topology for stitching.
+
+== Topology Service ==
+This feature will be validated after GEC17, 
+
+''TO DO: Need to define testing scenarios....''
+
+== Negotiation Support == 
+
+'''Stitching with Negotiation:''' [[BR]]
+
+Validate negotiation workflow and use of ticket calls. Verify that requests result into either a commit phase (success) or release (failure) for negotiation:
+    - Submit allocation request, verify that this generates manifest.
+    - Submit provision request, verify the manifest.
+    - Submit perform operational action start/stop/restart and verify the manifest.
+    - Verify successful and unsucessful sliver start results into expected stitching resource state.
+  
+'''Stitching without Negotiation:'''
+  - Verify ability to request to a specific AM that does not support negotiation.
+
+
+= References =
+ - http://groups.geni.net/geni/attachment/wiki/GEC16Agenda/DevelopersGrabBag/geni-gec16-stitching.pdf
+ - http://groups.geni.net/geni/wiki/GeniNetworkStitching
+ - http://geni.maxgigapop.net/twiki/bin/view/GENI/NetworkStitchingOverview
+ - http://geni.maxgigapop.net/twiki/bin/view/GENI/NetworkStitchingGeniApiAndRspec
+ - http://geni.maxgigapop.net/twiki/bin/view/GENI/NetworkStitchingArch
+ - http://geni.maxgigapop.net/twiki/bin/view/GENI/NetworkStitchingAPI