wiki:OpenFlow/Slicer/TestPlan

Version 18 (modified by tupty@bbn.com, 9 years ago) (diff)

--

GENI OpenFlow Slicer Test Plan

This test plan is for validating software that implements the GENI Slicer Requirements. The focus of this test plan is testing that the requirements are satisfied in an environment that is close to how the slicer will be used in practice once it is deployed in GENI. In other words, it is not focused on functional unit testing. Tests may exercise specific functions defined in the requirements, but the tests themselves should focus on validating that the tests pass within the context of how they will be used in GENI. With all of that said, this is also not a full blown integration test. A lot of the data plane traffic is simple ICMP ping traffic, which is not necessarily indicative of all kinds of traffic one might see in GENI. Also, any of the limitations of the substrate will not be held against the slicer as part of this test plan.

Expected test outcome

  • All test steps verifying a MUST requirement will be satisfied.
  • All test steps verifying a SHOULD requirement will be satisfied with special exceptions made only when they seem reasonable.
  • Any test steps that verify a MAY will have the results documented as features that operators can consider, but failing these steps they will not count against the test plan.
  • Any test steps that verify an item from the wishlist will have the results documented as features that operators can consider, but failing these steps will not count against the test plan.
  • All test steps verifying something that doesn't map back directly to a requirement will be satisfied with special exceptions made only when they seem reasonable.

Tracking progress during testing

Assumptions and Dependencies

  • Slicer software must interact with OpenFlow v1.0
  • Slicer must interact any hardware or software OpenFlow 1.0 switch that is not running in a VLAN-based hybrid mode.
  • Documentation must is available describing configuration and usage
  • Slice operations will be performed using the GENI AM API where possible. Most tests cannot be completed without the AM API.

Slicer Test Cases

OF-OPR-SLCR-FN Testing Scenario:

Step 1: Setup

  1. Reserve 5 slices (using the add_slice function) (005-a):
    1. Slice 'samevlan' with two ports on a single switch, each on the same VLAN. Ensure the slicer state is accurate. (001-a)
    2. Slice 'diffvlans' with two ports on the same switch, each associated with a different VLAN. Ensure the slicer state is accurate. (001-a)
    3. Slice 'tenvlans' with 10 VLANs spanning across 10 switches. Ensure the slicer state is accurate. (001-a)
    4. Slice 'untaggedports' with two untagged ports on a single switch (doesn't need to be possible to pass). Ensure the slicer state is accurate. (001-b)
    5. Slice 'fullports' with two full ports on a single switch (doesn't need to be possible to pass). Ensure the slicer state is accurate. (001-c)
  2. Add a hosts attached to each port with the appropriate data plane VLAN connections for 'samevlan', 'diffvlans', and hosts for at least two of the VLANs for 'tenvlans'
  3. Add two hosts which are on the same VLAN but do not correspond to an existing slice.

Step 2: Establish that basic connectivity works

  1. Run a modified pox forwarding.l2_learning controller for 'samevlan' that installs a send-to-controller rule for unmatched traffic within the VLAN.
  2. Start a packet capture on the slicer facing both the switch and the controller.
  3. Using a host that is not in a slice, send ICMP ping traffic on the data plane and make sure it never reaches the other host. (002) (004-a) (004-c)
  4. Using 'samevlan', send ICMP ping data plane traffic between the two hosts. Ensure that it gets through to the two hosts in 'samevlan', but not to the hosts in 'diffvlans'. (001) (003)
  5. Stop the packet capture, and check for the following:
    1. Check that traffic from the hosts that don't belong to a slice never shows up at the slicer. (004-a) (004-c)
    2. Check that traffic from the slicer destined for a controller is only sent to a single controller (004-b) (004-d)
    3. Check that packet-ins from the switch have a VLAN tag, but packet-ins going to the controller do not. Also ensure that nothing else is different. (014-b) (009)
    4. Check that packet-outs from the controller have no VLAN tag, but packet-outs going to the switch do. Also ensure that nothing else is different. (014-a) (009)
    5. Traffic sent to the OFPP_ALL port from the controller is funnelled into the set of ports in the slice by the slicer. (003-c)

Step 3: Test for VLAN enforcement and translation

  1. Stop the pox forwarding.l2_learning controller for 'samevlan'. Start a floodlight instance in its place with static flow pusher enabled.
  2. Push a flow that matches one of the ports and VLANs, and tries to do a VLAN rewrite to a different VLAN ID. Ensure this gets rejected by the slicer, and an error message is returned. (003-b) (004-e) (008-a)
  3. Push a flow that matches one of the ports and VLANs, and tries to do a strip-VLAN action. Ensure this gets rejected by the slicer, and an error message is returned. (003-a) (004-e) (008-a)
  4. Run a modified pox forwarding.l2_learning controller for 'diffvlans' that installs a send-to-controller rule for unmatched traffic within the VLAN.
  5. Send ICMP ping traffic between the 'diffvlans' hosts and make sure it gets exchanged properly. (014-d)

Step 4: Test for OF functionality

  1. Start a packet capture on the slicer facing both the switch and the controller.
  2. Push a flow_stats request from the controller.
  3. Stop the packet capture, and check for the following: the flow_stats reply only went to the controller that requested it. (004-g)
  4. Run some OFTest checks to make sure that (004-f) and (009) are met.
  5. Test a few synchronous messages using OFTest to make sure (006) is met.
  6. Test that (007-a), (007-b), and (007-c) all generate error messages.

Step 5: Test for stacked slicer support

  1. For 'diffvlans' stop the pox forwarding.l2_learning controller. In its place, stand up another instance of the slicer. In this higher-level slicer, create an identical slice for 'diffvlans' (can be done manually), and point it at a new controller URL. At that new controller URL, run an instance of floodlight forwarding with static flow pusher. Push down a flow that matches on the slice's VLAN for the switch and forwards traffic to the controller.
  2. Send ICMP ping data plane traffic between the 'diffvlans' hosts. (010)

Step 6: Test for any extra fucntionality

  1. Test that any documented control plane performance isoaltion solution works as expected. (011)
  2. Test that any documented data plane performance isoaltion solution works as expected. (012)
  3. Test that any per-slice flow limits work as expected. (012)

Step 7: Test API

  1. Call dump_slices and make sure it works as expected. Ensure it returns which slices are active and which are disabled along with however the AM identifies the slices. (005-d) (008-b) (008-c)
  2. Call change_controller and make sure it is reflected in the slicer state (005-c)
  3. Call update_slice if possible and report back what happens (005-e)
  4. Call delete_slice and clean up all slices (005-b)

OF-OPF-SLCR-OP Testing Scenario

Step 1: Setup

  1. Clear any existing log files that the slicer may have created.
  2. Create 10 slices named slice{1-10}. Each slice should have a single unique VLAN ID spanning across 10 switches between two hosts. Generate ping traffic from a host at the end of each of the slices and make sure it is working. Leave this traffic running for the duration of this scenario. (003)

Step 2: Collect monitoring data

  1. Make sure there is some way to collect monitoring data for all of the slices that have been set up. (002) (010)

Step 3: Test slice operations

  1. Create a new slice 'testslice', use the modified pox forwarding.l2_learning controller, and ensure that pings in the other slices continue to flow. Ensure that this slice is added with no operator intervention. (004) (001-a)
  2. Change the controller URL for 'testslice'. Ensure that the change takes place with no operator intervention. (001-c)
  3. If a slicer is capable of disabling a slice, trigger 'testslice' to get disabled. Ensure that the slicer owner either gets notified, or that at the very least, 'dump_slices' shows that the slice is disabled. Re-enable the slice if it was disabled. (001-d)
  4. Disconnect the controller. Wait for about a minute, and try sending traffic. Ensure that it doesn't get through. (008)
  5. Delete the slice and make sure that doing so does not require operator intervention. Check right away and make sure all flows for that slice have been removed. (001-b) (009)

Step 4: Substrate-generated traffic

  1. Connect a host to the substrate that is not part of a slice, and that does not share a data plane port with the other slices. Send a high rate of traffic out of its data plane interface and ensure that the other slices remain stable. (007)
  2. Try sending traffic that the slicer cannot process (e.g. non-OpenFlow traffic or an OpenFlow version that the slicer doesn't currently understand) directly to the southbound interface of the slicer. Make sure the slicer remains stable. (011)

Step 5: Test logging

  1. Check that a log exists for the slicer. (005)
  2. Configure the logging to rotate on a daily basis.
  3. Look at the entries for the log. Any entries that exist in the default case should be reasonable. At this point, the size of the log should also be reasonable. (006-a)
  4. Try changing the log level while the slicer is running to the most verbose level. Verify that the logs are showing more data. (006-b)
  5. Set the logging back to a standard level. Leave the setup running overnight. In the morning, verify that the logs have been rotated, and that logging is still working. (006-c)

OF-OPR-SLCR-SW Testing

Step 1: Perform validation steps

  1. Ensure there is a support mailing list and some level of documentation for users and administrators. (001) (002) (009)
  2. Ensure with the development team that there is a plan to support OpenFlow 1.3 in the future. (003)
  3. Ensure that there is a repository which we can acccess. (004)
  4. Ensure that upgrades of software don't destroy previous configuration. (007)
  5. Work with the development team to make sure they can satisfy remaining software requirements. (005) (006) (008)

Wishlist Testing

Step 1: Check documentation

  • Does the software come in a package? If so, are there instructions for installing the software as a package?
  • Is untagged traffic supported in the slicer? If so, does the documentation show how to reserve the untagged space on a port?
  • Is data plane isolation supported (unlikely until future OpenFlow versions are supported)?

Step 2: Set up the environment

  • Create a slice that uses two ports that connect two hosts in a slice. Use untagged space on those ports if possible.
  • Run Floodlight static flow pusher, and push down a set of flows that will forward traffic between the two ports.
  • Make sure that you can exchange traffic between the hosts.

Step 3: Test that priority setting works

  • Install an identical set of flows with a higher priority and no actions. When doing this, do not remove the old rules.
  • If possible, check the flow table and ensure that all four of the flows are in the flow table.
  • Ensure that traffic is not exchanged between the hosts.
  • Remove the higher-priority flows and make sure that traffic flows again.

Test Methodology

Describe how testing is to be conducted, how issues are to be tracked, how status will be made available. Describe how test cases are named.