wiki:OpenFlow/Slicer/TestPlan

Version 30 (modified by tupty@bbn.com, 5 years ago) (diff)

--

GENI OpenFlow Slicer Test Plan

This test plan is for validating software that implements the GENI Slicer Requirements. The focus of this test plan is testing that the requirements are satisfied in an environment that is close to how the slicer will be used in practice once it is deployed in GENI. In other words, it is not focused on functional unit testing. Tests may exercise specific functions defined in the requirements, but the tests themselves should focus on validating that the tests pass within the context of how they will be used in GENI. With all of that said, this is also not a full blown integration test. A lot of the data plane traffic is simple ICMP ping traffic, which is not necessarily indicative of all kinds of traffic one might see in GENI. Also, any of the limitations of the substrate will not be held against the slicer as part of this test plan.

Expected test outcome

  • All test steps verifying a MUST requirement will be satisfied.
  • All test steps verifying a SHOULD requirement will be satisfied with special exceptions made only when they seem reasonable.
  • Any test steps that verify a MAY will have the results documented as features that operators can consider, but failing these steps they will not count against the test plan.
  • Any test steps that verify an item from the wishlist will have the results documented as features that operators can consider, but failing these steps will not count against the test plan.
  • All test steps verifying something that doesn't map back directly to a requirement will be satisfied with special exceptions made only when they seem reasonable.

Tracking progress during testing

Assumptions and Dependencies

  • Slicer software must interact with OpenFlow v1.0
  • Slicer must interact any hardware or software OpenFlow 1.0 switch that is not running in a VLAN-based hybrid mode.
  • Documentation must is available describing configuration and usage
  • Slice operations will be performed using the GENI AM API where possible. Most tests cannot be completed without the AM API.

Slicer Test Cases

OF-OPR-SLCR-FN Testing Scenario:

Environment

Ideally, this test case will be run with a hardware OpenFlow switch and hosts attached to that switch. The switch should not be running in VLAN-hybrid mode. If this test is run with vSwithches, please note this in the test plan results.

Step 1: Setup

  1. Reserve 5 slices (using the add_slice function) (005-a):
    1. Slice 'samevlan' with two ports on a single switch, each on the same VLAN. Create this slice such that it handles any VLAN translation at the slicer. Ensure the slicer state is accurate. (001-a)
    2. Slice 'diffvlans' with two ports on the same switch, each associated with a different VLAN. Create this slice such that it handles any VLAN translation at the slicer. Ensure the slicer state is accurate. (001-a)
    3. Slice 'tenvlans' with 10 VLANs on the same switch. Ensure the slicer state is accurate. (001-a)
    4. Slice 'untaggedports' with two untagged ports on a single switch. Ensure the slicer state is accurate. (001-b)
    5. Slice 'fullports' with two full ports on a single switch. Ensure the slicer state is accurate. (001-c)
  2. Attach hosts to each port with the appropriate data plane VLAN connections for 'samevlan', 'diffvlans', and hosts for at least two of the VLANs for 'tenvlans'
  3. Add two hosts which are on the same VLAN but do not correspond to an existing slice.

Step 2: Establish that basic connectivity works

  1. Run a gosal instance and a modified pox forwarding.l2_learning controller for 'samevlan' that installs a send-to-controller rule for unmatched traffic within the VLAN.
  2. Start a packet capture on the slicer facing the switch.
  3. Start a packet capture on the host running gosal facing the controller.
  4. Using a host that is not in a slice, send ICMP ping traffic on the data plane and make sure it never reaches the other host that is not in a VLAN. (002) (004-a) (004-c)
  5. Using 'samevlan', send ICMP ping data plane traffic between the two hosts. Ensure that it gets through to the two hosts in 'samevlan', but not to the hosts in 'diffvlans'. (001) (003)
  6. Stop the packet capture(s) at the slicer, and check for the following:
    1. Check that traffic from the hosts that don't belong to a slice never shows up at the slicer. (004-a) (004-c)
    2. Check that traffic from the slicer destined for a controller is only sent to a single controller (004-b) (004-d)
    3. Check that packet-ins from the switch have a VLAN tag, but packet-ins going to the controller do not. Also ensure that nothing else is different. (014-b) (009)
    4. Check that packet-outs from the controller have no VLAN tag, but packet-outs going to the switch do. Also ensure that nothing else is different. (014-a) (009)
    5. Traffic sent to the OFPP_ALL port from the controller is funnelled into the set of ports in the slice by the slicer. (003-c)
  7. Stop the gosal instance and pox forwarding.l2_learning controller for 'samevlan'.

Step 3: Test for VLAN enforcement and translation

  1. Start a floodlight instance with static flow pusher enabled.
  2. Push a flow that matches one of the ports and VLANs in 'samevlan' and tries to do a VLAN rewrite to a different VLAN ID before outputting to the other port in the slice. Ensure this gets rejected by the slicer, and an error message is returned. (003-b) (004-e) (008-a)
  3. Push a flow that matches one of the ports and VLANs, and tries to do a strip-VLAN action. Ensure this gets rejected by the slicer, and an error message is returned. (003-a) (004-e) (008-a)
  4. Run a gosal instance and a modified pox forwarding.l2_learning controller for 'diffvlans' that installs a send-to-controller rule for unmatched traffic for the specific VLAN+port matches that fall within the slice.
  5. Send ICMP ping traffic between the 'diffvlans' hosts and make sure it gets exchanged properly. (014-d)

Step 4: Check data plane throughput

  1. If it is not in place already, start a floodlight instance with static flow pusher enabled for the 'samevlan' slice. Make sure no flows are in place.
  2. Push flows that match traffic from one of the ports and sends the traffic to the other port in each direction.
  3. Run a tcp iperf between the two hosts and record the results compared to the known substrate bandwidth.
  4. Run a 1 Mbps udp iperf between the two hosts and record the results compared to the known substrate bandwidth.
  5. Run a 10 Mbps udp iperf between the two hosts and record the results compared to the known substrate bandwidth.

Step 5: Test for OF functionality

  1. Start a packet capture on the slicer facing both the switch and the controller.
  2. Push a flow_stats request from the controller.
  3. Stop the packet capture, and check that the flow_stats reply only went to the controller that requested it. (004-g)
  4. Run some OFTest checks within the 'samevlan' slice to make sure that (004-f) and (009) are met.
    1. Test OFPT_GET_CONFIG_REQUEST and make sure an OFPT_GET_CONFIG_REPLY comes back. Use the OFTest GetConfigReply test.
    2. Run a test that verifies OFPT_FLOW_REMOVED messages are properly returned to the controller. Use the OFTest FlowExpire test.
    3. Run a test that verifies that OFPT_BARRIER_REQUESTS send back an OFPT_BARRIER_REPLY. Use the OFTest BarrierRequestReply test.
  5. Test a few synchronous messages using OFTest to make sure (006) is met.
  6. Test that (007-a) and (007-b) both generate error messages if (007-c) is not met.
    1. Test that sending an OFPT_PORT_MOD from the controller either generates an error, or at the very least doesn't go to the substrate. Use the OFTest PortModPacketIn test.
    2. Test that sending an OFPT_SET_CONFIG message either generates an error, or at the very least doesn't go to the substrate. Use the OFTest SetConfigRequest test.

Step 6: Test for stacked slicer support

  1. For 'diffvlans' stop the pox forwarding.l2_learning controller. In its place, stand up another instance of the slicer. In this higher-level slicer, create an identical slice for 'diffvlans' (can be done manually), and point it at a new controller URL. At that new controller URL, run an instance of floodlight forwarding with static flow pusher. Push down a flow that matches on the slice's VLAN for the switch and forwards traffic to the controller.
  2. Send ICMP ping data plane traffic between the 'diffvlans' hosts. (010)

Step 7: Test for any extra functionality

  1. Test that any documented control plane performance isolation solution works as expected. Consider using the following modules from OFTest: PacketInLoad, FlowModLoad, PacketOutLoad, FlowRemovedLoad. (011)
  2. Test that any documented data plane performance isolation solution works as expected. This will involve appropriately setting up two slices, proactively installing flows, and starting UDP iperfs to make sure expected throughput is matched. (012)
  3. Test that any per-slice flow limits work as expected. Consider using the OFTest FlowModLoad test or Floodlight static flow pusher. (012)

Step 8: Test management API

  1. Call dump_slices and make sure it works as expected. Ensure it returns which slices are active and which are disabled along with however the AM identifies the slices. (005-d) (008-b) (008-c)
  2. Call change_controller and make sure it is reflected in the slicer state (005-c)
  3. Call update_slice if possible and report back what happens (005-e)
  4. Call delete_slice and clean up all slices (005-b)

OF-OPF-SLCR-OP Testing Scenario

Environment

This test may be run with vSwitches in GENI or using mininet.

Step 1: Setup

  1. Clear any existing log files that the slicer may have created.
  2. Create 10 slices named slice{1-10}. Each slice should have a single unique VLAN ID spanning across 10 switches between two hosts. For each of these slices, install static flows for passing traffic between the hosts. Generate ping traffic from a host at the end of each of the slices and make sure it is working. Leave this traffic running for the duration of this scenario. (003)

Step 2: Collect monitoring data

  1. Make sure there is some way to collect monitoring data for all of the slices that have been set up. This could be something like a read-only slice, or an administrative call to see things like flow stats for all slices. (002) (010)

Step 3: Test slice operations

  1. Create a new slice 'testslice', use the modified pox forwarding.l2_learning controller, and ensure that pings in the other slices continue to flow. Ensure that this slice is added with no operator intervention. (004) (001-a)
  2. Change the controller URL for 'testslice'. Ensure that the change takes place with no operator intervention. (001-c)
  3. If a slicer is capable of disabling a slice, trigger 'testslice' to get disabled. Ensure that either the slicer owner either gets notified, or that 'dump_slices' shows that the slice is disabled. Administratively re-enable the slice if it was disabled. (001-d)
  4. Disconnect the 'testslice' controller. Wait for about a minute, and try sending ICMP ping traffic in the data plane. Ensure that it doesn't get through. (008)
  5. Delete the slice and make sure that doing so does not require operator intervention. Check right away and make sure all flows for that slice have been removed. (001-b) (009)

Step 4: Substrate-generated traffic

  1. Connect a host to the substrate that is not part of a slice, and that does not share a data plane port with the other slices. Send a high rate of traffic out of its data plane interface and ensure that the other slices remain stable. (007)
  2. Try sending traffic that the slicer cannot process (e.g. non-OpenFlow traffic or an OpenFlow version that the slicer doesn't currently understand) directly to the southbound interface of the slicer. Make sure the slicer remains stable. (011)

Step 5: Test logging

  1. Check that a log exists for the slicer. (005)
  2. Configure the logging to rotate on a daily basis.
  3. Look at the entries for the log. Any entries that exist in the default case should be reasonable. At this point, the size of the log should also be reasonable. (006-a)
  4. Try changing the log level while the slicer is running to the most verbose level. Verify that the logs are showing more data. (006-b)
  5. Set the logging back to a standard level. Leave the setup running overnight. In the morning, verify that the logs have been rotated, the logs are of reasonable size, and that logging is still working. (006-c)

OF-OPR-SLCR-SW Testing

Step 1: Perform validation steps

  1. Ensure there is a support mailing list and some level of documentation for users and administrators. (001) (002) (009)
  2. Ensure with the development team that there is a plan to support OpenFlow 1.3 in the future. (003)
  3. Ensure that there is a repository which we can acccess. (004)
  4. Ensure that upgrades of software don't destroy previous configuration. (007)
  5. Work with the development team to make sure they can satisfy remaining software requirements. (005) (006) (008)

Integration Testing

Environment

This test should be run with vSwitches in GENI which are interconnected through a GENI stitched connection.

Step 1: Setup

  1. Install two instances of the slicer software.
  2. Create a GENI slice comprised of two vSwitches interconnected through a GENI stitched connection. There should be 2 hosts attached to each vSwitch.
  3. Point the control plane for the one of the vSwitches to one of the OpenFlow slicers and another vSwitch to another OpenFlow slicer.
  4. Create a single OpenFlow slice in each slicer that connects the hosts to each vSwitch, and each vSwitch to the stitched connection. Point both of these reservations to a single controller URL and run a floodlight forwarding controller there.

Step 2: Exchange traffic

  1. Make sure that each host can exchange ICMP ping traffic with all other hosts.

Wishlist Testing (on request)

Environment

There are no restrictions on environment for these tests.

Step 1: Check documentation

  1. Does the software come in a package? If so, are there instructions for installing the software as a package?
  2. Is untagged traffic supported in the slicer? If so, does the documentation show how to reserve the untagged space on a port?
  3. Is data plane isolation supported (unlikely until future OpenFlow versions are supported)?

Step 2: Set up the environment

  1. Create a slice that uses two ports that connect two hosts in a slice. Use untagged space on those ports if possible.
  2. Run Floodlight static flow pusher, and push down a set of flows that will forward traffic between the two ports.
  3. Make sure that you can exchange traffic between the hosts.

Step 3: Test that priority setting works

  1. Install an identical set of flows with a higher priority and no actions. When doing this, do not remove the old rules.
  2. If possible, check the flow table and ensure that all four of the flows are in the flow table.
  3. Ensure that traffic is not exchanged between the hosts.
  4. Remove the higher-priority flows and make sure that traffic flows again.