[[PageOutline]] = OpenFlow Slicer Test Plan Results = This page tracks the results of a piece of software run through the [http://groups.geni.net/geni/wiki/OpenFlow/Slicer/TestPlan OpenFlow Slicer Test Plan]. ||'''Software''' ||'''Version'''|| ||Flowspace Firewall || 1.0.4b || = Results Legend = Individual test case status may be one of the following: * [[Color(green,Pass)]] - Test case completed and met all criteria * [[Color(#98FB98,Pass: most criteria)]] - Test case completed and met most criteria with exceptions documented * [[Color(red,Fail)]] - Test case completed and failed. * [[Color(orange,Blocked)]] - Remaining testing blocked by ticketed issues. * [[Color(#AAAAAA,N/A)]] - Tests which are not applicable (only used sparingly). = OF-OPR-SLCR-FN Testing = == Results == === Step 1: Setup === ||'''Test Case'''||'''Result'''||'''Note'''|| || 1.a.i. ||[[Color(green,Pass)]]|| || || 1.a.ii. ||[[Color(green,Pass)]]|| || || 1.a.iii. ||[[Color(green,Pass)]]|| || || 1.a.iv. ||[[Color(green,Pass)]]|| || || 1.a.v. ||[[Color(green,Pass)]]|| || || 1.b. ||[[Color(green,Pass)]]|| || || 1.c. ||[[Color(green,Pass)]]|| || === Step 2: Establish that basic connectivity works === ||'''Test Case'''||'''Result'''||'''Note'''|| || 2.a. ||[[Color(green,Pass)]]|| || || 2.b. ||[[Color(green,Pass)]]|| || || 2.c. ||[[Color(green,Pass)]]|| || || 2.d. ||[[Color(green,Pass)]]|| || || 2.e. ||[[Color(green,Pass)]]|| || || 2.f.i. ||[[Color(red,Fail)]] ||[attachment:slicer.pcap Slicer capture]. This failure might be an issue with environment setup. Is slicer responsible for installing a "drop unmatched" rule? [https://github.com/GlobalNOC/FlowSpaceFirewall/issues/114 Feature request]|| || 2.f.ii. ||[[Color(green,Pass)]]||[attachment:slicer.pcap Slicer capture] || || 2.f.iii. ||[[Color(green,Pass)]]||[attachment:slicer.pcap Slicer capture], [attachment:controller.pcap Controller capture]|| || 2.f.iv. ||[[Color(green,Pass)]]||[attachment:slicer.pcap Slicer capture], [attachment:controller.pcap Controller capture]|| || 2.f.v. ||[[Color(green,Pass)]]||[attachment:slicer.pcap Slicer capture], [attachment:controller.pcap Controller capture]|| || 2.g. ||[[Color(green,Pass)]]|| || === Step 3: Test for VLAN enforcement and translation === ||'''Test Case'''||'''Result'''||'''Note'''|| || 3.a. ||[[Color(green,Pass)]]|| || || 3.b. ||[[Color(green,Pass)]]||[attachment:controller.2.pcap Controller capture]|| || 3.c. ||[[Color(green,Pass)]]||[attachment:controller.2.pcap Controller capture]|| || 3.d. ||[[Color(green,Pass)]]|| || || 3.e. ||[[Color(green,Pass)]]|| || === Step 4: Check data plane throughput === ||'''Test Case'''||'''Result'''||'''Note'''|| || 4.a. ||[[Color(green,Pass)]]|| || || 4.b. ||[[Color(green,Pass)]]|| || || 4.c. ||[[Color(green,Pass)]]|| || || 4.d. ||[[Color(green,Pass)]]|| || || 4.e. ||[[Color(green,Pass)]]|| || === Step 5: Test for OF functionality === ||'''Test Case'''||'''Result'''||'''Note'''|| || 5.a. ||[[Color(green,Pass)]]|| || || 5.b. ||[[Color(green,Pass)]]|| || || 5.c. ||[[Color(green,Pass)]]|| || || 5.d.i. ||[[Color(red,Fail)]]||[https://github.com/GlobalNOC/FlowSpaceFirewall/issues/83 Known issue], will be fixed in next release. We should look into other similar instances where symmetric messages which are served by FSFW (and not the substrate) appropriately set the XID. || || 5.d.ii. ||[[Color(green,Pass)]]|| || || 5.d.iii. ||[[Color(green,Pass)]]|| || || 5.e.i. ||[[Color(green,Pass)]]||[attachment:slicer.2.pcap Slicer capture], Slicer responds but never sends to substrate. || || 5.e.ii. ||[[Color(green,Pass)]]||[attachment:slicer.2.pcap Slicer capture], Slicer doesn't respond, message never sent to substrate. || === Test 6: Test for stacked slicer support === ||'''Test Case'''||'''Result'''||'''Note'''|| || 6.a. ||[[Color(green,Pass)]]|| || || 6.b. ||[[Color(green,Pass)]]|| || === Test 7: Test for any extra functionality === ||'''Test Case'''||'''Result'''||'''Note'''|| || 7.a. ||[[Color(#98FB98,Pass: most criteria)]]||[attachment:controller.2.pcap Controller capture], Limits seem to be enforced. Cases in which slice is disabled don't seem to be documented. Specific types of OF error messages sent in different cases don't seem to be documented. || || 7.b. ||[[Color(#AAAAAA,N/A)]]|| No data plane limiters. || || 7.c. ||[[Color(#98FB98,Pass: most criteria)]]||[attachment:controller.2.pcap Controller capture], Limits seem to be enforced. Cases in which slice is disabled don't seem to be documented. Specific types of OF error messages sent in different cases don't seem to be documented. || === Test 8: Test management API === ||'''Test Case'''||'''Result'''||'''Note'''|| || 8.a. ||[[Color(orange,Blocked)]]|| Management API software under development. || || 8.b. ||[[Color(orange,Blocked)]]|| Management API software under development. || || 8.c. ||[[Color(orange,Blocked)]]|| Management API software under development. || || 8.d. ||[[Color(orange,Blocked)]]|| Management API software under development. || == Logs == This test was performed in GENI using vSwitches. = OF-OPR-SLCR-OP Testing (in GENI) = == Results == === Step 1: Setup === ||'''Test Case'''||'''Result'''||'''Note'''|| || 1.a. ||[[Color(green,Pass)]]|| || || 1.b. ||[[Color(green,Pass)]]|| || === Step 2: Collect monitoring data === ||'''Test Case'''||'''Result'''||'''Note'''|| || 2.a. ||[[Color(green,Pass)]]||[attachment:slices.json Slice list], [attachment:slice1-status.json Slice1 status], [attachment:slice1-flows.json Slice1 flows]|| === Step 3: Test slice operations === ||'''Test Case'''||'''Result'''||'''Note'''|| || 3.a. ||[[Color(green,Pass)]]|| || || 3.b. ||[[Color(green,Pass)]]|| || || 3.c. ||[[Color(green,Pass)]]|| || || 3.d. ||[[Color(green,Pass)]]|| || || 3.e. ||[[Color(green,Pass)]]||[attachment:flows.json Substrate flows], no flows with testslice's VLAN ID of 200|| === Step 4: Substrate-generated traffic === ||'''Test Case'''||'''Result'''||'''Note'''|| || 4.a. ||[[Color(green,Pass)]]|| || || 4.b. ||[[Color(green,Pass)]]|| || === Step 5: Test logging === ||'''Test Case'''||'''Result'''||'''Note'''|| || 5.a. ||[[Color(green,Pass)]]|| || || 5.b. ||[[Color(green,Pass)]]|| || || 5.c. ||[[Color(green,Pass)]]|| || || 5.d. ||[[Color(green,Pass)]]|| || || 5.e. ||[[Color(green,Pass)]]|| || == Logs == This test was run using GENI and virtual switches. Parts of test case 3 were changed after this run. = OF-OPR-SLCR-OP Testing (in hardware testbed) = == Results == === Step 1: Setup === ||'''Test Case'''||'''Result'''||'''Note'''|| || 1.a. ||[[Color(green,Pass)]]|| || || 1.b. ||[[Color(green,Pass)]]|| || === Step 2: Collect monitoring data === ||'''Test Case'''||'''Result'''||'''Note'''|| || 2.a. ||[[Color(green,Pass)]]||[attachment:slices.2.json Slice list], [attachment:slice1-status.2.json Slice1 status], [attachment:slice1-flows.2.json Slice1 flows]|| === Step 3: Test slice operations === ||'''Test Case'''||'''Result'''||'''Note'''|| || 3.a. ||[[Color(green,Pass)]]|| || || 3.b. ||[[Color(green,Pass)]]|| || || 3.c. ||[[Color(green,Pass)]]|| || || 3.d. ||[[Color(green,Pass)]]|| || || 3.e. ||[[Color(red,Fail)]]||[attachment:flows.2.json Substrate flows], flows with testslice's VLAN ID of 200 still exist. This should be somewhat fixed in the next release with a granularity of the FSFW flow stat polling rate.|| === Step 4: Substrate-generated traffic === ||'''Test Case'''||'''Result'''||'''Note'''|| || 4.a. ||[[Color(green,Pass)]]|| || || 4.b. ||[[Color(green,Pass)]]|| || === Step 5: Test logging === ||'''Test Case'''||'''Result'''||'''Note'''|| || 5.a. ||[[Color(green,Pass)]]|| || || 5.b. ||[[Color(green,Pass)]]|| || || 5.c. ||[[Color(green,Pass)]]|| || || 5.d. ||[[Color(green,Pass)]]|| || || 5.e. || || || == Logs == This test was run with a hardware substrate, including an HP 3800 in OF 1.0 mode. The control plane elements were in the GPO InstaGENI rack. = OF-OPR-SLCR-SW Testing = == Results == === Step 1: Perform validation steps === ||'''Test Case'''||'''Result'''||'''Note'''|| || 1.a. ||[[Color(green,Pass)]]|| See http://globalnoc.iu.edu/sdn/fsfw.html || || 1.b. ||[[Color(green,Pass)]]|| We've heard this verbally before. It will be a total rewrite. || || 1.c. ||[[Color(green,Pass)]]|| See https://github.com/GlobalNOC/FlowSpaceFirewall || || 1.d. ||[[Color(green,Pass)]]|| || || 1.e. ||[[Color(green,Pass)]]|| We're getting general agreement to fix reported issues. || == Logs == ''Fill these in as needed'' = Integration Testing = == Results == === Step 1: Setup === ||'''Test Case'''||'''Result'''||'''Note'''|| || 1.a. ||[[Color(green,Pass)]]|| || || 1.b. ||[[Color(green,Pass)]]|| || || 1.c. ||[[Color(green,Pass)]]|| || || 1.d. ||[[Color(green,Pass)]]|| || === Step 2: Collect monitoring data === ||'''Test Case'''||'''Result'''||'''Note'''|| || 2.a. ||[[Color(green,Pass)]]|| || = Wishlist Testing = == Results == === Step 1: Check documentation === ||'''Test Case'''||'''Result'''||'''Note'''|| || 1.a. ||[[Color(#98FB98,Pass: most criteria)]]|| Software comes as a package, and there is installation documentation, but it is pretty loose. It should be sufficient for anyone with experience using RPM packages. || || 1.b. ||[[Color(green,Pass)]]|| || || 1.c. ||[[Color(#AAAAAA,N/A)]]|| No data plane limiters. || === Step 2: Set up the environment === ||'''Test Case'''||'''Result'''||'''Note'''|| || 2.a. || || || || 2.b. || || || || 2.c. || || || === Step 3: Test that priority setting works === ||'''Test Case'''||'''Result'''||'''Note'''|| || 3.a. || || || || 3.b. || || || || 3.c. || || || || 3.d. || || || == Logs == ''Fill these in as needed''