Changes between Initial Version and Version 1 of QinqResults

Timestamp:

06/28/10 06:48:49 (14 years ago)

Author:

jwilliams@bbn.com

Comment:

migrated initial text from SysengWiki:QinQCapabilites

QinqResults

@@ +1 @@
+= Purpose = 
+[[PageOutline]]
+This document outlines Exploring QinQ using various vendor's switch hardware (DUT - Device Under Test). This document covers:
+ * initial QinQ overview
+ * how to configure each DUT for QinQ
+ * feasibility testing
+ * interoperability testing  
+
+= QinQ Overview =
+QinQ can be used to "tunnel" a particular VLAN of a "customer" network through a "service" network. This is a very important concept in GENI to allow multiple "customer" VLANS to be interconnected through the VLANS of regional service providers.
+
+[[Image(QinQOverview.jpg)]]
+
+In the image above VLAN A and VLAN B are two VLANs that span between "Network 1" and "Network 2". These networks can be at two separate sites.  VLANs A & B are tunneled through the "Intermediate network" using VLAN Q. This allows the customer network VLAN usage to be independent of the Intermediate Network while still allowing the customer VLAN traffic to transverse the intermediate network. The Intermediate Network has a "separate VLAN ID space" than the two other network sites. VLAN X could be the same VLAN ID as either VLAN A or VLAN B without any collision.
+
+The distinction between the different types of VLANs results in a discussion of the Ethernet Frame's !EtherType field. An Ethernet Frame's header uses an !EtherType field to specify what type of data is contained within the frame.  
+
+For VLANs the value in this field is, in large part, determined by the switch's port's configuration.
+|| '''VLAN type''' || '''!EtherType value''' || '''tag type'''|| '''port type''' || '''note''' ||
+|| none  || per protocol   || untagged frame || access port ||   IP = 0x800, ARP = 0x0806, etc  ||
+|| customer  || 0x8100 || tagged frame || 802.1q VLAN trunk port|| 802.1q VLAN tagging  ||
+|| service || 0x8a88  || QinQ tagged frame || 802.1ad (QinQ) VLAN trunk port|| 8021.ad provider tunnel  ||
+
+For 802.1q VLANs and 802.ad QinQ VLANs the !EtherType is actually a "Tag Protocol Identifier (TPID)" that, along with other tagging information, is inserted after the frame's source MAC address field. As this TPID is at the same byte offset as the original !EtherType field, it is common to refer to this field as the !EtherType field when discussing VLANs. The image below illustrates this distinction. Each tag adds 4 Bytes of data to the Ethernet frame.
+
+[[Image(untaggedTaggedQinqTagged.jpg)]]
+
+ * See http://en.wikipedia.org/wiki/Ethernet#Ethernet_frame_types_and_the_EtherType_field for more information on frame types.
+ * See http://en.wikipedia.org/wiki/IEEE_802.1Q  for more information on VLAN tagging and QinQ tagging.
+
+As the above illustration shows, the frame size was increased (by 4 Bytes) per each added tag. The size implications are shown in the table below.
+
+||'''use'''                           || '''header size'''|| '''tag size''' || '''MTU''' || '''FCS'''|| '''total frame size'' || 
+||standard ethernet   || 14  || 0 || 1500     || 4 || 1518  ||
+||802.1q VLAN trunk  || 14  ||+4  || 1500  || 4 ||  1522  || 
+||802.1ad (QinQ) VLAN "tunnel" || 14 ||+4 +4  || 1500 || 4 || 1526   || 
+
+The 802.3ac standard increased the maximum frame length from 1518 to  1522 Bytes specifically, and exclusively, for VLAN tags. If a switch vendor adheres to this standard then VLAN tags can be viewed as part of the header (MTU stays as 1500) while QinQ tags require an increase of MTU (to 1504) to handle the inner tag. The test [http://groups.geni.net/syseng/wiki/QinQCapabilites#MTUsinQinQ MTU's in QinQ] specifically addresses the per-vendor MTU implementation.
+
+== Simulated Network Topology ==
+A single network site's topology, as shown in the Overview diagram, is simulated for this test set up within a single switch to fully exercise the DUTs VLAN and QinQ capabilities.  A single physical switch will be implementing both customer and service VLANs. 
+
+A VLAN trunk between a DUT's customer and service VLANs looks like a "jumper" cable -- a customer VLAN trunk must be created out of a customer-level switch trunk port and conencted to a service-level switch's acess port. This is a consequence of simulating multiple virtual switches within a single physical switch  This jumper is shown below in the Simulated Network Topology diagram. For complete end-to-end testing Two DUT's, each configured as shown, would be connected using the respective QinQ port.
+
+[[Image(SimulatedTestTopology.jpg)]]
+
+== Device Summary ==
+
+|| Poblano || NEC IP 8800  switch (DUT) -- see ["hwNecIP8800"] for reference and usage  ||
+|| Habanero || HP !ProCurve 6600 switch (DUT) -- see ["hwHpProcurve6600"] for reference and usage ||
+|| Basil ||  Cisco Catalyst 3750 switch (DUT)-- ["hwCiscoCatalyst3750"] for reference and usage||
+|| Naboo || VM Server ||
+|| azzalle || host on Naboo for testing ||
+|| gotland || host on Naboo for testing ||
+|| lagnace || host on Naboo for testing ||
+|| skaldia  || host on Naboo for testing ||
+|| wireshark || host used for traffic monitoring (laptop) ||
+|| wasabi || Switch used for internal network connections ||
+
+== Test Topology with Internal Network ==
+This section outlines the configuration steps necessary to integrate the DUTs into the BBN internal network to allow for testing and configuration.
+
+[[Image(TestTopology.jpg)]]
+
+The above diagram represents all major "classes" of connections that are between between physical devices.  these connections are implemented as required per test.
+
+To set up the test network the following steps were necessary:
+
+ 1. Configure VM test hosts.
+ 1. Configure each DUT.
+   1. management IP addresses.
+   1. management network connections.
+   1. port and VLAN assignment.
+
+== Configuration ==
+All VMs and DUTs have management IP addresses on the network 128.89.91.0/24, with physical connection via wasabi and/or naboo.  This IP network is not used for any test traffic.
+
+=== VM hosts ===
+The following settings are from #506 and #537.
+
+|| '''host''' || '''DNS''' || '''IP''' ||
+|| azzalle || azzalle.gpolab.bbn.com   || 128.89.91.9 ||
+|| gotland || gotland.gpolab.bbn.com || 128.89.91.10 ||
+|| lagnace || lagnace.gpolab.bbn.com || 128.89.91.11 ||
+|| skaldia  || skaldia.gpolab.bbn.com || 128.89.91.12 ||
+
+Vm host's test network IP addresses and physical interfaces on Naboo.
+|| '''host''' || '''IP''' || '''[eth1] -- NIC on naboo''' ||
+|| azzalle || 10.20.1.9 || naboo[vmnic3] (2nd NIC on left card) ||
+|| gotland ||  10.20.1.10 || naboo[vmnic0] (3rd NIC on left card) ||
+||  lagnace || 10.20.1.11 || naboo[vmnic1] (4th NIC on left card) ||
+|| skaldia || 10.20.1.12 ||:naboo[vmnic6] (the top NIC on the second card) ||
+
+=== DUT's Port and VLAN Assignments ===
+It is advantageous to use the same port assignments for each DUT to ensure consistency and prevent confusion. Each DUT will have the same port assignment as shown below. This configuration allows for a single configuration to accommodate all planned testing without reconfiguration between tests.
+
+[[Image(PortstwoQinQtunnels.jpg)]]
+
+|| '''Port''' || '''Note''' ||
+|| 1  || QinQ Port ||
+|| 2  || management port ||
+|| 3  || extra management port reserved for direct connection ||
+|| 4  || host port ||
+|| 5  || cvlan trunk - to port 6 ||
+|| 6  || svlan access - to port 5 ||
+|| 7  || cvlan trunk - to port 8 ||
+|| 8  || svlan access - to port 7 ||
+|| 9  || Not used ||
+|| 10 || host port ||
+|| 11 || host port ||
+|| 12 || host port || 
+|| 13  || cvlan trunk - to port 14 ||
+|| 14  || svlan access - to port 13 ||
+|| 15  || host port ||
+|| 16 || Not used ||
+
+=== Management IP Addresses ===
+||'''switch''' || '''IP subnet''' || '''VLAN''' || '''management port''' || '''wasabi's port''' ||
+|| poblano || 128.89.91.6/25 || 900 || 2 || gi0/2 ||
+|| habanero || 128.89.91.7/25 || 900  || 2 || gi0/3 ||
+|| basil || 128.89.91.8/25 || 900 || 2 || gi0/4 ||
+See OpsNumbering for gateway, broadcast, etc. 
+
+= Test Outline  =
+This section outlines the various tests to perform on a DUT as well as between DUTs.
+
+== Unit Tests ==
+These tests are performed on a single DUT.
+
+=== QinQ Tagging ===
+'''Purpose'''[[BR]]
+Verify that a given DUT's QinQ port sends double-tagged QinQ frames in the expected format. For switches to understand that the trunking mechanism is a QinQ VLAN trunk the Ethernet's Header must contain the appropriate QinQ header field  type indication (0x8a88).
+
+[[Image(QinQTaggingTestOverview.jpg)]]
+
+'''Method''' [[BR]]
+Verify frames originating from the test host are tagged as appropriate using the Wireshark host.
+
+ * HP:   cvlans and svlans  are used to distinguish port type. svlan trunk (QinQ) ports use the 0x88a8 value.
+ * NEC: configures a QinQ trunk-port explicitly with the setting "switch dot1q ethertype 8a88" for a given port.
+ * CISCO: The access port for the QinQ portion needs configured for QinQ, the QinQ trunk (ES) port is configured with 0x88a8. 
+
+=== QinQ with OpenFlow ===
+'''Purpose'''[[BR]]
+Verify that QinQ can operate within an OpenFlow enabled switch. This test verifies that a DUT can be configured to control QinQ VLANS while running the OpenFlow software. OpenFlow will not be configured to perform any flow-based traffic shaping.
+
+'''Method''' [[BR]]
+Enable OpenFlow and perform all experiments.
+
+=== QinQ Tag Exclusivity ===
+'''Purpose'''[[BR]]
+This test insures that the customer VLAN ID and service VLAN ID ranges are mutually exclusive.
+
+[[Image(QinQExclusitivityOverview.jpg)]]
+
+'''Method''' [[BR]]
+The DUT is configured with service VLAN 667 used for QinQ as well as a customer VLAN 667. If successful the Wireshark host should see a frame with outer tag ID 667 as well as inner tag 667.
+
+=== VLAN and QinQ VLAN on Same Port ===
+'''Purpose'''[[BR]]
+This test explores the behavior of allowing a normal VLAN  trunk and a service VLAN (QinQ) trunk to be allowed on the same port.
+
+[[Image(CustomerAndServiceSamePort.jpg)]]
+
+'''Method''' [[BR]]
+ * VLAN 128 is a customer VLAN going out of port 1. 
+ * Service VLAN 667 tunneling  customer VLAN 3702 will also be trunked on port 1.
+ * Use wireshark to verify the appropriate frames are tagged for VLAN 128 or QinQ-tagged with outer VLAN 667 and inner VLAN 3702.
+
+== Integration Tests ==
+
+===  QinQ between DUTs ===
+'''Purpose'''[[BR]]
+Verify that hosts in the same VLAN on opposite sides of a QinQ tunnel can communicate.
+
+[[Image(QinQBetweenDUTs.jpg)]]
+
+'''Method''' [[BR]]
+Ping between hosts in the same tunneled VLAN.
+
+=== MTU's in QinQ  ===
+'''Purpose'''[[BR]]
+For QinQ to work efficiently, the ports trunking QinQ frames must accept allow for a frame size of 1526. A particular vendor's switch will require correct MTU configuration to prevent fragmentation.  
+
+According to 802.3ac frame size was increased to 1522 to allow a 1500 Byte MTU for VLAN tags. QinQ Trunks would require an MTU of 1504.
+
+
+   ||use                            || header size || MTU size || total size || switch MTU command|| note                       ||  
+   ||standard ethernet   || 18                || 1500     || 1518       || none  required    || standard frame             ||
+   ||802.1q VLAN trunk  || 18 + 4        || 1500     || 1522       || none required     || effects header only        ||
+   ||QinQ VLAN "tunnel" || 18 + 4        || 1500 + 4 || 1526       || system MTU 1504   || MTU adjusted for inner tag ||   
+
+This ignores other MTU modification requirements such as:
+ *  MPLS VPN pass-through (two 4-byte labels)
+ * Various Frame in Frame tunneling schemes (18 Bytes to 50 Bytes)
+
+'''Method'''   [[BR]]
+{{{
+ping 10.20.1.11 -M do -s "$((1500-20-8))" -c 1 > MTU_validation.txt
+}}}
+ * -M do: (return error if ping packet would fragment)
+ * -s packetsize: size of payload:
+   * 1500 = desired MTU
+   * 20 = IP header size (would be IN the frame's MTU)
+   * 8 = ICMP header size (would be IN the frame's MTU)
+
+=== VLAN and QinQ VLAN on Same Port ===
+'''Purpose'''[[BR]]
+If this was feasible for two DUTs this test verifies that the hosts in this VLAN on separate DUTs can communicate.
+
+[[Image(CustomerAndServiceSamePortQinQ.jpg)]]
+
+'''Method'''[[BR]] 
+Simply ping between hosts in the same customer VLAN going out of the service port.
+
+=== Latency: VLAN vs QinQ ===
+'''Purpose'''[[BR]]
+This test compares the best-case Round Trip latency of two hosts using a standard VLAN trunk and then a QinQ VLAN trunk.
+
+'''VLAN Latency'''[[BR]]
+
+[[Image(LatencyVlan.jpg)]]
+
+'''QINQ Latency'''[[BR]]
+As this test relies on end-to-end host  connectivity over a QinQ tunnel, the setup is the same as "the test QinQ between DUTs". This diagram is included again here for completeness.
+[[Image(LatencyQinQ.jpg)]]
+
+'''Method'''[[BR]] 
+This will be accomplished by using ping to report the round trip latency over 10 seconds. This test will ping the corresponding host once to   "primes the queue" to prevent the ARP request from the first ping skewing the max, average and mdev values.
+{{{
+ping 10.20.1.11 -c 1; ping 10.20.1.11 -c 10 | tee aFile.txt
+}}}
+
+=== QinQ Traffic Simulations ===
+'''Purpose'''[[BR]]
+This test will involve inter-VLAN traffic tunneled across a QinQ Tunnel.
+
+[[Image(TwoQinQTunnelsOneVlanPerTunnel.jpg)]]
+
+'''Method'''[[BR]] 
+ Use iperf to generate multiple TCP and UDP streams.
+
+'''TCP'''[[BR]]
+TCP streams allow for max throughput simulations.
+{{{
+iperf -c 10.20.1.11 -i 60 -t $(("60*60*8"))>aFile.txt &
+
+server:
+iperf -s -i 60 > aFile.txt &
+}}}
+
+'''UDP'''[[BR]]
+As UDP doesn't have an ACK mechanism it is necessary to "prime the queue" to prevent the server from dropping any traffic due to fast-sender issues. This is necessary as iperf UDP server would see large amounts of dropped traffic until the ARP request resolved.
+{{{
+client:
+ping -c 1 10.20.1.11; iperf -c 10.20.1.11 -u -i 1 -b 50M | tee aFile.txt
+
+server:
+iperf -s -u -i 1 | tee aFile.txt
+}}}
+
+----
+
+= Unit Testing =
+
+== NEC ==
+
+=== Configuration ===
+VLAN port participation:
+{{{
+poblano# show vlan config
+Date 2010/04/17 17:00:15 UTC
+VLAN counts:12
+ID   Name             Status  Ports
+   1 VLAN0001         Down    -
+ 128 VLAN0128         Up      0/1,0/4
+ 667 VLAN0667         Up      0/1,0/6,0/13-15
+ 668 VLAN0668         Up      0/1,0/8
+ 900 VLAN0900         Up      0/2-3
+3701 VLAN3701         Down    -
+3702 VLAN3702         Up      0/5,0/11
+3703 VLAN3703         Up      0/7,0/12
+3704 VLAN3704         Up      0/5,0/10
+}}}
+
+VLAN port membership with trunking configuration. Only relevant interface info  is shown, for complete configuration see the NEC Appendix.
+{{{
+interface gigabitethernet 0/1
+  switchport dot1q ethertype 88a8
+  switchport mode trunk
+  switchport trunk allowed vlan 128,667-668
+!
+interface gigabitethernet 0/2
+  switchport mode access
+  switchport access vlan 900
+!
+interface gigabitethernet 0/3
+  switchport mode access
+  switchport access vlan 900
+!
+interface gigabitethernet 0/4
+  switchport mode access
+  switchport access vlan 128
+!
+interface gigabitethernet 0/5
+  switchport dot1q ethertype 8100
+  switchport mode trunk
+  switchport trunk allowed vlan 3702,3704
+!
+interface gigabitethernet 0/6
+  switchport mode dot1q-tunnel
+  switchport access vlan 667
+!
+interface gigabitethernet 0/7
+  switchport dot1q ethertype 8100
+  switchport mode trunk
+  switchport trunk allowed vlan 3703
+!
+interface gigabitethernet 0/8
+  switchport mode dot1q-tunnel
+  switchport access vlan 668
+!
+interface gigabitethernet 0/10
+  switchport mode access
+  switchport access vlan 3704
+!
+interface gigabitethernet 0/11
+  switchport mode access
+  switchport access vlan 3702
+!
+interface gigabitethernet 0/12
+  switchport mode access
+  switchport access vlan 3703
+!
+interface gigabitethernet 0/13
+  switchport dot1q ethertype 8100
+  switchport mode trunk
+  switchport trunk allowed vlan 667
+!
+interface gigabitethernet 0/14
+  switchport mode dot1q-tunnel
+  switchport access vlan 667
+!
+interface gigabitethernet 0/15
+  switchport mode access
+  switchport access vlan 667
+}}}
+
+=== QinQ Tagging ===
+ the NEC correctly tagged the frames for QinQ transmission, as shown below.
+Wireshark sees QinQ double-tagged frame 667:3702. (e.g. 667 is the outer vlan, 3702 is the "wrapped" vlan)
+{{{
+Ethernet frame
+    type: 802.1ad Provider Bridge (QinQ) (0x88a8)
+    IEEEE 802.1ad ID:667
+        ID667
+        type 802.1Q virtual LAN (0x8100) ID 3702
+            ID 3702
+            IP (0x08000)
+                PAYLOAD
+}}}
+
+[[Image(NEC_QinQ_inner_3702_outer_667.jpg)]]
+
+=== QinQ with OpenFlow ===
+
+All QinQ testing was conducted while Poblano was running OpenFlow 0.9 firmware; no flows were active. Regardless of which Firmware is used to boot the device (original or OpenFlow) the start-up configuration is retained.
+
+Ports and VLANs can either be used by OpenFlow or as part of the production network. Future tests will explore QinQ with multiple active flows.
+
+=== Customer VLAN  out QinQ Port ===
+{{{
+128 VLAN0128         Up      0/1,0/4 
+}}}
+
+VLAN 128's ports were configured as follows:
+{{{
+interface gigabitethernet 0/1
+  switchport dot1q ethertype 88a8
+  switchport mode trunk
+  switchport trunk allowed vlan 128,667-668
+!
+interface gigabitethernet 0/4
+  switchport mode access
+  switchport access vlan 128
+}}}
+
+[[Image(NEC_VLAN128NoQinQ.jpg)]]
+
+VLAN 128 is capable of being sent out port 1, however it's tagged type is "0x8a88" (service VLAN), "not 0x8100" (customer VLAN). This implies that the Switch on the other side  of the trunk must be a service VLAN; sending the VLAN as a "normal" VLAN isn't possible in this configuration.
+
+
+=== Same Inner and Outer VLAN Tags ===
+{{{
+ 667 VLAN0667         Up      0/1,0/6,0/13-15
+}}}
+From VLAN 667's port participation it's appears that there's no distinction between customer and service VLANs despite various ports being configured to tag for QinQ vs normal tagging. If all the ports are indeed in the same VLAN the VLAN trunk (aka "jumper) from ports13<->14 should create a broadcast storm which would be observable on port 1 (STP is disabled). 
+
+{{{
+1. connecte azzalle to Nec's port 15
+2. Wireshark connected to NEC port 1.
+3  On azzalle: 10.20.1.11 -c 1 
+}}}
+
+[[Image(NEC_SameIds_broadcast_storm.jpg)]]
+A broadcast storm was induced from a single ping packet. The image also shows the continual nesting of VLAN headers as the frame continues to loop between access and trunk ports. From this it doesn't look possible to tunnel the same customer and service VLAN using one switch. This is an artifact caused by trying to emulate, on a single physical switch, service and customer VLANS of the with the same VLAN ID bridged with an Ethernet cable. 
+
+----
+
+== HP ==
+=== Overview ===
+The HP needs explicit configuration to operate using both customer and service VLANS.  This configuration is applied as the preparatory global step of assigning mixed vlan mode.  
+
+=== Configuration ===
+'''NOTE:''' The HP Procurve manual states that the switch only supports 2048 VLANS (half of the usual 4096). 
+
+'''Note:''' Make sure the command below is the first configuration step.  Setting or changing this value causes the switch to immediately reboot – and completely wipe its running configuration during the process – to take effect.
+{{{
+qinq mixedvlan
+}}}
+
+Ports used in svlans are not allowed to participate in the GARP VLAN Registration Protocol (GVRP).  The switch prompts when an incorrect assignment is attempted and entering the following command fixes the problem:
+{{{
+int 1 unknown-vlans disable
+}}}
+For this reason, VLAN 667 (in the test example) was configured as an svlan and 3702 as a vlan.
+
+{{{
+habanero# show vlan
+
+ Status and Counters - VLAN Information
+
+  Maximum VLANs to support : 2000                 
+  Primary VLAN : DEFAULT_VLAN
+  Management VLAN :            
+
+ VLAN ID Name                 Type  | Status     Voice Jumbo
+  ------- -------------------- ----- + ---------- ----- -----
+  1       DEFAULT_VLAN         cvlan | Port-based No    No   
+  128     VLAN128              svlan | Port-based No    No   
+  667     VLAN667              svlan | Port-based No    No   
+  668     VLAN668              svlan | Port-based No    No   
+  900     VLAN900              cvlan | Port-based No    No   
+  3702    VLAN3702             cvlan | Port-based No    No   
+  3703    VLAN3703             cvlan | Port-based No    No   
+  3704    VLAN3704             cvlan | Port-based No    No   
+}}}
+
+{{{
+vlan 1 
+   name "DEFAULT_VLAN" 
+   untagged 9,13-48,49-50,51-52 
+   no untagged 1-8,10-12 
+   no ip address 
+   exit 
+vlan 3702 
+   name "VLAN3702" 
+   untagged 11 
+   tagged 5 
+   no ip address 
+   exit 
+vlan 3703 
+   name "VLAN3703" 
+   untagged 12 
+   tagged 7 
+   no ip address 
+   exit 
+vlan 3704 
+   name "VLAN3704" 
+   untagged 10 
+   tagged 5 
+   no ip address 
+   exit 
+vlan 900 
+   name "VLAN900" 
+   untagged 2-3 
+   ip address 128.89.91.7 255.255.255.128 
+   exit 
+qinq mixedvlan
+svlan 128
+   name "VLAN128"
+   tagged 1
+   untagged 4
+   exit
+svlan 667
+   name "VLAN667"
+   tagged 1
+   untagged 6
+   exit
+svlan 668
+   name "VLAN668"
+   tagged 1
+   untagged 8
+   exit
+}}}
+
+=== QinQ Tagging ===
+The HP setup worked similarly to the NEC. However to specify a VLAN as used for QinQ the VLAN must be marked as a service VLAN. Note the HP needs to be configured to support mixed VLANs (see HP appendix) before this can be configured - changing this setting removes all VLAN configuration from the switch. 
+
+Wireshark sees QinQ double tagged frame 667:2702 (e.g. 667 is the outer vlan, 3702 is the "wrapped" vlan).
+Wireshark reports the correct QinQ frame header type
+{{{
+Ethernet frame
+    type: 802.1ad Provider Bridge (QinQ) (0x88a8)
+    IEEEE 802.1ad ID:667
+        ID667
+        type 802.1Q virtual LAN (0x8100) ID 3702
+            ID 3702
+            IP (0x08000)
+                PAYLOAD
+}}}
+
+[[Image(HP QinQ tags.jpg)]]
+
+=== QinQ with OpenFlow ===
+
+All QinQ testing was conducted while Habanero was running OpenFlow 0.9 firmware; no flows were active. Regardless of which Firmware is used to boot the device (original or OpenFlow) the start-up configuration is retained.
+
+Ports and VLANs can either be used by OpenFlow or as part of the production network. Future tests will explore QinQ with multiple active flows.
+
+=== Customer VLAN Over QinQ Port ===
+
+Creating VLAN 128 as a customer VLAN did not work; adding port 1 to the VLAN's participating ports failed:
+{{{
+habanero(vlan-128)# tagged 1  
+Ports 1 will lose their svlan memberships.
+Do you want to continue? [y/n] n
+}}}
+Clearly, port 1 should remain a service port to stay a QinQ port - the answer was no.
+
+However, deleting the customer vlan 128 and then creating a service VLAN 128 worked:
+{{{
+habanero(config)# svlan 128
+habanero(svlan-128)# interface svlan 128
+habanero(svlan-128)# untagged 4
+Interfaces that are GVRP enabled cannot be members of svlans.
+Use the interface level 'unknown-vlans' command to disable port gvrp.
+habanero(svlan-128)# exit     
+habanero(config)# interface 4 unknown-vlans disable 
+habanero(config)# interface svlan 128
+habanero(svlan-128)# untagged 4
+Ports 4 will lose their cvlan memberships.
+Do you want to continue? [y/n] y
+habanero(svlan-128)# show vlan 128
+habanero(svlan-128)# tagged 1
+habanero(svlan-128)# show vlans 128
+
+ Status and Counters - VLAN Information - VLAN 128
+
+  VLAN ID : 128    
+  Name : VLAN128             
+  Type : svlan
+  Status : Port-based
+  Voice : No 
+  Jumbo : No 
+
+  Port Information Mode     Unknown VLAN Status    
+  ---------------- -------- ------------ ----------
+  1                Tagged   Disable      Up        
+  4                Untagged Disable      Down      
+}}}
+
+This behavior seems consistent with the NEC. It is possible to send a non-tunneled VLANs out a service trunk port. However, those VLANs would be service VLANs, not customer VLANs.
+
+=== Same Inner and Outer VLAN Tags ===
+Tried to create cvlan of 667 fails as there's already a svlan with 667. 
+{{{
+habanero(config)# vlan 667
+VLAN type mismatch. VID 667 is of type 'svlan'.
+}}}
+This was not as expected; HP's distinction between customer and service VLANs seems to imply (besides the implicit type tagging on trunk ports) that this configuration would be possible. Though surprising, this behavior is consistent with the NEC.
+
+
+----
+
+== Cisco ==
+=== Overview ===
+The cisco 3750 requires the SFP ES module for QinQ operation. installed as well as the appropriate licensing. See http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps5532/prod_qas09186a00801eb822.html  Note that all configuration in this section refers to "port 1" This refers to this ES port (!GigabitEthernet?1/1/1 ). Ticket #533 outlines the procedure for getting the Cisco configured properly.
+
+
+
+=== Configuration ===
+ 
+ '''Trunk Negotiation'''::
+To allow for QinQ the Cisco Discovery protocol (CDP) should be disabled per dot1q (normal) VLAN tunk port. 
+{{{
+interface FastEthernet1/0/6
+ switchport access vlan 667
+ switchport mode dot1q-tunnel
+ no cdp enable                  <--HERE
+!         
+interface FastEthernet1/0/8
+ switchport access vlan 668
+ switchport mode dot1q-tunnel
+ no cdp enable                    <--HERE
+}}}
+
+ '''MTU'''::
+The standard `system mtu` does not change ES port configuration. To adjust the MTU on these ports use the `system mtu jumbo` command to allow for QinQ tagging.
+{{{
+    3750(config)# system mtu jumbo 9000
+    3750(config)# exit
+    3750# reload
+}}}
+
+ ''' QinQ access ports'''::
+The QinQ Access ports (which connect to the standard dotq trunk ports) are configured to by using `switchport mode dot1q-tunnel.
+{{{
+ interface FastEthernet1/0/6
+ switchport access vlan 667
+ switchport mode dot1q-tunnel
+ no cdp enable
+}}}
+
+ ''' QinQ Trunk ports'''::
+The QinQ trunk ports are set to participate in the same VLANs as the QinQ access ports. The trunk ethertype is set to the QinQ type: 0x88A8
+{{{
+ interface GigabitEthernet1/1/1
+ switchport trunk allowed vlan 128,667,668
+ switchport mode trunk
+ switchport nonegotiate
+ switchport trunk dot1q ethertype 88A8
+ speed auto 1000
+}}}
+
+=== QinQ Tagging ===
+Wireshark sees QinQ double tagged frame 667:2702 (e.g. 667 is the outer vlan, 3702 is the "wrapped" vlan). Wireshark reports the correct QinQ frame header type
+{{{
+Ethernet frame
+    type: 802.1ad Provider Bridge (QinQ) (0x88a8)
+    IEEEE 802.1ad ID:667
+        ID667
+        type 802.1Q virtual LAN (0x8100) ID 3702
+            ID 3702
+            IP (0x08000)
+                PAYLOAD
+}}}
+
+=== QinQ with OpenFlow ===
+None - Cisco doesn't support OpenFlow firmware.
+
+=== Customer VLAN Out QinQ Port ===
+VLAN 128 is capable of being sent out ES port 1, however it's tagged type is "0x8a88" (service VLAN), "not 0x8100" (customer VLAN). This implies that the Switch on the other side of the trunk must be a service VLAN; sending the VLAN as a "normal" VLAN isn't possible in this configuration. This is the same behavior as the NEC switch.
+
+=== Same Inner and Outer VLAN Tags ===
+Simular to NEC configuration ports with identical inner and outer VLAN IDs, when connected together via a jumer, cause a broadcast storm.
+
+----
+
+= Interoperability Testing =
+
+== NEC<->HP ==
+
+=== QinQ Between DUTs ===
+Ping from azzalle to gotland succeeded.
+
+=== MTU's in QinQ ===
+QinQ between NEC and HP can transmit an MTU of 1500 without fragmentation. 
+
+Results:
+{{{
+ping 10.20.1.11 -M do -s "$((1500-20-8))" -c 1 > MTU_validation.txt
+1480 bytes from 10.20.1.11: icmp_seq=1 ttl=64 time=0.434 ms
+}}}
+while adding 1 byte gave:
+{{{
+ping 10.20.1.11 -M do -s "$((1500-20-8+1))" -c 1
+From 10.20.1.9 icmp_seq=1 Frag needed and DF set (mtu = 1500)
+}}}
+
+'''NOTE'''
+Poblano's current configuration for QinQ , due to previous experimental configuration, has a QinQ MTU of 1508 and  a VLAN Trunk MTU of 1504 - This configuration is different than 802.3ac, but may be correct for this vendor.  Habanero doesn't currently have any explicit MTU settings.   Setting Poblano's QinQ MTU to 1504 and the VLAN trunk MTU to 1500 results in no "Echo" nor "Destination Unreachable (Fragmentation required, and DF flag set)" responses. This seems to imply that the ICMP Echo Request successfully made it to the destination host, but the ICMP Echo Response was dropped when leaving  the HP's QinQ port.  More investigation is required.
+
+See the NEC IP8800 Manual: Configuration Settings, Vol. 3, section 1.4.3 for more information.
+=== VLAN and QinQ VLAN on Same Port ===
+
+After configuring both the NEC's and HP's VLAN 128 to be a service VLAN (as outlined in the Unit test sections) end-to-end communication was possible.
+
+=== Latency: VLAN vs QinQ ===
+
+Ping across the tested ports. The first "ping -c 1"  "primes the queue" to prevent the ARP
+request from the first ping skewing the max, average and mdev values.
+{{{
+ping 10.20.1.11 -c 1; ping 10.20.1.11 -c 10 | tee aFile.txt
+}}}
+
+'''VLAN Trunk Only'''
+{{{
+--- 10.20.1.11 ping statistics ---
+10 packets transmitted, 10 received, 0% packet loss, time 9066ms
+rtt min/avg/max/mdev = 0.192/0.239/0.315/0.036 ms
+}}}
+
+'''QinQ Trunk'''
+{{{
+--- 10.20.1.11 ping statistics ---
+10 packets transmitted, 10 received, 0% packet loss, time 9072ms
+rtt min/avg/max/mdev = 0.196/0.244/0.284/0.038 ms
+}}}
+
+With all the extra hops for QinQ, an added RTL of 0.244-0.239=0.005ms seems more than reasonable. Given the short length of cabling this is a close approximation to "switching delay". Again, this was an Ideal baseline; there was no other traffic on the link for this baseline measurement.
+
+=== QinQ Traffic Simulations ===
+
+'''TCP'''[[BR]]
+Naboo's VM hosts were capping out at ~430 Mbps for TCP traffic (+/- 7Mbps based on quick scanning of my iperf log files per 10sec over 10 minutes) This is a limitation of Naboo (VM server) and is not a limitation of any DUTs. This was with only 1 pair communicating - full 1Gb capacity was available. Testing both pairs over QinQ still resulted in transmission of ~430Mbps per pair (logged every minute over 8 hours). I noticed no downward performance trend - but again I am currently eyeballing. With two end-to-end pairs, we're still under the max capacity of the link.
+
+{{{
+iperf -c 10.20.1.11 -i 60 -t $(("60*60*8"))>file.txt &
+}}}
+
+
+'''UDP'''[[BR]]
+UDP testing revealed minimal packet-loss and jitter was 0.023 ms ± 0.002ms.  
+
+{{{
+ping -c 1 10.20.1.11; iperf -c 10.20.1.11 -u -i 1 -b 50M
+}}}
+
+
+----
+
+== NEC<->CISCO ==
+=== QinQ Between DUTs ===
+Ping from azzalle to gotland succeeded. 
+
+=== MTU's in QinQ ===
+QinQ between NEC and Cisco can transmit an MTU of 1500 without fragmentation.
+
+Results:
+{{{
+ping 10.20.1.11 -M do -s "$((1500-20-8))" -c 1 > MTU_validation.txt
+1480 bytes from 10.20.1.11: icmp_seq=1 ttl=64 time=0.434 ms
+}}}
+while adding 1 byte gave:
+{{{
+ping 10.20.1.11 -M do -s "$((1500-20-8+1))" -c 1
+From 10.20.1.9 icmp_seq=1 Frag needed and DF set (mtu = 1500)
+}}}
+
+=== VLAN and QinQ VLAN on same port ===
+After configuring both the NEC's and Cisco's VLAN 128 to be a ''service'' VLAN (as outlined in the Unit test sections) end-to-end communication was possible. 
+
+=== Latency: VLAN vs QinQ ===
+Was not tested
+
+=== QinQ Traffic Simulations ===
+'''TCP'''[[BR]]
+Naboo's VM hosts were capping out at ~230 Mbps for TCP traffic (+/- 7Mbps based on quick scanning of my iperf log files per 10sec over 10 minutes) This is a limitation of Naboo (VM server) and is not a limitation of any DUTs. This was with only 1 pair communicating - full 1Gb capacity was available. This bandwidth is less than the amount possible when the NEC<->HOP tests were conducted. However, there are quite a few more VMs on Naboo the now.
+
+'''UDP'''[[BR]]
+UDP testing indicated no packet-loss. 
+
+----
+
+== HP<->CISCO ==
+=== QinQ Between DUTs ===
+=== MTU's in QinQ ===
+QinQ between NEC and HP can transmit an MTU of 1500 without fragmentation.
+
+Results:
+{{{
+ping 10.20.1.11 -M do -s "$((1500-20-8))" -c 1 > MTU_validation.txt
+1480 bytes from 10.20.1.11: icmp_seq=1 ttl=64 time=0.434 ms
+}}}
+
+while adding 1 byte gave:
+{{{
+ping 10.20.1.11 -M do -s "$((1500-20-8+1))" -c 1
+From 10.20.1.9 icmp_seq=1 Frag needed and DF set (mtu = 1500)
+}}}
+
+See the NEC IP8800 Manual: Configuration Settings, Vol. 3, section 1.4.3 for more information. 
+
+=== VLAN and QinQ VLAN on same port ===
+After configuring both the HP's and Cisco's VLAN 128 to be a ''service'' VLAN (as outlined in the Unit test sections) end-to-end communication was possible. 
+
+=== Latency: VLAN vs QinQ ===
+Was not tested
+
+=== QinQ Traffic Simulations ===
+'''TCP'''[[BR]]
+Naboo's VM hosts were capping out at ~230 Mbps for TCP traffic (+/- 7Mbps based on quick scanning of my iperf log files per 10sec over 10 minutes) This is a limitation of Naboo (VM server) and is not a limitation of any DUTs. This was with only 1 pair communicating - full 1Gb capacity was available. This bandwidth is less than the amount possible when the NEC<->HOP tests were conducted. However, there are quite a few more VMs on Naboo the now.
+
+'''UDP'''[[BR]]
+UDP testing indicated no packet-loss. 
+
+----
+
+= Appendix =
+
+== NEC ==
+See ["hwNecIP8800"] for reference and usage.
+
+=== Configuration ===
+{{{
+poblano# show running-config 
+#Last modified by operator at Sat Apr 17 17:58:25 2010 with version 11.1.C
+!
+hostname "poblano"
+!
+ip host poblano 128.89.91.6
+!
+ip domain name bbn.com
+!
+ip name-server 128.33.0.20
+!
+vlan 1
+  name "VLAN0001"
+!
+vlan 22
+  name "BBN OpenFlow 1"
+!
+vlan 23
+  name "BBN OpenFlow 2"
+!
+vlan 24
+  name "BBN OpenFlow Control Vlan"
+!
+vlan 128
+!
+vlan 667
+!
+vlan 668
+!
+vlan 900
+!
+vlan 3701
+!
+vlan 3702
+!
+vlan 3703
+!
+vlan 3704
+!
+spanning-tree disable
+spanning-tree mode pvst
+!
+interface gigabitethernet 0/1
+  media-type rj45
+  mtu 1508
+  switchport dot1q ethertype 88a8
+  switchport mode trunk
+  switchport trunk allowed vlan 128,667-668
+!
+interface gigabitethernet 0/2
+  media-type rj45
+  switchport mode access
+  switchport access vlan 900
+!
+interface gigabitethernet 0/3
+  media-type rj45
+  switchport mode access
+  switchport access vlan 900
+!
+interface gigabitethernet 0/4
+  media-type rj45
+  switchport mode access
+  switchport access vlan 128
+!
+interface gigabitethernet 0/5
+  mtu 1504
+  switchport dot1q ethertype 8100
+  switchport mode trunk
+  switchport trunk allowed vlan 3702,3704
+!
+interface gigabitethernet 0/6
+  mtu 1504
+  switchport mode dot1q-tunnel
+  switchport access vlan 667
+!
+interface gigabitethernet 0/7
+  mtu 1504
+  switchport dot1q ethertype 8100
+  switchport mode trunk
+  switchport trunk allowed vlan 3703
+!
+interface gigabitethernet 0/8
+  mtu 1504
+  switchport mode dot1q-tunnel
+  switchport access vlan 668
+!
+interface gigabitethernet 0/9
+  switchport mode dot1q-tunnel
+  switchport access vlan 22
+!
+interface gigabitethernet 0/10
+  switchport mode access
+  switchport access vlan 3704
+!
+interface gigabitethernet 0/11
+  switchport mode access
+  switchport access vlan 3702
+!
+interface gigabitethernet 0/12
+  switchport mode access
+  switchport access vlan 3703
+!
+interface gigabitethernet 0/13
+  mtu 1504
+  switchport dot1q ethertype 8100
+  switchport mode trunk
+  switchport trunk allowed vlan 667
+!
+interface gigabitethernet 0/14
+  switchport mode dot1q-tunnel
+  switchport access vlan 667
+!
+interface gigabitethernet 0/15
+  switchport mode access
+  switchport access vlan 667
+!
+interface gigabitethernet 0/16
+  switchport mode dot1q-tunnel
+  switchport access vlan 23
+!
+interface gigabitethernet 0/17
+  switchport mode trunk
+  switchport trunk allowed vlan 22-23
+!
+interface gigabitethernet 0/18
+  switchport mode access
+  switchport access vlan 24
+!
+interface gigabitethernet 0/19
+  switchport mode access
+!
+interface gigabitethernet 0/20
+  switchport mode access
+!
+interface gigabitethernet 0/21
+  switchport mode access
+!
+interface gigabitethernet 0/22
+  switchport mode access
+!
+interface gigabitethernet 0/23
+  switchport mode access
+!
+interface gigabitethernet 0/24
+  switchport mode access
+!
+interface tengigabitethernet 0/25
+  switchport mode access
+!
+interface tengigabitethernet 0/26
+  switchport mode access
+!
+interface vlan 1
+!
+interface vlan 24
+  ip address 171.67.74.60 255.255.255.240
+  no ip proxy-arp
+!
+interface vlan 900
+  ip address 128.89.91.6 255.255.255.128
+!
+ip route 0.0.0.0 0.0.0.0 128.89.91.1
+!
+line vty 0 2
+!
+ftp-server
+!
+ntp server 192.1.100.189
+ntp server 192.1.249.10
+!
+poblano#   
+
+}}}
+
+=== Useful Commands ===
+The following are some notes taken while learning the NEC switch syntax:
+See ["hwNecIP8800"] for more discussion.
+
+'''Getting started'''
+ * login: operator
+
+'''Administrative commands'''
+Enable mode (necessary to do just about anything and doesn’t prompt for a password)
+{{{
+enable  
+}}}
+enter configuration mode
+{{{
+configure       Enter configuration mode
+(When making configuration changes, the console prints a "!" character to indicate there are unsaved changes...)
+}}}
+Password management
+{{{
+clear password <username>  #Clear the user's password.  
+                                                     #The password utility does not allow setting a NULL password
+                                                     #use this command to clear it
+password <username>          #Change a user's password - defaults to the currently logged-in user
+}}}
+
+'''config (mode)'''
+{{{
+save    Save current configuration – look for the “!” characters!
+vlan [vlan number]      Activate the specified VLAN.  No parameters assigned, just the entry is available to be used elsewhere...
+interface gig eth 0/1   Select the interface.  It's really "interface gigabitethernet 0/1" but it allows abbreviations when they are non-ambiguous...
+interface vlan [vlan id]        Select the vlan to configure.  The only (useful) thing I've found is the ability to associate a VLAN with an IP address and THAT was only useful to define the HOME interface of the device...
+media [rj45/sfp]        While configuring an interface.  Ports 1-4 are dual-option, RJ-45 is ethernet, SFP is the fiber port.  One or the other is enabled...
+switch dot1q ethertype 8a88     Switches the ethertype announced between 802/1ad (“tunnel”) and 802.1q mode on this interface.
+}}}
+
+----
+
+== HP ==
+
+See ["hwHpProcurve6600"] for reference and usage.
+
+=== Current Configuration ===
+'''NOTE''' the actual password for the HP has been replaced with XXXXX for display. If you intend on using this output as a configuration you must replace XXXXX with the appropriate password.
+{{{
+habanero# show running-config 
+
+Running configuration:
+
+; J9452A Configuration Editor; Created on release #K.14.53o
+
+hostname "habanero" 
+max-vlans 2000 
+module 2 type J94yyA 
+module 3 type J94zzA 
+module 5 type J94wwA 
+module 6 type J94wwA 
+no stack 
+interface 1 
+   unknown-vlans Disable 
+exit
+interface 4 
+   unknown-vlans Disable 
+exit
+interface 6 
+   unknown-vlans Disable 
+exit
+interface 8 
+   unknown-vlans Disable 
+exit
+ip default-gateway 128.89.91.1 
+vlan 1 
+   name "DEFAULT_VLAN" 
+   untagged 9,13-48,49-50,51-52 
+   no untagged 1-8,10-12 
+   no ip address 
+   exit 
+vlan 3702 
+   name "VLAN3702" 
+   untagged 11 
+   tagged 5 
+   no ip address 
+   jumbo 
+   exit 
+vlan 3703 
+   name "VLAN3703" 
+   untagged 12 
+   tagged 7 
+   no ip address 
+   jumbo 
+   exit 
+vlan 3704 
+   name "VLAN3704" 
+   untagged 10 
+   tagged 5 
+   no ip address 
+   jumbo 
+   exit 
+vlan 900 
+   name "VLAN900" 
+   untagged 2-3 
+   ip address 128.89.91.7 255.255.255.128 
+   exit 
+qinq mixedvlan
+svlan 128
+   name "VLAN128"
+   tagged 1
+   untagged 4
+   exit
+svlan 667
+   name "VLAN667"
+   tagged 1
+   untagged 6
+   exit
+svlan 668
+   name "VLAN668"
+   tagged 1
+   untagged 8
+   exit
+jumbo ip-mtu 1508
+jumbo max-frame-size 1526
+sntp server priority 1 192.1.249.10 3
+ip ssh filetransfer
+snmp-server community "public" unrestricted
+oobm
+   ip address dhcp-bootp
+   exit
+no tftp client
+no tftp server
+no autorun
+password XXXXX
+}}}
+
+=== Useful Commands ===
+HP ProCurve 6600 Useful Commands
+The following are some notes taken while learning the HP switch syntax:
+(no login username)
+Enters "setup" screen to set things like the name (habanero), IP address and netmask:
+{{{
+setup   
+        System Name: habanero
+        Default Gateway: 128.89.72.1
+        IP Config [Manual]
+        Spanning Tree Enabled [No]
+        IP Address: 128.89.72.141
+        Subnet Mask: 255.255.254.0
+        "Save" saves and exits...
+}}}
+
+show relevant info
+{{{
+show [stuff]
+}}}
+
+IP address information
+{{{
+show ip 
+}}}
+
+VLAN information
+{{{
+show vlan       
+}}}
+
+'''VLANs Can Be Tagged and Untagged'''
+Resets the configuration to a default state - make sure the current config is backed up before doing it!)
+{{{
+qinq mixedmode  
+}}}
+
+'''Other VLAN commands'''
+vlan refers to customer VLANs, svlan refers to service VLANs
+{{{
+vlan 3702 untagged 37           #Causes port 37 to stop tagging VLAN 3702 traffic (undoes a “vlan tag” command)
+svlan 667 tagged 38                     #Causes port 38 to tags traffic as SVLAN 667 (turns it into a tunneling-trunk port)
+int 45 unknown-vlans disable    #Disables GVRP [needed for SVLAN membership]
+int 45 unknown-vlans learn      #Enables GVRP
+int 46 qinq port-type customer-network
+vlan 1 no tagged 37,45,46       Removes the ports from VLAN 1 if they were configured as untagged participants
+vlan 1 no untagged 37,45,46     Removes the ports from VLAN 1 if they were configured as tagging participants
+}}}
+
+----
+
+== Cisco ==
+ See ["hwCiscoCatalyst3750"] for reference and usage.
+
+=== Configuration ===
+{{{
+basil#show running-config 
+Building configuration...
+
+Current configuration : 2772 bytes
+!
+version 12.2
+no service pad
+service timestamps debug uptime
+service timestamps log uptime
+no service password-encryption
+!
+hostname basil
+!
+enable secret 5 $1$m1O6$lT/GyoO4dZOw0bvD9j/wH/
+enable password operator
+!
+no aaa new-model
+system mtu routing 1500
+ip subnet-zero
+!
+vtp mode transparent
+!
+no file verify auto
+!
+spanning-tree mode pvst
+spanning-tree extend system-id
+no spanning-tree vlan 1,128,667-668,900,3702-3704
+!         
+!
+!
+vlan internal allocation policy ascending
+!
+vlan 128,667-668,900,3702-3704 
+!
+!
+interface FastEthernet1/0/1
+!
+interface FastEthernet1/0/2
+ switchport access vlan 900
+ switchport mode access
+!
+interface FastEthernet1/0/3
+ switchport access vlan 900
+ switchport mode access
+!
+interface FastEthernet1/0/4
+ switchport access vlan 128
+ switchport mode access
+!
+interface FastEthernet1/0/5
+ switchport trunk encapsulation dot1q
+ switchport trunk allowed vlan 3702,3704
+ switchport mode trunk
+ switchport nonegotiate
+!
+interface FastEthernet1/0/6
+ switchport access vlan 667
+ switchport mode dot1q-tunnel
+ no cdp enable
+!
+interface FastEthernet1/0/7
+ switchport trunk encapsulation dot1q
+ switchport trunk allowed vlan 3703
+ switchport mode trunk
+ switchport nonegotiate
+!
+interface FastEthernet1/0/8
+ switchport access vlan 668
+ switchport mode dot1q-tunnel
+ no cdp enable
+!
+interface FastEthernet1/0/9
+!
+interface FastEthernet1/0/10
+ switchport access vlan 3704
+ switchport mode access
+!
+interface FastEthernet1/0/11
+ switchport access vlan 3702
+ switchport mode access
+!
+interface FastEthernet1/0/12
+ switchport access vlan 3703
+ switchport mode access
+!
+interface FastEthernet1/0/13
+ switchport trunk encapsulation dot1q
+ switchport trunk allowed vlan 667
+ switchport mode trunk
+ switchport nonegotiate
+!
+interface FastEthernet1/0/14
+ switchport access vlan 667
+ switchport mode dot1q-tunnel
+ no cdp enable
+!
+interface FastEthernet1/0/15
+ switchport access vlan 667
+ switchport mode access
+!
+interface FastEthernet1/0/16
+!
+interface FastEthernet1/0/17
+!
+interface FastEthernet1/0/18
+!
+interface FastEthernet1/0/19
+!
+interface FastEthernet1/0/20
+!
+interface FastEthernet1/0/21
+!
+interface FastEthernet1/0/22
+!
+interface FastEthernet1/0/23
+!
+interface FastEthernet1/0/24
+!
+interface GigabitEthernet1/0/1
+!
+interface GigabitEthernet1/0/2
+!
+interface GigabitEthernet1/1/1
+ switchport trunk allowed vlan 128,667,668
+ switchport mode trunk
+ switchport nonegotiate
+ switchport trunk dot1q ethertype 88A8
+ speed auto 1000
+ no cdp enable
+!
+interface GigabitEthernet1/1/2
+ speed auto 1000
+!
+interface Vlan1
+ no ip address
+!
+interface Vlan900
+ ip address 128.89.91.8 255.255.255.128
+!
+ip classless
+ip http server
+!
+!
+!
+!
+!
+control-plane
+!
+!
+line con 0
+ exec-timeout 0 0
+line vty 0 4
+ password operator
+ login
+line vty 5 15
+ password operator
+ login
+!
+end
+}}}
+=== Useful Commands ===
+
+----
+
+== Related Wiki Pages ==
+ * OpsSwitchConfig - guide for configuring switches for use in lab network
+ * OpsNumbering - Various configuration paramters, IP dresses, gateways, etc.
+
+== Related Tickets ==
+ * #333 - Initial NEC  (and later HP) configuration.
+ * #533 - Initial exploration of QinQ support for Cisco switch.
+ * #501 - QinQ report for QinQ capabilities of NEC switch (depends on this and the performance testing). Outlines Report.
+ * #535 - QinQ interoperability testing (this document).
+ * #539 - Install NEC OpenFlow Firmware.
+ * #456 - Install HP OpenFlow Firmware.
+ * #506 - Internal network configuration for switch management IPs and host VMS for testing. 
+ * #537 - Configuring switches for internal network management access.