wiki:NetworkCore/BackboneLimitations

Version 2 (modified by tupty@bbn.com, 13 years ago) (diff)

--

Limitations for Connecting into the Backbone

There are some known limitations with the ways that sites currently connect to the GENI OpenFlow backbone in Spiral 3. The general problem is that no two sites can share non-OpenFlow gear in the path to the OpenFlow core without separation of broadcast domains. We can solve this problem in a few ways:

  • By using separate VLANs for sites that share a path to the core
  • By ensuring that sites do not share any gear in the path to the GENI Core
  • By expanding the OpenFlow core out closer to sites participating in GENI

Each of these solutions have practical limitations which are explained in the potential solutions section.

Internet2 ION

When sites want to establish a connection into the GENI OpenFlow backbone using Internet2 ION, they need to create a circuit between the endpoint of their ION provider and an endpoint that faces an OpenFlow backbone switch. The destination VLAN of the endpoint facing the backbone switch must be the same as the destination backbone VLAN. Each OpenFlow backbone switch has one single ION endpoint facing it, and ION's provisioning service will not let two circuits terminate on the same VLAN at the same ION endpoint. Therefore, for each backbone switch, only one site can use ION to reach a specific backbone VLAN.

The above example shows a situation where Rutgers has an ION circuit provisioned between its ION provider at MAGPI and the endpoint that faces the GENI OpenFlow backbone on VLAN 3715. If MAX tries to create an ION circuit that terminates at openflow.wash.ion.internet2.edu and is put onto VLAN 3715, ION will fail to provision the circuit.

NLR FrameNet

When a site connects to the GENI OpenFlow backbone through NLR FrameNet, they usually connect into the FrameNet VLAN that corresponds to the VLAN in the OpenFlow backbone. However, multiple sites that share a FrameNet endpoint cannot use the same VLAN before reaching the OpenFlow backbone; otherwise, their traffic will be sent through the normal switch processing path on the FrameNet switch, and it will bypass the OpenFlow processing in the backbone altogether. In addition, there is only a single connection between the FrameNet switch and the backbone switch, so VLAN translation cannot be done on the connection from the FrameNet switch.

The above example shows a situation where Wisconsin and BBN share a FrameNet endpoint. If BBN and Wisconsin both use the same FrameNet VLAN, then their traffic bypasses the OpenFlow backbone.

Potential Solutions to Limitations

Short Term Solutions

Physical Connections on Backbone Switch for VLAN Translation

If multiple sites want to share a FrameNet endpoint, then they can use different VLAN IDs in FrameNet to reach the OpenFlow backbone. However, this still leaves the problem of VLAN translation to be done on the backbone switches. Currently, this is done using cross-connect cables locally on the switch. This solution is not scalable, because it burns two ports on the OF switch per site that gets added in.

We are currently doing this in NLR Chicago OpenFlow switch because BBN and Wisconsin share a FrameNet switch. We are also doing this between Clemson and Georgia Tech because they share a Regional. We have not done anything like this with sites that use ION to connect to the GENI core, but that is also a possibility.

Use ProtoGENI Backbone Switches as OpenFlow Switches

Currently, there are ProtoGENI backbone switches deployed at a handful of Internet2 PoPs. The switches are OpenFlow capable HPs. If OpenFlow were enabled on the switches, and if they were hooked into the existing GENI OpenFlow backbone, then these switches could be leveraged as additional OpenFlow backbone ingress points. This option is highly dependent on using the ProtoGENI switches, which are currently maintained by the Utah ProtoGENI folks. These switches are already connected into ION.

Aggregate Switches Between I2/NLR and Backbone Switches

A switch could be inserted in between the OpenFlow backbone switch and the FrameNet switch or ION endpoint. This switch would take a trunk port from the FrameNet switch or ION endpoint, and then it would be hooked up to the backbone switch in a way that each VLAN would be translated upon entering the backbone switch. Deploying new switches for this is not an option, because this would cost a significant amount of money both in hardware costs and in operations costs; however, if the ProtoGENI switches deployed in at the Internet2 PoPs currently have a large number of free ports, then they could potentially be used as an aggregating switch. This option is highly dependent on using the ProtoGENI switches, which are currently maintained by the Utah ProtoGENI folks.

Increase Number of ION Endpoints Going to the Core

There is a possibility that we can increase the number of ION endpoints that connect to OpenFlow switches in the core. There may be unused ports on GENI equipment in the Internet2 core that could provide additional connections between core OpenFlow switches and ION equipment. Multiple sites could then hook into the same OpenFlow core switch on the same VLAN via ION. This requires that there are open ports on the ION device, and would also require the purchase of new transceiver modules. This option would provide temporary relief while we scale up, but this option does not scale to very many additional campuses. This solution would also need to translate to DYNES, which will eventually provide most Internet2 dynamic layer 2 connections for campuses.

Long Term Solutions

VLAN Translation in OpenFlow Switches

If OpenFlow switches could do fast VLAN translation internally, rather than through the use of VLAN cross-connect cables, then the current solutions for sites that share a FrameNet switch or an ION endpoint would scale well. We believe that this functionality currently exists in NEC production switches, although we have not verified it. We are unsure if this is supported by the IP8800s running the production firmware, or if new hardware is required. We are also unsure if many-to-one VLAN translation is supported. Testing and research is needed to see what hardware and software supports this feature, and whether or not the translation is done at wire speed.

Sites Directly Connect to Backbone

Some sites are physically close enough to an OpenFlow backbone switch that they may have the option to hook directly into the backbone switch with minimal expense. This solution has trade-offs, but for the limited number of sites that are currently near a backbone switch, this is a possibility. Direct connections are also possible for sites further away from backbone PoPs, but circuit costs tend to make this an unlikely option with the current topology. This solution will become more feasible as the NDDI deployment expands, providing more potential direct connection locations.

Regionals Use OpenFlow Switches

If Regionals had OpenFlow switches, then OpenFlow processing would be done before reaching a FrameNet or ION endpoint. Furthermore, sites that share a Regional would not need to pass through FrameNet or ION in order to reach the OpenFlow backbone. GENI intends to expand OpenFlow switch deployment to regional starting in Spiral 4, but this option will take time to establish.

Summary

The table below shows the cost of different solutions when bringing on a new site that shares a regional with another site. This table also assumes that both sites want to connect to the core through the same OpenFlow switch.

Solution Backbone Switch PortsRegional VLANsRegional PortsFrameNet VLANs/ION CircuitsExtra Hardware Additional Requirements
Short term
Physical Connections on OpenFlow Backbone Switch 2 per-site-per-VLAN 1 per-site-per-VLAN 1 per site 1 per-site-per-VLAN
Use PG Switches as OpenFlow Switches in Core 1 1 per-site-per-VLAN 1 per site 1 per-site-per-VLAN Use of ProtoGENI switches Enable OpenFlow on ProtoGENI switches
Aggregate Switches Between I2/NLR and Backbone Switches1 per-site-per-VLAN 1 per-site-per-VLAN 1 per site 1 per-site-per-VLAN Use of ProtoGENI Switches
Increase Number of ION Endpoints to Core 1 per new ION endpoint 1 per-site-per-VLAN 1 per site 1 per-site-per-VLAN 1 transceiver per new ION endpoint identify and reallocate existing unused GENI interfaces in I2 backbone
Long term
VLAN Translation in OpenFlow Switches 1 per-site-per-VLAN 1 per site 1 per-site-per-VLAN This needs to be implemented on a switch
Sites Directly Connect to Backbone 1 per site Site must be close to backbone switch to minimize cross connect cost
Regionals Use OpenFlow Switches 1 per core VLAN* 1 per site 1 per core VLAN* 1 OpenFlow switch per Regional

*The regional would need one VLAN per backbone VLAN in place, but they could remove all VLANs that are currently dedicated to specific sites, so this is basically no additional cost for regionals that already connect a site into the backbone.

Note: The costs in this table are relative to what we already have in place. In the units "per-site-per-VLAN", VLAN means backbone VLAN (i.e. 3715 and 3716).

Goals

Starting in the Summer of 2011, we are planning to increase the number of campuses which are interconnected through the GENI mesoscale deployment. Over the course of about 12 months, we are targeting an increase to 50 campuses, and these should all somehow be connected to the GENI OpenFlow backbone. Through some combination of the proposed solutions, we believe that this is an achievable goal. We would like the backbone to be as scalable and flexible as possible, therefore the long term solutions are preferable; however, given what we currently have to work with, we will be using the short term solutions where necessary.

Attachments (3)

Download all attachments as: .zip