wiki:GENIExperimenter/Tutorials/WiMAXOpenFlow/Design-Setup

Version 5 (modified by Ryan Izard, 10 years ago) (diff)

--

OpenFlow-Based Vertical Handoff over WiFi and WiMAX in the Orbit Testbed

Image Map

1. Design the Experiment

The goal of this tutorial is to perform a vertical handoff over WiFi and WiMAX in the Orbit testbed. Let's talk specifics with repect to each Orbit node. As briefly mentioned in the Overview, the experiment will be conducted with three Orbit nodes -- the client, the server, and the AP. Up first, we have the client node. The client is equipped with an Intel 6250 WiFi/WiMAX network adapter and an Atheros 5000 series WiFi network adapter. The Intel 6250 has a limitation – it can support the operation of either WiFi or WiMAX but not both simultaneously. As such, the Intel 6250 will serve as the WiMAX adapter and the Atheros 5000 series will serve as the WiFi adapter on the client. The OpenFlow-based handoff solution will choose which of these network interfaces it would like to use in order to relay messages to the server. Speaking of which, the server is equipped with the same wireless cards as the client. In order to receive packets from the client over both WiFi and WiMAX, the server will listen for the client's traffic over both its WiFi and WiMAX interfaces. To do this seamlessly, the server will also implement an OpenFlow-based solution. More details about how OpenFlow is used in both the server and the client are provided as the tutorial progresses. The third and final Orbit node necessary for this tutorial is the AP. Like the client and the server, the AP also contains both an Intel 6250 and an Atheros 5000 series wireless card; however, it will only use the Atheros card. The Atheros card on the AP is used by the software package hostapd to provide a WiFi network for both the client and the server machines. If you are interested in expanding upon this experiment, it is worth noting that Atheros cards play very well with hostapd-based networks; however, some Intel cards (such as the Intel 6250) are not directly supported. This is the reason we choose to use the Atheros over the Intel 6250 to provide the WiFi network.

Now, in order to provide a seamless handoff, the application should not be aware that the handoff takes place. The "seamlessness" of the handoff is still in development by the SDN team at Clemson. But for starters, from a networking point of view, if a network interface is brought up or is taken down, the IP address will be added to or removed from that network interface. This means the application's network socket will be broken if a break-before-make handoff occurs (which is oftentimes the case). When an application socket breaks, that means the application can no longer communicate over then network with it, and if this happens, the application must have some scheme to recreate the socket, otherwise the connection will be lost. In order to put as little reliance on the application layer as possible, a virtual tap interface is used. All applications will bind to the IP on this tap interface, and since tap interfaces are not physical, the only time it can “go down” is if we as programmers/network administrators take it down ourselves. Thus, it is a reliable way to make sure the application's socket stays “up”. In this tutorial, all application traffic will originate from and terminate at the tap interface.

The end-user will use the virtual tap interface (created by OpenVPN) for network connections, while the handoff execution will handle which physical interface to use. Now, the metric that decides which interface to use and when to perform the handoff is beyond the scope of this tutorial. Instead, this tutorial is designed to provide a basic framework for a handoff. To conduct the handoff, the Static Flow Pusher API in Floodlight, the OpenFlow controller, is used to insert OpenFlow flows manually, as determined by the handoff decision (e.g. a novel and super-cool metric a GENI experimenter has implemented). A Python script leverages the Static Flow Pusher API to add and remove flows.

The next logical question is where should this Static Flow Pusher insert the flows? In order to leverage the capabilities of OpenFlow, we need to have hardware and/or software that supports its use. Hence, Open vSwitch (OVS) is used in this tutorial. OVS is a software package that implements a virtual OpenFlow switch. This tutorial uses a pre-installed and pre-configured OVS network that incorporates the WiFi, WiMAX, and tap interfaces of the client as ports of virtual switches. The following is a picture depicting the general OVS topology used in this tutorial.

Each physical and tap interface has a corresponding OVS bridge. (In OVS terminology, an "OVS bridge" is essentially an "OpenFlow switch." It is called a bridge rather than a switch, since from the Linux kernel's perspective, the OVS bridge "bridges" the physical interface it is associated with.) These OVS bridges are linked together as if you actually plugged a physical switch into another with an Ethernet cable. The interfaces of the machine appear as ports on the OVS bridges, as do the internal links between the OVS bridges. With OVS, one can build a vitual network contained entirely within a single entity -- namely the client and server nodes of this tutorial. Each will employ internal OVS networks to alter the default behavior of the kernel with regard to networking. In essence, OVS allows the experimenter to circumvent the networking kernel and allow OpenFlow flows to route and modify packets instead.

2. Establish the Environment

  1. Using the Orbit images for the client, server, and AP, load each onto a separate node in an Orbit testbed. Make sure the node supports the hardware requirements for the experiment (Intel 6250 WiMAX and Atheros 5000 or 9000 series for the client and server, and Atheros 5000 or 9000 series for the AP). Boot these images after they are loaded and SSH into each machine.
  1. Remove the Forwarding module from the Floodlight OpenFlow controller. Floodlight uses what it calls a module loading system, where the user can write modules to perform a certain task or set of tasks. Each module can register for certain events. For example, the Forwarding module registers for PACKET_IN events where the controller is sent a packet from a connected switch. Upon such an event, the Forwarding module will send the packet out the correct port(s) depending on the destination. This module essentially implements a standard learning switch function where the OpenFlow-enabled switch behaves as if it were a standard network switch. We do not want this functionality, since we would like to have exclusive control over which port(s) our packets get forwarded.
    1. In the client and the server nodes, remove the Forwarding module.
    2. The module loading system maintains a list of the modules to be loaded at runtime. To remove the Forwarding module from this list (and thus disable it), open the /root/floodlight-0.90/src/main/resources/floodlightdefault.properties file and remove the line net.floodlightcontroller.forwarding.Forwarding,\.
    3. Browse to the root directory of the Floodlight project -- /root/floodlight-0.90. Execute ant in the terminal. ant is a Java-based build tool to build and compile Java projects. Upon success, ant will produce an updated executable jar file in the /root/floodlight-0.90/target directory named floodlight.jar.
  2. In the client, determine the MAC addresses of the WiMAX and WiFi network interface cards (NICs) for use in OpenFlow flows.
    1. Load the kernel modules for each of the NICs by executing modprobe i2400m_usb and modprobe ath5k, respectively.
    2. After the kernel modules are loaded, the devices should be visible in the system. Execute ifconfig wmx0 and ifconfig wlan1, respectively and note the MAC addresses of each NIC. These will be used in later steps.
      $ ifconfig wmx0
         eth0      Link encap:Ethernet  HWaddr 00:0c:29:04:5c:41  
             UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
             RX packets:0 errors:0 dropped:0 overruns:0 frame:0
             TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
             collisions:0 txqueuelen:1000 
             RX bytes:0 (8.3 KiB)  TX bytes:0 (6.1 KiB)
             Interrupt:18 Base address:0x1424 
      $ ifconfig wlan1
         eth1      Link encap:Ethernet  HWaddr 00:0c:29:04:5c:4b  
             UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
             RX packets:0 errors:0 dropped:0 overruns:0 frame:0
             TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
             collisions:0 txqueuelen:1000 
             RX bytes:0 (31.0 KiB)  TX bytes:0 (5.1 KiB)
             Interrupt:19 Base address:0x14a4 
      
  3. Also in the client, customize the setup script gec18_setup.sh in /root/StartupScripts. This script is designed to (1) define user variables, (2) configure the tap interface with OpenVPN, (3) start Floodlight, (4) initialize and start OpenVswitch, and (5) configure Linux networking.
$ ifconfig
eth0      Link encap:Ethernet  HWaddr 00:0c:29:04:5c:41  
    inet addr:192.168.93.128  Bcast:192.168.93.255  Mask:255.255.255.0
    inet6 addr: fe80::20c:29ff:fe04:5c41/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
    RX packets:48 errors:0 dropped:0 overruns:0 frame:0
    TX packets:38 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000 
    RX bytes:8503 (8.3 KiB)  TX bytes:6274 (6.1 KiB)
    Interrupt:18 Base address:0x1424 

eth1      Link encap:Ethernet  HWaddr 00:0c:29:04:5c:4b  
    inet addr:192.168.193.132  Bcast:192.168.193.255  Mask:255.255.255.0
    inet6 addr: fe80::20c:29ff:fe04:5c4b/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
    RX packets:152 errors:0 dropped:0 overruns:0 frame:0
    TX packets:26 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000 
    RX bytes:31818 (31.0 KiB)  TX bytes:5234 (5.1 KiB)
    Interrupt:19 Base address:0x14a4 

eth2      Link encap:Ethernet  HWaddr 00:0c:29:04:5c:55  
    inet addr:192.168.193.129  Bcast:192.168.193.255  Mask:255.255.255.0
    inet6 addr: fe80::20c:29ff:fe04:5c55/64 Scope:Link
    UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
    RX packets:138 errors:0 dropped:0 overruns:0 frame:0
    TX packets:31 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:1000 
    RX bytes:27881 (27.2 KiB)  TX bytes:6277 (6.1 KiB)
    Interrupt:16 Base address:0x1824 

lo        Link encap:Local Loopback  
    inet addr:127.0.0.1  Mask:255.0.0.0
    inet6 addr: ::1/128 Scope:Host
    UP LOOPBACK RUNNING  MTU:16436  Metric:1
    RX packets:8 errors:0 dropped:0 overruns:0 frame:0
    TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
    collisions:0 txqueuelen:0 
    RX bytes:560 (560.0 B)  TX bytes:560 (560.0 B)

Note the subnets and names given to each of the network interfaces. Recall, when the VM was initialized, we configured 1 NAT interface and 2 host-only interfaces. The two interfaces on the same subnet are the host-only interfaces. Make notes of each interface name and its IP and subnet mask.

  1. With this information, to the setup script directory:
    $ cd /root/06-03-13
    $ ls
    ... system_setup.sh ...
    
  2. Open the script with the text editor of your choice (vi, gedit, pico, nano, etc):
    $ gedit system_setup.sh
    
  3. There are numerous user defined variables at the top of the script. These are placeholders for commonly used system and configuration specific information throughout the script. We need to change a few of them to suit our needs for this tutorial. Modify the physical interface names for IFACE_wlan0 and IFACE_wimax to match those names of the host-only interfaces noted from ifconfig. Also, modify the IFACE_tap_IP variable to be an IP in the same subnet of the host-only interfaces (e.g. 192.168.193.155 would work for the host-only subnet 192.168.193/24).
    ###################
    #USR DEF VARIABLES#
    ###################
    IFACE_bridge_eth=br_eth
    IFACE_bridge_wlan0=br_wifi0
    IFACE_bridge_wimax=br_wimax
    IFACE_bridge_int=br_tap
    
    IFACE_tap=tap0
    IFACE_tap_IP=192.168.193.110
    IFACE_ethernet=eth0
    IFACE_wlan0=eth1
    IFACE_wimax=eth2
    
  4. Next, we need to create our tap interface. This is the network interface that will funnel all outbound packets from the userspace on our VM and send them into our OpenVswitch network.
    ###############
    #ADD TAP IFACE#
    ###############
    
    echo "OVPN: Installing tap interface, $IFACE_tap"
    openvpn --mktun --dev $IFACE_tap --lladdr 12:51:16:90:8f:ee
    
  5. Now, it is sometimes desirable to automate the start of Floodlight; however, for the purposes of this tutorial, we will launch it from within Eclipse. Comment out the following lines of the system_setup.sh script:
    ##################
    #START FLOODLIGHT#
    ##################
    
    echo "FL: Starting Floodlight..."
    cd / && ((java -jar ./root/floodlight/target/floodlight.jar) > floodlight-output 2>&1 &)
    echo "FL: Finished!"
    
  6. Next, we need to insert the OpenVswitch kernel module:
          ###################
          #START OPENVSWITCH#
          ###################
          
          echo "OVS: Configuring OVS..."
          echo "OVS: Checking for kernel module..."
          if [ -e $(lsmod | grep openvswitch) ]      
          then
    	   echo "OVS: ...inserting kernel module"
               /sbin/rmmod bridge
               sleep 1
    	   insmod /root/openvswitch-1.7.1/datapath/linux/openvswitch.ko
          else
    	   echo "OVS: ...kernel module already present"
          fi
    
  7. After that, we need to initialize OpenVswitch and its database:
    echo "OVS: Creating database"
    ovsdb-server --remote=punix:/usr/local/var/run/openvswitch/db.sock \
    --remote=db:Open_vSwitch,manager_options \
    --private-key=db:SSL,private_key \
    --certificate=db:SSL,certificate \
    --bootstrap-ca-cert=db:SSL,ca_cert \
    --pidfile --detach
    echo "OVS: Initializing OVS..."
    ovs-vsctl --no-wait init
    echo "OVS: Starting OVS..."
    ovs-vswitchd --pidfile --detach
    
  8. Now, we need to take down any pre-existing OVS bridges:
          echo "OVS: Removing any existing bridge, $IFACE_bridge_eth $IFACE_bridge_wlan0"
          echo "OVS: $IFACE_bridge_wimax $IFACE_bridge_int ..."
          
          ...
          
          ## Wlan0
          echo "OVS: Removing any existing bridge, $IFACE_bridge_wlan0..."
          if [ -n "$(ovs-vsctl show | grep $IFACE_bridge_wlan0)" ]      
          then
               echo "OVS: ...removing $IFACE_bridge_wlan0"
               ovs-vsctl del-br $IFACE_bridge_wlan0
          fi
          
          ## WiMAX
          if [ -n "$(ovs-vsctl show | grep $IFACE_bridge_wimax)" ]
          then
    	   echo "OVS: ...removing $IFACE_bridge_wimax"
    	   ovs-vsctl del-br $IFACE_bridge_wimax
          fi
          
          ## Internal
          if [ -n "$(ovs-vsctl show | grep $IFACE_bridge_int)" ]
          then
    	   echo "OVS: ...removing $IFACE_bridge_int"
    	   ovs-vsctl del-br $IFACE_bridge_int
          fi
    
          ######################
          #ADD INTERNAL BRIDGES#
          ######################
    
          ...
    
          ## Wlan0
          echo "OVS: Adding interface bridge, $IFACE_bridge_wlan0..."
          ovs-vsctl add-br $IFACE_bridge_wlan0
          echo "OVS: ...with port $IFACE_wlan0"
          ovs-vsctl add-port $IFACE_bridge_wlan0 $IFACE_wlan0 -- set Interface $IFACE_wlan0 ofport=$port_eth1
          echo "OVS: ...with port $IFACE_patch_wlan0_to_tap"
          ovs-vsctl add-port $IFACE_bridge_wlan0 $IFACE_patch_wlan0_to_tap -- set Interface $IFACE_patch_wlan0_to_tap ofport=$port_wlan0_to_tap
    
          ## WiMAX
          echo "OVS: Adding interface bridge, $IFACE_bridge_wimax..."
          ovs-vsctl add-br $IFACE_bridge_wimax
          echo "OVS: ...with port $IFACE_wimax"
          ovs-vsctl add-port $IFACE_bridge_wimax $IFACE_wimax -- set Interface $IFACE_wimax ofport=$port_eth2
          echo "OVS: ...with port $IFACE_patch_wimax_to_tap"
          ovs-vsctl add-port $IFACE_bridge_wimax $IFACE_patch_wimax_to_tap -- set Interface $IFACE_patch_wimax_to_tap ofport=$port_wimax_to_tap
    
          ## Internal
          echo "OVS: Adding interface bridge, $IFACE_bridge_int..."
          ovs-vsctl add-br $IFACE_bridge_int
          echo "OVS: ...with port $IFACE_tap"
          ovs-vsctl add-port $IFACE_bridge_int $IFACE_tap -- set Interface $IFACE_tap ofport=$port_tap0
          #echo "OVS: ...with port $IFACE_patch_tap_to_eth"
          #ovs-vsctl add-port $IFACE_bridge_int $IFACE_patch_tap_to_eth -- set Interface $IFACE_patch_tap_to_eth ofport=$port_tap_to_eth
          echo "OVS: ...with port $IFACE_patch_tap_to_wlan0"
          ovs-vsctl add-port $IFACE_bridge_int $IFACE_patch_tap_to_wlan0 -- set Interface $IFACE_patch_tap_to_wlan0 ofport=$port_tap_to_wlan0
          echo "OVS: ...with port $IFACE_patch_tap_to_wimax"
          ovs-vsctl add-port $IFACE_bridge_int $IFACE_patch_tap_to_wimax -- set Interface $IFACE_patch_tap_to_wimax ofport=$port_tap_to_wimax
    
  9. At this point, we're ready to set the patch ports between the OVS bridges. These create links between the OVS tap bridge and the OVS WiFi and WiMAX bridges in order to facilitate the flow of packets from the tap bridge to the physical interface of choice.
    #################
    #SET PATCH PORTS#
    #################
    
    ...
    
    echo "OVS: Patching ports $IFACE_patch_tap_to_wlan0, $IFACE_patch_wlan0_to_tap"
    ovs-vsctl set interface $IFACE_patch_tap_to_wlan0 type=patch
    ovs-vsctl set interface $IFACE_patch_tap_to_wlan0 options:peer=$IFACE_patch_wlan0_to_tap
    ovs-vsctl set interface $IFACE_patch_wlan0_to_tap type=patch
    ovs-vsctl set interface $IFACE_patch_wlan0_to_tap options:peer=$IFACE_patch_tap_to_wlan0
    
    echo "OVS: Patching ports $IFACE_patch_tap_to_wimax, $IFACE_patch_wimax_to_tap"
    ovs-vsctl set interface $IFACE_patch_tap_to_wimax type=patch
    ovs-vsctl set interface $IFACE_patch_tap_to_wimax options:peer=$IFACE_patch_wimax_to_tap
    ovs-vsctl set interface $IFACE_patch_wimax_to_tap type=patch
    ovs-vsctl set interface $IFACE_patch_wimax_to_tap options:peer=$IFACE_patch_tap_to_wimax
    
  10. Now, we need to assign each OVS bridge a unique ID (DPID) and point them to the address of the Floodlight controller. Floodlight will be run on the localhost, so the loopback address is defined within a the variable OVS_controllerIP.
    ##########
    #SET DPID#
    ##########
    
    ...
    
    ## Set Wlan0 DPID
    echo "OVS: Setting $IFACE_bridge_wlan0 DPID to $OVS_switchDPID_wlan0..."
    ovs-vsctl set bridge $IFACE_bridge_wlan0 other-config:datapath-id=$OVS_switchDPID_wlan0
    
    ## Set WiMAX DPID
    echo "OVS: Setting $IFACE_bridge_wimax DPID to $OVS_switchDPID_wimax..."
    ovs-vsctl set bridge $IFACE_bridge_wimax other-config:datapath-id=$OVS_switchDPID_wimax
    
    ## Set Tap DPID
    echo "OVS: Setting $IFACE_bridge_int DPID to $OVS_switchDPID_tap..."
    ovs-vsctl set bridge $IFACE_bridge_int other-config:datapath-id=$OVS_switchDPID_tap
    
    ...
    
    ## Wlan0
    echo "OVS: Connecting $IFACE_bridge_wlan0 to controller at $OVS_controllerIP"
    ovs-vsctl set-controller $IFACE_bridge_wlan0 tcp:$OVS_controllerIP
    
    ## WiMAX
    echo "OVS: Connecting $IFACE_bridge_wimax to controller at $OVS_controllerIP"
    ovs-vsctl set-controller $IFACE_bridge_wimax tcp:$OVS_controllerIP
    
    ## Internal
    echo "OVS: Connecting $IFACE_bridge_int to controller at $OVS_controllerIP"
    ovs-vsctl set-controller $IFACE_bridge_int tcp:$OVS_controllerIP
    ovs-vsctl set bridge br_tap other-config:hwaddr=12:51:16:90:8f:ee
    
    echo "OVS: Finished!"
    
  11. Now, the second-to-last thing to do in the setup script is to configure our network connections. We need to revoke the IPs from our physical interfaces and assign them to the OVS bridge interfaces corresponding to each inferface. This will allow us to inject data/packets into our OVS network. We also need to configure our OVS tap bridge as an available IP address in the same subnet as our VM's host-only network (noted earlier with ifconfig). And finally, we need to disable kernel IP forwarding.
    ##########################
    #CONFIGURE NETWORK ACCESS#
    ##########################
    
    ifconfig lo up
    
    ## Disable IP on physical interfaces
    echo "NTWK: Taking down $IFACE_wimax..."
    ifconfig $IFACE_wimax 0.0.0.0
    
    echo "NTWK: Taking down $IFACE_wlan0..."
    ifconfig $IFACE_wlan0 0.0.0.0
    
    ## Assign static IP addresses for bridge interfaces
    echo "NTWK: Assigning $IFACE_bridge_int IP as $IFACE_tap_IP..."
    ifconfig $IFACE_bridge_int $IFACE_tap_IP netmask 255.255.255.0
    
    ## Get an IP for bridge interfaces using DHCP
    echo "NTWK: Assigning $IFACE_bridge_wimax IP via dhclient..."
    dhclient $IFACE_bridge_wimax
    
    echo "NTWK: Assigning $IFACE_bridge_wlan0 IP via dhclient..."
    dhclient $IFACE_bridge_wlan0
    
    ...
    
    ## Turn off IP Forwarding
    echo "NTWK: Disabling IP Forwarding..."
    echo "0" > /proc/sys/net/ipv4/ip_forward
    
    ...     
    
    
  12. Finally, look towards the bottom of system_setup.sh script. Notice the lines:
    #To add 100ms to all outbound traffic on br_wimax
    #tc qdisc add dev br_wimax root netem delay 100ms
    
    This command allows you to add a simulated delay on a particular interface. So we can see the handoff when it occurs, we will add a 100ms delay to the br_wimax interface. Uncomment this line to do so.
  13. Save system_setup.sh and close your text editor.

Warnings

Warning Be on the lookout for typos in your scripts!

Notes

Note Write down your interface names, IP addresses, and subnet masks. All subnets must be the same for a Layer-2 handoff.

Tips

Tip If you need assistance, please ask for help!

Previous: Introduction

Next: Execute

Attachments (2)

Download all attachments as: .zip