wiki:ClickExampleExperiment

Version 13 (modified by Mark Berman, 7 years ago) (diff)

--

Note - Improvements Coming

Important Note

This page is being revised to improve the example experiment. The instructions may be inconsistent or confusing until updates are complete.

Please click here to access the most recent working version.





Example Experiment - Click Routers

In this example experiment, you will configure and run a non-IP software routing configuration, using the Click modular router and ProtoGENI hosts. In this example, we'll be running click in user mode. Once you have the prerequisites in place, you should be able to complete this example experiment in under an hour.

Please note that you can't just cut and paste all of the commands. There are additional instructions in the text.

Prerequisites

Before beginning this experiment, you should:

  • Have a GENI credential. If you don't, check out SignMeUp.
  • Configure omni (version 1.5.2 or later) on your machine. Be sufficiently comfortable with omni to verify that a listresources command works and to know when your slice is ready using sliverstatus.

Information on obtaining GENI credentials and omni is available at SignMeUp and HowToUseOmni or by contacting help@geni.net.

Setup

Create a new directory, click-example, on your machine. You will do the rest of your work from this directory.

mkdir click-example
cd click-example

Download the attached rspec file and save it in your click-example directory, using the name click-example.rspec. Be sure to use raw mode.

If you haven't already, add your omni source and examples directories to your PATH:

export PATH=$PATH:/path/to/omni/src:/path/to/omni/examples

If you haven't already, add your omni source directory to your PYTHONPATH:

export PYTHONPATH=$PYTHONPATH:/path/to/omni/src

(Optional) You'll be making heavy use of your private key to log into your ProtoGENI hosts in the steps below. If your key is encrypted, you may want to load it in a key manager so you don't have to type your passphrase many times. Alternately, you can remove the passphrase from your key using the steps below. (You don't need to do this step, however, you will be prompted for your passphrase quite a few times when following the steps below.)

To remove passphrase (assuming your private key file is id_rsa):

mv id_rsa id_rsa.encrypted
openssl rsa -in id_rsa.encrypted -out id_rsa
<type your passphrase when prompted>
chmod 400 id_rsa

Obtain your resources

Create your slice. Please don't use my stupid slice name. This example will use the Utah ProtoGENI site. You can choose a different site by selecting a different aggregate manager with the -a switch.

omni.py createslice -a http://www.emulab.net/protogeni/xmlrpc/am StupidSliceName

Create a sliver and add resources. (You changed the slice name, right?)

omni.py createsliver -a http://www.emulab.net/protogeni/xmlrpc/am StupidSliceName click-example.rspec

Wait until your sliver is ready, typically a few minutes. (You can monitor your sliver with Flack or use omni's sliverstatus command as shown below.)

omni.py sliverstatus -a http://www.emulab.net/protogeni/xmlrpc/am StupidSliceName

When your sliver is ready, run the readyToLogin script to get login information for your nodes. Use the same aggregate manager and slice name that you used for your sliverstatus command above.

readyToLogin.py -a http://www.emulab.net/protogeni/xmlrpc/am StupidSliceName

You'll get a big chunk of information, but you're interested in the ssh command information near the end.

... <lots of output> ...
================================================================================
Aggregate [http://www.emulab.net/protogeni/xmlrpc/am] has a ProtoGENI sliver.


pc278.emulab.net's geni_status is: ready
Login using:
	xterm -e ssh -i /Users/mberman/.ssh/id_rsa mberman@pc278.emulab.net &

pc336.emulab.net's geni_status is: ready
Login using:
	xterm -e ssh -i /Users/mberman/.ssh/id_rsa mberman@pc336.emulab.net &

pc347.emulab.net's geni_status is: ready
Login using:
	xterm -e ssh -i /Users/mberman/.ssh/id_rsa mberman@pc347.emulab.net &

pc358.emulab.net's geni_status is: ready
Login using:
	xterm -e ssh -i /Users/mberman/.ssh/id_rsa mberman@pc358.emulab.net &

pc348.emulab.net's geni_status is: ready
Login using:
	xterm -e ssh -i /Users/mberman/.ssh/id_rsa mberman@pc348.emulab.net &

pc353.emulab.net's geni_status is: ready
Login using:
	xterm -e ssh -i /Users/mberman/.ssh/id_rsa mberman@pc353.emulab.net &

================================================================================

The six lines that start with xterm -e ssh ... contain the information you need to connect to your nodes. There are six nodes in this experiment, and you will want to open a window for each. Copy and paste the commands, adding the -A switch to each ssh command as shown below. You will get six new terminal windows, one on each node of your experiment. (You should say yes if you are prompted to accept the RSA key of the remote host.)

xterm -e ssh -A -i /Users/mberman/.ssh/id_rsa mberman@pc336.emulab.net &

The shell prompt in each of your new terminal windows will tell you the host name. The four routers in your experiment are named top, left, right, and bottom. The two end hosts are named hosta and hostb. You may want to rearrange the windows on your screen, putting the router hosts in the locations indicated, with the end hosts off to the side.

Configure your routers

In each of your four router hosts, run the extractClickConfig script as shown below.

[mberman@top ~]$ /local/click-example/extractClickConfig.py 

You'll get output something like this:

Your host information:
        hostA: hostA.StupidSliceName.emulab-net.emulab.net pc347.emulab.net
        top: top.StupidSliceName.emulab-net.emulab.net pc336.emulab.net
        left: left.StupidSliceName.emulab-net.emulab.net pc358.emulab.net
        right: right.StupidSliceName.emulab-net.emulab.net pc278.emulab.net
        bottom: bottom.StupidSliceName.emulab-net.emulab.net pc348.emulab.net
        hostB: hostB.StupidSliceName.emulab-net.emulab.net pc353.emulab.net
Done.

(If you are prompted for a password, check to make sure that you provided the -A switch in your xterm command above.)

The extractClickConfig script produces router configurations for your experiment. It also creates a diagram of your experiment. Copy it back to the click-example directory on your local host so you can view it. There's a copy on each of your routers hosts, I've chosen top, but it doesn't matter. You'll need to copy the host name from the script output above.

scp top.StupidSliceName.emulab-net.emulab.net:myslice.png .

Your slice will look something like the one below (see sampleDiagram.png). The overall configuration should be the same, with two end hosts, named hostA and hostB, and four routers (top, left, right, bottom) in a diamond configuration. The host names, interface names, and MAC addresses will be different, depending on the actual resources assigned to your slice.

Example network configuration diagram

The four routers interconnected by solid lines are your "core network," which runs a non-standard, non-IP protocol. The dashed lines out to the end hosts carry standard IP traffic.

Start your routers

The extractor script produces a click configuration file for each of your routers. The deployment script, deployClick.sh copies these files to your routers and starts a user-space router on each. Run the script, and you should get output like this:

[mberman@molasses click-example]$ sh deployClick.sh 
Setting user shell on top.StupidSliceName.emulab-net.emulab.net
Changing shell for mberman.
Shell changed.
Copying top.click to top.StupidSliceName.emulab-net.emulab.net
Waiting for click to build on top.StupidSliceName.emulab-net.emulab.net ...................
Stopping any running click router on top.StupidSliceName.emulab-net.emulab.net
Disabling IP routing on top.StupidSliceName.emulab-net.emulab.net
Starting click router on top.StupidSliceName.emulab-net.emulab.net

<similar output for {left,right,bottom} routers>

Depending on how long you waited between creating your slice and running the deployClick.sh script (and how powerful your allocated router machines are), you may need to wait a few minutes while click finishes compiling.

You are now running a non-IP core network on your four routers, along with a (primitive) non-IP multipath routing algorithm. You're ready to experiment with this configuration.

Send some traffic

Now you'll log into your two edge hosts, hostA and hostB, and send some traffic between them. Since these end hosts are not running your modified protocol, they'll rely on the top and bottom routers to transform their IP packets into your modified protocol on entry to the core network and back into IP packets on exit.

In two different windows, use ssh -F ssh_config hostA and ssh -F ssh_config hostB to connect to your two end hosts. (Don't forget to work in your click-example directory.)

On hostB, instruct nc to listen for a UDP connection on port 24565 (or some other port that catches your fancy).

[mberman@hostb ~]$ nc -ul 24565

Then connect to it from hostA:

[mberman@hosta ~]$ nc -u hostb 24565

You've established a simple text chat connection. Enter a line of text in either window, and it should appear in the other. Of course to do this, the text is travelling through your core network, using your non-standard protocol and routing. So type a message into each window, and make sure it appears in the other.

That's it! Now, let's look inside to see what's going on.

Looking under the hood

Please note: the interface names and MAC addresses below are for the sample configuration shown in the figure above. You will want to refer to your network diagram to get the correct interfaces and addresses for your configuration.

Let's take a look at what's happening in the four routers in your configuration. There are two basic router configurations.

Packet transformation

The more interesting configuration appears here, in the top.click configuration file.

// This portion accepts IP packets,
// reformats them, and routes them
// to an internal router.
route :: Classifier(27/01%01,-);

modify :: Unstrip(2) ->
    StoreData(0, "AliceWasHere3546") ->
    route;

FromDevice(eth3, PROMISC true) -> 
    Classifier(12/0800) ->
    modify;

route[0] -> left :: EtherEncap(0x7744, 00:02:b3:35:f1:b7, 00:02:b3:86:1d:13) ->
    SimpleQueue ->
    Print(outL) ->
    ToDevice(eth1);

route[1] -> right :: EtherEncap(0x7744, 00:03:47:94:c7:fd, 00:02:b3:65:d1:2b) ->
    SimpleQueue ->
    Print(outR) ->
    ToDevice(eth2);

// This portion accepts non-IP packets
// with an ether type of 0x7744
// from an internal router, restores
// them to IP format, and forwards.
restore :: SimpleQueue ->
    Strip(30) ->
    EtherEncap(0x800, 00:03:47:73:8e:bd, 00:02:b3:3f:7a:a1) ->
    ToDevice(eth3);

FromDevice(eth1) -> Classifier(12/7744) -> Print(inL) -> restore;
FromDevice(eth2) ->  Classifier(12/7744) -> Print(inR) -> restore;

As indicated in the comments, the top portion of the configuration listens (FromDevice) for IP packets arriving on the interface connected to hostA (that's eth3 in this example). It then creates a new 16-byte field at the head of the packet (two bytes added by the Unstrip operation, plus the existing 14-byte Ethernet header. It fills that field with what could be important routing instructions, but in this case is just graffiti (StoreData). The route operation then routes the packet via either the left or right router toward hostB. In either case, it wraps the packet in a fresh Ethernet header (EtherEncap) with a distinctive ether type code (0x7744), logs the new packet on its way out (Print) and sends it out on the correct interface (ToDevice).

The bottom portion of the configuration is intended for packets coming out of the core network to hostA. It accepts packets from either the left or right router, logs them, strips off thirty bytes (Ethernet header plus your 16-byte new header field), puts on a fresh Ethernet header, and sends them along to hostA.

The configuration for the bottom router is exactly symmetric, routing packets between hostB and the core network, but using different graffiti.

Simple Forwarding

The left router configuration is much simpler. Here's the left.click file:

// Copy packets from top to bottom.
FromDevice(eth1) ->
    StoreEtherAddress(00:02:b3:86:1a:4b, dst) ->
    StoreEtherAddress(00:03:47:95:7a:fe, src) ->
    SimpleQueue ->
    Print(top) ->
    ToDevice(eth2);
// Copy packets from bottom to top.
FromDevice(eth2) ->
    StoreEtherAddress(00:02:b3:35:f1:b7, dst) ->
    StoreEtherAddress(00:02:b3:86:1d:13, src) ->
    SimpleQueue ->
    Print(bottom) ->
    ToDevice(eth1);

This configuration just blindly forwards packets. It picks up any packet from the top router, updates the Ethernet header, and passes it along to the bottom router. The same applies in the reverse direction. Again, the configuration for the right router is exactly analogous.

Monitoring your core network

Let's watch how the packets travel through the network. In addition to the windows you already have open to the two end hosts, you'll need four more windows to watch the logs for all four routers. In each window, you should use ssh to connect to one router host, ssh -F ssh_config top, ssh -F ssh_config left, ssh -F ssh_config right and ssh -F ssh_config bottom. You can monitor the click log files with tail:

[mberman@top ~]$ tail -f click.log 

The log files may have a couple of warnings at the top complaining that your interfaces have no IPv4 addresses. No worries, we did that on purpose, since we don't use IP in the core network. Other than those warnings, you should just see packet log messages, one line per packet. Type <enter> a few times in each router window to create a space below the existing log entries, so you can identify new log messages as they appear.

Now go to your window for hostA, where your nc command is still running. Type a message into this window. You should see a log message in three of your four router windows. In this example, you might see:

In the top router log:

outR:   76 | 0002b365 d12b0003 4794c7fd 7744416c 69636557 61734865

This log entry says that the top router received a packet from hostA, modified it, and sent it out to the right router. If the entry started with outL, that would indicate that it sent the packet out to the left router. Let's look a bit at the start of the packet (the first 24 bytes are logged). It starts with an Ethernet header. The first six bytes are the MAC address of the destination interface, that's 00:02:b3:65:d1:2b, the MAC address of eth1 on right. The next six bytes are the MAC address of the source interface, 00:03:47:94:c7:fd, or eth2 on top. Next comes your ether type, 0x7744. The remaining bytes, "416c 69636557 61734865" are the start of the first field in your new protocol, "AliceWasHe" in ASCII.

Since this packet was routed to the right, there's an entry in the right router log. This entry indicates that a packet was received from top. The logged contents show the packet sent to bottom, with rewritten MAC addresses, corresponding to eth2 on bottom and eth2 on right.

top:   76 | 00034794 c7fc0003 4794c1f7 7744416c 69636557 61734865

Finally, here's the entry on bottom:

inR:   76 | 00034794 c7fc0003 4794c1f7 7744416c 69636557 61734865

It shows the same packet received from right. After logging, the packet is rewritten into IP and sent to hostB.

Try typing a few different lines to hostA. You should see some packets routed to the left and some to the right. The routing decision is based on the route :: Classifier(27/01%01,-); entry in the top router configuration. Here, the router is looking at the low-order bit of the checksum on the initial IP packet (now at byte position 27 with the addition of the new sixteen byte field at the start of the header). Packets with odd checksums go to the left; those with even checksums go right.

Try typing a few lines to hostB. You should see similar behavior, but starting from bottom and working up. You'll also see the different value inserted in the new header field.

Clean up

When you're done, please release your resources so they'll be available to others.

/path/to/omni/src/omni.py deletesliver -a http://www.emulab.net/protogeni/xmlrpc/am StupidSliceName

Moving forward with your experiment

This sample experiment is just a very simple demonstration of how to run a Click-based routing configuration using ProtoGENI. For a more meaningful experiment, you may want to try some of the variations described below. We'd love to hear what you're doing with Click and GENI, and we're here to help. Please let us know at help@geni.net.

Improved routing

Instead of writing "AliceWasHere" in your packets, perhaps include some real routing instructions. Modify the Click configurations to route packets accordingly.

Richer topology

Incorporate additional hosts into a core network topology that's more interesting than a simple diamond.

Improved performance with kernel-level Click

The Click router supports kernel-level operation. The principles are the same, but the setup is a bit more involved. To use kernel-level Click, you will probably want first to develop and debug your setup at the user level. Then, follow the steps above, with these differences:

  • Before running createsliver, modify your rspec to delete the execute tags on your routers. You may also want to add a hardware_type tag to specify a particular hardware configuration so your performance comparisons are valid and so the next step doesn't take too long.
  • After running createsliver, you need to rebuild the kernel on your routers and then build Click with its corresponding kernel module. Log into each router, and run sh /local/build-click-kernel.sh to patch and build your new kernel. This process will take a while, after which the machine will reboot, and you'll need to log back in. Now build Click with sh /local/build-click.sh.
  • Run extractClickConfig.py as before (or create your Click configuration by hand), but you'll need to edit the deployClick.sh script to run Click at the kernel level. You do this by changing /usr/local/bin/click <file> to /usr/local/sbin/click-install <file> each time it occurs (four times in the example experiment above).

Wide-area routing

Modify your rspec to include some ProtoGENI hosts on the Internet2 backbone or at multiple ProtoGENI sites. Additional information available at http://protogeni.net or help@geni.net.

Attachments (3)

Download all attachments as: .zip