[[PageOutline]] = Example Experiment - Click Routers = In this example experiment, you will configure and run a non-IP software routing configuration, using the [http://read.cs.ucla.edu/click/click Click] modular router and [http://protogeni.net ProtoGENI] hosts. In this example, we'll be running click in user mode. Please note that you can't just cut and paste all of the commands. There are additional instructions in the text. == 1. Add another user to your experiment == Omni gives you the capability of giving access to other users on your compute resources. Depending on which AM you are using to get resources from, this is done in a different way. Ask the team next to you about their username and do the following: i. While in a terminal, download their public key under `~/.ssh/` : {{{ cd ~/.ssh wget http://www.gpolab.bbn.com/experiment-support/gec15/adv-omni/pub-keys/_key.pub }}} i. Follow the instructions [wiki:HowTo/MulitpleUsersWithOmni these instructions] and add another user for ProtoGENI AMs == 1. Create your experiment == In this step, we are going to setup the experiment. In this tutorial we assume that you are sufficiently comfortable with omni to verify that a '''listresources''' command works and to know when your slice is ready using '''sliverstatus'''. i. '''Create a slice''', use the slicename given to you in the paper slip: {{{ omni.py createslice }}} i. '''Create a sliver''' using the rspec from the URL given in your paper slip: {{{ omni.py createsliver -a pg-utah }}} i. '''Check the status of your sliver''' {{{ omni.py sliverstatus -a pg-utah }}} == 2. Install scripts == While you wait for your sliver to become ready, we will see how we can automate the installation of our experiment with install scripts. In this experiment we are going to use software routers in order to write our own forwarding scheme. This means that in any experiment we are going to run we want the basic installation of the software router to always be present. The configuration might change from run to run, but the software should always be installed. The software to be installed, and the scripts to be executed at boot time, are defined in the rspecs. Follow these steps to locate your install script and identify the different parts. i. Download your rspec {{{ cd /tmp wget }}} i. Open your rspec and look for the `install` tag and copy the value of the URL attribute. i. Download and untar the software {{{ cd /tmp wget tar xvfz }}} i. Look in your rspec and locate the `execute` tag. Note what script is being executed at boot time. i. Locate the script and open it. Can you identify the different parts? == 3. Configure your routers == Once our sliver is ready we will go ahead and configure our click routers. In this example we have 4 routers, so instead of logging into each one of them and configuring it, we are going to use remote execution and configure them from our VM. === 3a. Login and remote execution === Run the `readyToLogin.py` script to get information about logging in to nodes. The script has a lot of output so lets put that in a file so that we can easily search for the information we want. {{{ readyToLogin.py -a pg-utah > login.out 2>&1 }}} You'll get a big chunk of information, but you're interested in the '''ssh configuration info''' information near the end. {{{ ... ... ================================================================================ SSH CONFIGURATION INFO for User inki ================================================================================ Host left Port 30778 HostName pc403.emulab.net User inki IdentityFile /home/geni/.ssh/geni_key Host hostB Port 30779 HostName pc490.emulab.net User inki IdentityFile /home/geni/.ssh/geni_key Host hostA Port 30778 HostName pc545.emulab.net User inki IdentityFile /home/geni/.ssh/geni_key Host bottom Port 30778 HostName pc490.emulab.net User inki IdentityFile /home/geni/.ssh/geni_key Host right Port 30778 HostName pc411.emulab.net User inki IdentityFile /home/geni/.ssh/geni_key Host top Port 30779 HostName pc545.emulab.net User inki IdentityFile /home/geni/.ssh/geni_key ...... }}} Copy all the above information and paste it into your `.ssh/config` file, then you can very easily login into your nodes, just by using the nickname (client_id) of the nodes. Your `~/.ssh/config` file should look like {{{ IdentityFile /home/geni/.ssh/geni_key Host left Port 30778 HostName pc403.emulab.net User inki IdentityFile /home/geni/.ssh/geni_key Host hostB Port 30779 HostName pc490.emulab.net User inki IdentityFile /home/geni/.ssh/geni_key Host hostA Port 30778 HostName pc545.emulab.net User inki IdentityFile /home/geni/.ssh/geni_key Host bottom Port 30778 HostName pc490.emulab.net User inki IdentityFile /home/geni/.ssh/geni_key Host right Port 30778 HostName pc411.emulab.net User inki IdentityFile /home/geni/.ssh/geni_key Host top Port 30779 HostName pc545.emulab.net User inki IdentityFile /home/geni/.ssh/geni_key }}} Let's login to our two hosts, the nicknames are `hostA` and `hostB` i. Open two new terminals i. In one terminal type {{{ ssh -A hostA }}} and in the other {{{ ssh -A hostB }}} ==== Test remote execution ==== You can execute commands in a remote host using `ssh`. To do this just follow your ssh command with the command you want to execute in quotes. i. In your local terminal type : {{{ ssh -A top "ls -a" }}} This will list all the files under the home directory on host `top`. The output should look like: {{{ geni@geni-VirtualBox:~$ ssh -A top "ls -a" . .. .bash_logout .bash_profile .bashrc .forward .kshrc .ssh .zshrc }}} If you get something similar you are all set for controlling your nodes from your computer. === 3b. Configure your routers === We are going to use remote execution to configure our routers. i. In a local terminal run the following command four time, each time substituting the with one of the top, bottom, left, right: {{{ geni@geni-VirtualBox:~$ ssh -A "/local/click-example/extractClickConfig.py " }}} You'll get output something like this: {{{ Your host information: hostA: hostA.StupidSliceName.emulab-net.emulab.net pc347.emulab.net top: top.StupidSliceName.emulab-net.emulab.net pc336.emulab.net left: left.StupidSliceName.emulab-net.emulab.net pc358.emulab.net right: right.StupidSliceName.emulab-net.emulab.net pc278.emulab.net bottom: bottom.StupidSliceName.emulab-net.emulab.net pc348.emulab.net hostB: hostB.StupidSliceName.emulab-net.emulab.net pc353.emulab.net Done. }}} (If you are prompted for a password, check to make sure that you provided the -A switch in your ssh command above.) i. The extractClickConfig script produces router configurations for your experiment. It also creates a diagram of your experiment. Get a copy locally from one of the routers, by typing in a local terminal: {{{ scp top:myslice.png }}} i. View the diagram by typing : {{{ eog myslice.png & }}} Your slice will look something like the one below (see [attachment:myslice.png]). The overall configuration should be the same, with two end hosts, named hostA and hostB, and four routers (top, left, right, bottom) in a diamond configuration. The host names, interface names, and MAC addresses will be different, depending on the actual resources assigned to your slice. [[Image(myslice.png, 25%)]] The four routers interconnected by solid lines are your "core network," which will run a non-standard, non-IP protocol. The dashed lines out to the end hosts carry standard IP traffic. == Turn off internet protocol == At this point, your network is still running IP. You can check by running a ping. In your '''hosta''' terminal window, run this command. {{{ ping -c 3 hostb }}} The command should succeed, with output like this: {{{ PING hostB-link-B (10.10.6.2) 56(84) bytes of data. 64 bytes from hostB-link-B (10.10.6.2): icmp_seq=1 ttl=61 time=1.38 ms 64 bytes from hostB-link-B (10.10.6.2): icmp_seq=2 ttl=61 time=1.19 ms 64 bytes from hostB-link-B (10.10.6.2): icmp_seq=3 ttl=61 time=1.53 ms --- hostB-link-B ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2004ms rtt min/avg/max/mdev = 1.193/1.370/1.531/0.138 ms }}} Since our experiment doesn't want IP, let's turn it off. ''On each of your four router hosts,'' run this command: {{{ sh ./stopIP.sh }}} You'll get output like this (the interface names may be different): {{{ Disabling IP on interface eth2 Disabling IP on interface eth4 }}} To verify that IP is really off, try another ping. On '''hosta''': {{{ ping -c 3 hostb }}} The command should take twelve seconds to time out, then fail with output like this: {{{ PING hostB-link-B (10.10.6.2) 56(84) bytes of data. --- hostB-link-B ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 11999ms }}} == Start your routers == The extractor script produces a click configuration file for each of your routers. ''On each of your four router hosts,'' run this command: {{{ sh ./startClick.sh }}} You'll get output like this. (Don't worry about the warning messages, Click is just reminding you that you have no IP addresses in your core network.) The shell prompt won't come back, but you'll see the debugging output of the Click router in the terminal window. {{{ Stopping any running Click routers Starting Click router top.click:34: While initializing ‘FromDevice@18 :: FromDevice’: warning: eth2: no IPv4 address assigned top.click:35: While initializing ‘FromDevice@21 :: FromDevice’: warning: eth4: no IPv4 address assigned }}} Congratulations! You are now running a non-IP core network on your four routers, along with a (primitive) non-IP multipath routing algorithm. You're ready to experiment with this configuration. == Send some traffic == Now you'll use your two edge hosts, '''hostA''' and '''hostB''' to send traffic along your network. Since these end hosts are not running your modified protocol, they'll rely on the '''top''' and '''bottom''' routers to transform their IP packets into your modified protocol on entry to the core network and back into IP packets on exit. In your terminal window on '''hostB''', instruct '''nc''' to listen for a UDP connection on port 24565 (or some other port that catches your fancy). {{{ [mberman@hostb ~]$ nc -ul 24565 }}} Then connect to it from your terminal window on '''hostA''': {{{ [mberman@hosta ~]$ nc -u hostb 24565 }}} You've established a simple text chat connection. Enter a line of text in either window, and it should appear in the other. Of course to do this, the text is travelling through your core network, using your non-standard protocol and routing. So type a message into each window, and make sure it appears in the other. That's it! Now, let's look inside to see what's going on. == Looking under the hood == Please note: the interface names and MAC addresses below are for the sample configuration shown in the figure above. You will want to refer to your network diagram to get the correct interfaces and addresses for your configuration. Let's take a look at what's happening in the four routers in your configuration. There are two basic router configurations. (You can find all of these files on any of your router hosts.) === Packet transformation === The more interesting configuration appears here, in the '''top.click''' configuration file. {{{ // This portion accepts IP packets, // reformats them, and routes them // to an internal router. route :: Classifier(27/01%01,-); modify :: Unstrip(2) -> StoreData(0, "AliceWasHere3546") -> route; FromDevice(eth3, PROMISC true) -> Classifier(12/0800) -> modify; route[0] -> left :: EtherEncap(0x7744, 00:04:23:b7:14:76, 00:04:23:b7:18:fa) -> SimpleQueue -> Print(outL) -> ToDevice(eth2); route[1] -> right :: EtherEncap(0x7744, 00:04:23:b7:1c:e0, 00:04:23:b7:19:2e) -> SimpleQueue -> Print(outR) -> ToDevice(eth4); // This portion accepts non-IP packets // with an ether type of 0x7744 // from an internal router, restores // them to IP format, and forwards. restore :: SimpleQueue -> Strip(30) -> EtherEncap(0x800, 00:04:23:b7:14:77, 00:04:23:b7:20:00) -> ToDevice(eth3); FromDevice(eth2) -> Classifier(12/7744) -> Print(inL) -> restore; FromDevice(eth4) -> Classifier(12/7744) -> Print(inR) -> restore; }}} As indicated in the comments, the top portion of the configuration listens ('''!FromDevice''') for IP packets arriving on the interface connected to '''hostA''' (that's '''eth3''' in this example). It then creates a new 16-byte field at the head of the packet (two bytes added by the '''Unstrip''' operation, plus the existing 14-byte Ethernet header. It fills that field with what could be important routing instructions, but in this case is just graffiti ('''!StoreData'''). The '''route''' operation then routes the packet via either the '''left''' or '''right''' router toward '''hostB'''. In either case, it wraps the packet in a fresh Ethernet header ('''!EtherEncap''') with a distinctive ether type code (0x7744), logs the new packet on its way out ('''Print''') and sends it out on the correct interface ('''!ToDevice'''). The bottom portion of the configuration is intended for packets coming out of the core network to '''hostA'''. It accepts packets from either the '''left''' or '''right''' router, logs them, strips off thirty bytes (Ethernet header plus your 16-byte new header field), puts on a fresh Ethernet header, and sends them along to '''hostA'''. The configuration for the '''bottom''' router is exactly symmetric, routing packets between '''hostB''' and the core network, but using different graffiti. === Simple Forwarding === The '''left''' router configuration is much simpler. Here's the '''left.click''' file: {{{ // Copy packets from top to bottom. FromDevice(eth2) -> StoreEtherAddress(00:04:23:b7:42:b6, dst) -> StoreEtherAddress(00:04:23:b7:18:fb, src) -> SimpleQueue -> Print(top) -> ToDevice(eth3); // Copy packets from bottom to top. FromDevice(eth3) -> StoreEtherAddress(00:04:23:b7:14:76, dst) -> StoreEtherAddress(00:04:23:b7:18:fa, src) -> SimpleQueue -> Print(bottom) -> ToDevice(eth2); }}} This configuration just blindly forwards packets. It picks up any packet from the '''top''' router, updates the Ethernet header, and passes it along to the '''bottom''' router. The same applies in the reverse direction. Again, the configuration for the '''right''' router is exactly analogous. === Monitoring your core network === Let's watch how the packets travel through the network. The terminal windows on your four router hosts show the log output from the '''Print''' statements shown above, one line per packet. Type a few times in each router window to create a space below the existing log entries, so you can identify new log messages as they appear. Now go to your window for hostA, where your '''nc''' command is still running. Type a message into this window. You should see a log message in three of your four router windows. In this example, you might see: In the '''top''' router log: {{{ outR: 76 | 000423b7 192e0004 23b71ce0 7744416c 69636557 61734865 }}} This log entry says that the '''top''' router received a packet from '''hostA''', modified it, and sent it out to the '''right''' router. If the entry started with '''outL''', that would indicate that it sent the packet out to the '''left''' router. Let's look a bit at the start of the packet (the first 24 bytes are logged). It starts with an Ethernet header. The first six bytes are the MAC address of the destination interface, that's 00:04:23:B7:19:2E, the MAC address of '''eth4''' on '''right'''. The next six bytes are the MAC address of the source interface, 00:04:23:B7:1C:E0, or '''eth4''' on '''top'''. Next comes your ether type, 0x7744. The remaining bytes, "416c 69636557 61734865" are the start of the first field in your new protocol, "!AliceWasHe" in ASCII. Since this packet was routed to the right, there's an entry in the '''right''' router log. This entry indicates that a packet was received from '''top'''. The logged contents show the packet sent to '''bottom''', with rewritten MAC addresses, corresponding to '''eth5''' on '''bottom''' and '''eth5''' on '''right'''. {{{ top: 76 | 000423b7 42590004 23b7192f 7744416c 69636557 61734865 }}} Finally, here's the entry on '''bottom''': {{{ inR: 76 | 000423b7 42590004 23b7192f 7744416c 69636557 61734865 }}} It shows the same packet received from '''right'''. After logging, the packet is rewritten into IP and sent to '''hostB'''. Try typing a few different lines to hostA. You should see some packets routed to the left and some to the right. The routing decision is based on the '''route :: Classifier(27/01%01,-);''' entry in the '''top''' router configuration. Here, the router is looking at the low-order bit of the checksum on the initial IP packet (now at byte position 27 with the addition of the new sixteen byte field at the start of the header). Packets with odd checksums go to the left; those with even checksums go right. Try typing a few lines to hostB. You should see similar behavior, but starting from '''bottom''' and working up. You'll also see the different value inserted in the new header field. == Clean up == When you're done, please release your resources so they'll be available to others. {{{ omni.py deletesliver -a http://www.emulab.net/protogeni/xmlrpc/am StupidSliceName }}} == Moving forward with your experiment == This sample experiment is just a very simple demonstration of how to run a Click-based routing configuration using ProtoGENI. For a more meaningful experiment, you may want to try some of the variations described below. We'd love to hear what you're doing with Click and GENI, and we're here to help. Please let us know at [mailto:help@geni.net]. === Improved routing === Instead of writing "!AliceWasHere" in your packets, perhaps include some real routing instructions. Modify the Click configurations to route packets accordingly. === Richer topology === Incorporate additional hosts into a core network topology that's more interesting than a simple diamond. === Improved performance with kernel-level Click === The Click router supports kernel-level operation. The principles are the same, but the setup is a bit more involved. To use kernel-level Click, you will probably want first to develop and debug your setup at the user level. Drop us a line at [mailto:help@geni.net] for help setting up Click in kernel mode. === Wide-area routing === Modify your rspec to include some ProtoGENI hosts on the Internet2 backbone or at multiple ProtoGENI sites. Additional information available at http://protogeni.net or [mailto:help@geni.net].