Changes between Initial Version and Version 1 of ClickExampleExperiment


Ignore:
Timestamp:
09/07/11 13:54:25 (10 years ago)
Author:
Mark Berman
Comment:

Draft version

Legend:

Unmodified
Added
Removed
Modified
  • ClickExampleExperiment

    v1 v1  
     1[[PageOutline]]
     2= UNDER DEVELOPMENT =
     3'''This page is under development. Don't try this at home (yet).'''
     4
     5= Example Experiment - Click Routers =
     6
     7In this example experiment, you will configure and run a non-IP software routing configuration, using the Click modular router and ProtoGENI hosts. In this example, we'll be running click in user mode. Once you have the prerequisites in place, you should be able to complete this example experiment in under an hour.
     8
     9Please note that you can't just cut and paste all of the commands. There are additional instructions in the text.
     10
     11== Prerequisites ==
     12
     13Before beginning this experiment, you should:
     14
     15 * Have a GENI credential recognized by the ProtoGENI confederation.
     16 * Configure omni (version 1.3.2 or later) on your machine. Get sufficiently comfortable with omni to verify that a '''listresources''' command works.
     17 * Install '''dot''' on your machine. (Optional but strongly recommended - available as part of the '''graphviz''' package. Install with "apt-get install graphviz", "yum install graphviz", or download from graphviz.org.)
     18
     19Information on obtaining GENI credentials and omni is available at [wiki:GENIExperimenter] or by contacting [mailto:help@geni.net].
     20
     21== Setup ==
     22
     23Create a new directory, click-example, on your machine. You will do the rest of your work from this directory.
     24
     25{{{
     26mkdir click-example
     27cd click-example
     28}}}
     29
     30Copy the following attached files into this directory:
     31
     32 * [attachment:click-example.rspec]
     33 * [attachment:extractClickConfig.py]
     34
     35(To copy the attached text files, first click on the attachment links above, then scroll to the bottom of the displayed text, where you will find the option to '''Download in other formats'''. Use the '''Original Format''' link and your browser's '''Save As''' option to save the plain text file in your '''click-example''' directory.)
     36
     37Set execute permission on extractClickConfig.py:
     38
     39{{{
     40chmod +x extractClickConfig.py
     41}}}
     42
     43Add your omni source directory to your PYTHONPATH:
     44
     45{{{
     46export PYTHONPATH=$PYTHONPATH:/path/to/omni/src
     47}}}
     48
     49(Optional) You'll be making heavy use of your private key to log into your ProtoGENI hosts in the steps below. If your key is encrypted, you may want to load it in a key manager so you don't have to type your passphrase many times. Alternately, you can remove the passphrase from your key using the steps below. (You don't need to do this step, however, you will be prompted for your passphrase quite a few times when running the extractor script below.)
     50
     51To remove passphrase (assuming your private key file is '''id_rsa'''):
     52{{{
     53mv id_rsa id_rsa.encrypted
     54openssl rsa -in id_rsa.encrypted -out id_rsa
     55<type your passphrase when prompted>
     56chmod 400 id_rsa
     57}}}
     58
     59== Obtain your resources ==
     60
     61Create your slice. Please don't use my stupid slice name.  This example will use the Utah ProtoGENI site. You can choose a different site by selecting a different aggregate manager with the -a switch.
     62
     63{{{
     64/path/to/omni/src/omni.py createslice -a http://www.emulab.net/protogeni/xmlrpc/am StupidSliceName
     65}}}
     66
     67Create a sliver and add resources. (You changed the slice name, right?)
     68
     69{{{
     70/path/to/omni/src/omni.py createsliver -a http://www.emulab.net/protogeni/xmlrpc/am StupidSliceName click-example.rspec
     71}}}
     72
     73Wait until your sliver is ready, typically a few minutes. (You can monitor your sliver with [http://protogeni.net Flack] or use omni's '''sliverstatus''' command as shown below.)
     74
     75{{{
     76/path/to/omni/src/omni.py sliverstatus -a http://www.emulab.net/protogeni/xmlrpc/am StupidSliceName
     77}}}
     78
     79Run the extractClickConfig.py script as shown below, using the same aggregate manager and slice name you just used to create your sliver. It will retrieve the manifest rspec for your slice. This is a blob of XML that describes the resources in your slice. If you're interested, feel free to look through the xml file that's left in your click-example directory. But you don't need to, because the extractor will pull out the relevant bits and organize them for you.
     80
     81{{{
     82./extractClickConfig.py -a http://www.emulab.net/protogeni/xmlrpc/am -n StupidSliceName
     83}}}
     84
     85You'll get output something like this.
     86
     87{{{
     88INFO:omni:Loading config file /home/mberman/.gcf/omni_config
     89INFO:omni:Using control framework utah-pg
     90INFO:omni:Saving output to a file.
     91INFO:omni:Gathering resources reserved for slice StupidSliceName.
     92INFO:omni:Listed resources on 1 out of 1 possible aggregates.
     93INFO:omni:Writing to 'StupidSliceName-rspec-www-emulab-net-protogeni.xml'
     94INFO:omni:Loading config file /home/mberman/.gcf/omni_config
     95INFO:omni:Using control framework utah-pg
     96Parsing manifest.
     97Writing ssh configuration to ssh_config.
     98Your host information:
     99        hostA: hostA.StupidSliceName.emulab-net.emulab.net (pc136.emulab.net)
     100        top: top.StupidSliceName.emulab-net.emulab.net (pc140.emulab.net)
     101        left: left.StupidSliceName.emulab-net.emulab.net (pc135.emulab.net)
     102        right: right.StupidSliceName.emulab-net.emulab.net (pc138.emulab.net)
     103        bottom: bottom.StupidSliceName.emulab-net.emulab.net (pc141.emulab.net)
     104        hostB: hostB.StupidSliceName.emulab-net.emulab.net (pc133.emulab.net)
     105Writing top router configuration to top.click.
     106Writing bottom router configuration to bottom.click.
     107Writing left router configuration to left.click.
     108Writing right router configuration to right.click.
     109Writing deployment script to deployClick.sh.
     110Run "sh deployClick.sh" to deploy router configurations.
     111Writing configuration graph to myslice.dot.
     112Run "dot myslice.dot -Tpng -o myslice.png" to generate diagram.
     113Done.
     114}}}
     115
     116''The extractor script makes some assumptions about your omni configuration, specifically that at least one ssh private key file listed for the ProtoGENI login user is available in the same location as the installed public key, but with the .pub suffix removed. This is the most common configuration, so unless you have done something unusual, you should be OK.''
     117
     118The extractor will generate eight output files in your click-example directory:
     119
     120 * Click configuration files: {top,left,right,bottom}.click
     121 * A ssh configuration file: ssh_config
     122 * A dot source diagram of your slice: myslice.dot
     123 * The manifest rspec for your slice: <slicename>-rspec-<aggregate>.xml
     124 * A deployment script for your click files: deployClick.sh
     125
     126(Optional) Run '''dot''' to make a diagram of your slice. There is lots of information on this diagram, and it will save you looking up host and interface names and numbers.
     127
     128{{{
     129dot myslice.dot -Tpng -o myslice.png
     130}}}
     131
     132Your slice will look something like the one below (see [attachment:sampleDiagram.png]). The overall configuration should be the same, with two end hosts, named hostA and hostB, and four routers (top, left, right, bottom) in a diamond configuration. The host names, interface names, and MAC addresses will be different, depending on the actual resources assigned to your slice.
     133
     134[[Image(sampleDiagram.png, nolink, 25%)]]
     135
     136The four routers interconnected by solid lines are your "core network," which runs a non-standard, non-IP protocol. The dashed lines out to the end hosts carry standard IP traffic.
     137
     138== Start your routers ==
     139
     140The extractor script produces a click configuration file for each of your routers. The deployment script, '''deployClick.sh''' copies these files to your routers and starts a user-space router on each. Run the script, and you should get output like this:
     141
     142{{{
     143[mberman@molasses click-example]$ sh deployClick.sh
     144Setting user shell on top.StupidSliceName.emulab-net.emulab.net
     145Changing shell for mberman.
     146Copying top.click to top.StupidSliceName.emulab-net.emulab.net
     147Waiting for click to build on top.StupidSliceName.emulab-net.emulab.net ...................
     148Stopping any running click router on top.StupidSliceName.emulab-net.emulab.net
     149Disabling IP routing on top.StupidSliceName.emulab-net.emulab.net
     150Starting click router on top.StupidSliceName.emulab-net.emulab.net
     151
     152<similar output for {left,right,bottom} routers>
     153}}}
     154
     155Depending on how long you waited between creating your slice and running the '''deployClick.sh''' script (and how powerful your allocated router machines are), you may need to wait a few minutes while click finishes compiling.
     156
     157You are now running a non-IP core network on your four routers, along with a (primitive) non-IP multipath routing algorithm. You're ready to experiment with this configuration.
     158
     159== Send some traffic ==
     160
     161Now you'll log into your two edge hosts, '''hostA''' and '''hostB''', and send some traffic between them.  Since these end hosts  are not running your modified protocol, they'll rely on the '''top''' and '''bottom''' routers to transform their IP packets into your modified protocol on entry to the core network and back into IP packets on exit.
     162
     163In two different windows, use '''ssh -F ssh_config hostA''' and '''ssh -F ssh_config hostB''' to connect to your two end hosts. (Don't forget to work in your '''click-example''' directory.)
     164
     165On '''hostB''', instruct '''nc''' to listen for a UDP connection on port 24565 (or some other port that catches your fancy).
     166{{{
     167[mberman@hostb ~]$ nc -ul 24565
     168}}}
     169
     170Then connect to it from '''hostA''':
     171{{{
     172[mberman@hosta ~]$ nc -u hostb 24565
     173}}}
     174
     175You've established a simple text chat connection. Enter a line of text in either window, and it should appear in the other. Of course to do this, the text is travelling through your core network, using your non-standard protocol and routing. So type a message into each window, and make sure it appears in the other.
     176
     177That's it!  Now, let's look inside to see what's going on.
     178
     179== Looking under the hood ==
     180
     181Please note: the interface names and MAC addresses below are for the sample configuration shown in the figure above. You will want to refer to your network diagram to get the correct interfaces and addresses for your configuration.
     182
     183Let's take a look at what's happening in the four routers in your configuration. There are two basic router configurations.
     184
     185=== Packet transformation ===
     186
     187The more interesting configuration appears here, in the '''top.click''' configuration file.
     188
     189{{{
     190// This portion accepts IP packets,
     191// reformats them, and routes them
     192// to an internal router.
     193route :: Classifier(27/01%01,-);
     194
     195modify :: Unstrip(2) ->
     196    StoreData(0, "AliceWasHere3546") ->
     197    route;
     198
     199FromDevice(eth3, PROMISC true) ->
     200    Classifier(12/0800) ->
     201    modify;
     202
     203route[0] -> left :: EtherEncap(0x7744, 00:02:b3:35:f1:b7, 00:02:b3:86:1d:13) ->
     204    SimpleQueue ->
     205    Print(outL) ->
     206    ToDevice(eth1);
     207
     208route[1] -> right :: EtherEncap(0x7744, 00:03:47:94:c7:fd, 00:02:b3:65:d1:2b) ->
     209    SimpleQueue ->
     210    Print(outR) ->
     211    ToDevice(eth2);
     212
     213// This portion accepts non-IP packets
     214// with an ether type of 0x7744
     215// from an internal router, restores
     216// them to IP format, and forwards.
     217restore :: SimpleQueue ->
     218    Strip(30) ->
     219    EtherEncap(0x800, 00:03:47:73:8e:bd, 00:02:b3:3f:7a:a1) ->
     220    ToDevice(eth3);
     221
     222FromDevice(eth1) -> Classifier(12/7744) -> Print(inL) -> restore;
     223FromDevice(eth2) ->  Classifier(12/7744) -> Print(inR) -> restore;
     224}}}
     225
     226As indicated in the comments, the top portion of the configuration listens ('''!FromDevice''') for IP packets arriving on the interface connected to '''hostA''' (that's '''eth3''' in this example). It then creates a new 16-byte field at the head of the packet (two bytes added by the '''Unstrip''' operation, plus the existing 14-byte Ethernet header. It fills that field with what could be important routing instructions, but in this case is just graffiti ('''!StoreData'''). The '''route''' operation then routes the packet via either the '''left''' or '''right''' router toward '''hostB'''. In either case, it wraps the packet in a fresh Ethernet header ('''!EtherEncap''') with a distinctive ether type code (0x7744), logs the new packet on its way out ('''Print''') and sends it out on the correct interface ('''!ToDevice''').
     227
     228The bottom portion of the configuration is intended for packets coming out of the core network to '''hostA'''.  It accepts packets from either the '''left''' or '''right''' router, logs them, strips off thirty bytes (Ethernet header plus your 16-byte new header field), puts on a fresh Ethernet header, and sends them along to '''hostA'''.
     229
     230The configuration for the '''bottom''' router is exactly symmetric, routing packets between '''hostB''' and the core network, but using different graffiti.
     231
     232=== Simple Forwarding ===
     233
     234The '''left''' router configuration is much simpler.  Here's the '''left.click''' file:
     235
     236{{{
     237FromDevice(eth1) ->
     238    StoreEtherAddress(00:02:b3:86:1a:4b, dst) ->
     239    StoreEtherAddress(00:03:47:95:7a:fe, src) ->
     240    SimpleQueue ->
     241    Print(top) ->
     242    ToDevice(eth2);
     243// Copy packets from bottom to top.
     244FromDevice(eth2) ->
     245    StoreEtherAddress(00:02:b3:35:f1:b7, dst) ->
     246    StoreEtherAddress(00:02:b3:86:1d:13, src) ->
     247    SimpleQueue ->
     248    Print(bottom) ->
     249    ToDevice(eth1);
     250}}}
     251
     252This configuration just blindly forwards packets. It picks up any packet from the '''top''' router, updates the Ethernet header, and passes it along to the '''bottom''' router. The same applies in the reverse direction.  Again, the configuration for the '''right''' router is exactly analogous.
     253
     254=== Monitoring your core network ===
     255
     256Let's watch how the packets travel through the network. In addition to the windows you already have open to the two end hosts, you'll need four more windows to watch the logs for all four routers.  In each window, you should use '''ssh''' to connect to one router host, '''ssh -F ssh_config top''', '''ssh -F ssh_config left''', '''ssh -F ssh_config right''' and '''ssh -F ssh_config bottom'''. You can monitor the click log files with '''tail''':
     257
     258{{{
     259[mberman@top ~]$ tail -f click.log
     260}}}
     261
     262The log files may have a couple of warnings at the top complaining that your interfaces have no IPv4 addresses. No worries, we did that on purpose, since we don't use IP in the core network. Other than those warnings, you should just see packet log messages, one line per packet. Type <enter> a few times in each router window to create a space below the existing log entries, so you can identify new log messages as they appear.
     263
     264Now go to your window for hostA, where your '''nc''' command is still running. Type a message into this window. You should see a log message in three of your four router windows. In this example, you might see:
     265
     266In the '''top''' router log:
     267{{{
     268outR:   76 | 0002b365 d12b0003 4794c7fd 7744416c 69636557 61734865
     269}}}
     270
     271This log entry says that the '''top''' router received a packet from '''hostA''', modified it, and sent it out to the '''right''' router. If the entry started with '''outL''', that would indicate that it sent the packet out to the '''left''' router. Let's look a bit at the start of the packet (the first 24 bytes are logged). It starts with an Ethernet header. The first six bytes are the MAC address of the destination interface, that's 00:02:b3:65:d1:2b, the MAC address of '''eth1''' on '''right'''. The next six bytes are the MAC address of the source interface, 00:03:47:94:c7:fd, or '''eth2''' on '''top'''. Next comes your ether type, 0x7744. The remaining bytes, "416c 69636557 61734865" are the start of the first field in your new protocol, "!AliceWasHe" in ASCII.
     272
     273Since this packet was routed to the right, there's an entry in the '''right''' router log. This entry indicates that a packet was received from '''top'''. The logged contents show the packet sent to '''bottom''', with rewritten MAC addresses, corresponding to '''eth2''' on '''bottom''' and '''eth2''' on '''right'''.
     274
     275{{{
     276top:   76 | 00034794 c7fc0003 4794c1f7 7744416c 69636557 61734865
     277}}}
     278
     279Finally, here's the entry on '''bottom''':
     280{{{
     281inR:   76 | 00034794 c7fc0003 4794c1f7 7744416c 69636557 61734865
     282}}}
     283
     284It shows the same packet received from '''right'''. After logging, the packet is rewritten into IP and sent to '''hostB'''.
     285
     286Try typing a few different lines to hostA. You should see some packets routed to the left and some to the right. The routing decision is based on the '''route :: Classifier(27/01%01,-);''' entry in the '''top''' router configuration. Here, the router is looking at the low-order bit of the checksum on the initial IP packet (now at byte position 27 with the addition of the new sixteen byte field at the start of the header). Packets with odd checksums go to the left; those with even checksums go right.
     287
     288Try typing a few lines to hostB. You should see similar behavior, but starting from '''bottom''' and working up. You'll also see the different value inserted in the new header field.
     289
     290== Clean up ==
     291
     292When you're done, please release your resources so they'll be available to others.
     293
     294{{{
     295/path/to/omni/src/omni.py deletesliver -a http://www.emulab.net/protogeni/xmlrpc/am StupidSliceName
     296}}}
     297
     298== Moving forward with your experiment ==
     299
     300This sample experiment is just a very simple demonstration of how to run a Click-based routing configuration using ProtoGENI. For a more meaningful experiment, you may want to try some of the variations described below. We'd love to hear what you're doing with Click and GENI, and we're here to help. Please let us know at [mailto:help@geni.net].
     301
     302=== Improved routing ===
     303
     304Instead of writing "!AliceWasHere" in your packets, perhaps include some real routing instructions. Modify the Click configurations to route packets accordingly.
     305
     306=== Richer topology ===
     307
     308Incorporate additional hosts into a core network topology that's more interesting than a simple diamond.
     309
     310=== Improved performance with kernel-level Click ===
     311
     312The Click router supports kernel-level operation. The principles are the same, but the setup is a bit more involved. To use kernel-level Click, you will probably want first to develop and debug your setup at the user level. Then, follow the steps above, with these differences:
     313
     314 * Before running '''createsliver''', modify your rspec to delete the '''execute''' tags on your routers. You may also want to add a '''hardware_type''' tag to specify a particular hardware configuration so your performance comparisons are valid and so the next step doesn't take too long.
     315
     316 * After running '''createsliver''', you need to rebuild the kernel on your routers and then build Click with its corresponding kernel module. Log into each router, and run '''sh /local/build-click-kernel.sh''' to patch and build your new kernel. This process will take a while, after which the machine will reboot, and you'll need to log back in. Now build Click with '''sh /local/build-click.sh'''.
     317
     318 * Run '''extractClickConfig.py''' as before (or create your Click configuration by hand), but you'll need to edit the '''deployClick.sh''' script to run Click at the kernel level. You do this by changing '''/usr/local/bin/click <file>''' to '''/usr/local/sbin/click-install <file>''' each time it occurs (four times in the example experiment above).
     319
     320=== Wide-area routing ===
     321
     322Modify your rspec to include some ProtoGENI hosts on the Internet2 backbone or at multiple ProtoGENI sites. Additional information available at http://protogeni.net or [mailto:help@geni.net].