[[PageOutline]] = FOAM = FOAM is an OpenFlow aggregate manager, which sites in GENI use to allow experimenters to allocate !OpenFlow resources. https://openflow.stanford.edu/display/FOAM has more information about FOAM (from the official site at Stanford), including [https://openflow.stanford.edu/display/FOAM/0.10.x+Install+Guide installation instructions], [https://openflow.stanford.edu/display/FOAM/FAQ a FAQ] with common error messages (for both experimenters and admins), etc. = Info for experimenters = The following sections are mostly of interest to GENI experimenters. == Rspecs == FOAM uses GENI v3 rspecs, with the !OpenFlow v3 extensions. We have a page about [wiki:HowTo/WriteOFv3Rspecs how to write OF v3 rspecs], including some examples, information about differences from previous versions, etc. If you need a hand, just drop a note to help@geni.net. == Getting your sliver approved == Many FOAM aggregates have enabled automatic approval of slivers whose flowspace doesn't overlap with any existing slivers. If your flowspace does overlap with someone else's, or if auto-approval isn't enabled, your reservation request may be held for approval, in which case a local FOAM admin at that site will need to approve your request before your sliver actually becomes live. You should get e-mail from FOAM when your sliver is created, and another message when it's been approved; if you don't hear back, you may be able to reach a FOAM admin by replying to that message (or to the contact information in the body of the message). FOAM can read your e-mail address from your slice credential, so if that address is valid, then you don't need to include an e-mail address in your rspec. If it isn't, make sure that you also provide a valid email address in your rspec so that you can get the notifications about status changes of your !OpenFlow sliver. If you're setting up a multi-campus topology, note that your sliver will need to be approved separately at each FOAM aggregate. = Info for admins = The following sections are mostly of interest to FOAM admins. == System requirements == FOAM is fairly lightweight, and can run on the same system as the !FlowVisor that it uses, without additional hardware requirements. You can also run it on a separate system for isolation purposes, e.g. so that if you need to reboot the FOAM server, that doesn't affect !FlowVisor. == Version == The GPO currently recommends version 0.10.5 (the latest stable release) for GENI sites. Note that there were significant changes between FOAM 0.6 and FOAM 0.8; http://groups.geni.net/geni/wiki/OpenFlow/FOAM?version=22 has the last version of this page with 0.6-specific information. == Slice Authority trust configuration == FOAM has a set of CA certificates that it uses for user authorization: If a user has a certificate signed by a CA that FOAM is configured to trust, then that user can talk to FOAM (and create FOAM slivers, which will be approved automatically if auto-approval is on, etc). To configure what Slice Authorities FOAM trusts, install or remove the CA cert for the Slice Authority in a file in `/opt/foam/etc/gcf-ca-certs`, and then rebuild the nginx CA cert bundle and restart FOAM and nginx: {{{ sudo foamctl admin:bundle-certs sudo service foam restart sudo service nginx restart }}} The GPO recommends that GENI FOAM aggregates trust the authorities in the GENI cert bundle; see below for detailed instructins on how to do that. == Initial configuration == The GPO has a variety of recommendations for configuring FOAM. We strongly encourage GENI sites to do all of these things, and encourage you to contact us (`gpo-infra@geni.net`) if you have any questions or concerns. === Administrative e-mail === When you install FOAM for the first time (i.e. not an upgrade to an existing installation), you should configure its ability to send you administrative e-mail; those steps are part of the official [https://openflow.stanford.edu/display/FOAM/0.10.x+Install+Guide FOAM 0.10 install guide]. === Slice Authorities === The GPO recommends that GENI FOAM aggregates trust the authorities in the GENI cert bundle. To do that, first download the bundle and check its MD5 checksum: {{{ wget -O geni-cert-bundle.tar.gz http://groups.geni.net/geni/attachment/wiki/GeniTrustAnchors/geni-cert-bundle.tar.gz?format=raw md5sum geni-cert-bundle.tar.gz }}} See http://groups.geni.net/geni/wiki/GeniTrustAnchors for the expected value of the checksum. If the checksum doesn't match, contact `gpo-infra@geni.net` and we'll take a look. If the checksum on the tar.gz file matches, unpack the bundle and check the sums on the files in it: {{{ tar xfz geni-cert-bundle.tar.gz cd geni-cert-bundle md5sum --check MD5SUMS }}} That should produce output like: {{{ ch.geni.net-ca.pem: OK ch.geni.net-ma.pem: OK emulab.net.pem: OK pgeni.gpolab.bbn.com.pem: OK plc.pem: OK }}} If any of them don't say "OK", or you get any other errors, contact `gpo-infra@geni.net`. If those do all check out ok, install the certs in the directory that FOAM uses, rebuild the cert file that nginx uses, and restart FOAM and nginx: {{{ sudo cp *.pem /opt/foam/etc/gcf-ca-certs sudo foamctl admin:bundle-certs sudo service foam restart sudo service nginx restart }}} Experimenters should then be able to use FOAM with credentials signed by any of those authorities; you can test this yourself if you have such credentials handy. === Admin password === We typically put the FOAM admin password into /etc/foam.passwd, so you don't have to type it every time you run a foamctl command. Make sure that it's only readable by people who should have full admin access to FOAM on your system! (World-readable might be fine if only FOAM admins have accounts on your FOAM server; or you might want to make it only group-readable by a group that includes the FOAM admins.) The examples on the rest of this page assume that you've done that. === !FlowVisor configuration === You'll then need to configure FOAM to point to your !FlowVisor, which you can do interactively with {{{foamctl config:set-flowvisor-info}}}, or non-interactively like this: {{{ foamctl config:set-value --key="flowvisor.hostname" --value="localhost" --passwd-file=/etc/foam.passwd foamctl config:set-value --key="flowvisor.passwd" --value="$(cat /etc/flowvisor.passwd)" --passwd-file=/etc/foam.passwd }}} This example works if your !FlowVisor is on the same host as FOAM; if it's not, use values that work for your installation. === Site tag === In order to work smoothly with GENI monitoring, you'll also need to set your site tag to a name that resolves, in DNS, to the system where your FOAM server runs (i.e. the hostname part of the GENI AM API URL). It defaults to the system's FQDN, and that's fine if you don't want to change it. You can also change it to something more abstract, like a CNAME such as "foam.gpolab.bbn.com", which is what we do. To change it, use a command like this, replacing the "value=" part with a good value for your site: {{{ foamctl config:set-value --key="geni.site-tag" --value="foam.gpolab.bbn.com" --passwd-file=/etc/foam.passwd }}} NOTE that changing later this will require experimenters to change their rspecs for your FOAM server, so (a) it's good to get this right the first time; (b) discuss it with the GPO before you change it on a server that experimenters are using. === Monitoring === You should then install and configure the FOAM monitoring package; we have a separate [wiki:TangoGeniMonitoring/FoamConfiguration guide] for that. === Other settings === For GENI purposes, we recommend also configuring your admin e-mail (which causes it to show up in sliverstatus), and max lease: * The admin e-mail setting should generally be the same address you used when you set up e-mail. We use foam-admin@gpolab.bbn.com (a list with a couple of admins on it) at BBN. * The max lease setting controls how the maximum value of the sliver expiration date (e.g. when an experimenter does a renewsliver). We use 9000 hours (about a year) at BBN, and recommend that; making this short just means that experimenters who want a long-lived sliver have to do a renewsliver more frequently. To set those with foamctl (replace the "value" part with the values you want to use): {{{ foamctl config:set-value --key="site.admin.email" --value="foam-admin@gpolab.bbn.com" --passwd-file=/etc/foam.passwd foamctl config:set-value --key="geni.max-lease" --value=9000 --passwd-file=/etc/foam.passwd }}} === Version holding === Finally, we also recommend telling dpkg to hold FOAM and !FlowVisor at their current versions, so that if you do a general upgrade of all of the packages on your system, they won't get updated along with that, but will only update when you explicitly say so. To do that on Ubunutu: {{{ /bin/echo "foam hold" | sudo /usr/bin/dpkg --set-selections /bin/echo "flowvisor hold" | sudo /usr/bin/dpkg --set-selections }}} If you run !FlowVisor on a different server, you'd want to run the second command there instead. == Upgrading == Here are some before-and-after tests you can do when you upgrade FOAM, to confirm that your state hasn't changed unexpectedly. On the FOAM server, get a "before" list of slivers, active and deleted: {{{ version= cd rm -rf ~/tmp/foam-upgrade/$version mkdir -p ~/tmp/foam-upgrade/$version/before cd ~/tmp/foam-upgrade/$version/before foamctl geni:list-slivers --passwd-file=/etc/foam.passwd > list-slivers-active.txt until [ "$(grep -v "HTTP Error 504: Gateway Time-out" list-slivers-deleted.txt)" != "" ] ; do foamctl geni:list-slivers --deleted --passwd-file=/etc/foam.passwd >| list-slivers-deleted.txt ; done egrep '(email|desc|slice_urn|sliver_urn)' list-slivers-active.txt | sort > list-slivers-active-cooked.txt egrep '(email|desc|slice_urn|sliver_urn)' list-slivers-deleted.txt | sort > list-slivers-deleted-cooked.txt }}} If you own any slivers on the server, you can also do a listresources and sliverstatus for a sliver or two, and save that output for later comparison. Then do the upgrade, as usual. For simple upgrades: {{{ sudo apt-get update sudo service foam stop sudo apt-get install -y --force-yes foam sudo service foam start }}} More complex upgrades might have additional steps, which will generally be described on a separate page, in the announcement of the upgrade, and/or in the release notes. Then, on the FOAM server, get an "after" list of slivers, active and deleted: {{{ cd rm -rf ~/tmp/foam-upgrade/$version/after mkdir -p ~/tmp/foam-upgrade/$version/after cd ~/tmp/foam-upgrade/$version/after foamctl geni:list-slivers --passwd-file=/etc/foam.passwd > list-slivers-active.txt until [ "$(grep -v "HTTP Error 504: Gateway Time-out" list-slivers-deleted.txt)" != "" ] ; do foamctl geni:list-slivers --deleted --passwd-file=/etc/foam.passwd >| list-slivers-deleted.txt ; done egrep '(email|desc|slice_urn|sliver_urn)' list-slivers-active.txt | sort > list-slivers-active-cooked.txt egrep '(email|desc|slice_urn|sliver_urn)' list-slivers-deleted.txt | sort > list-slivers-deleted-cooked.txt }}} Diff the results: {{{ cd ../before for file in *cooked.txt ; do diff -u $file ../after/$file ; done }}} You should typically expect no differences. If you own any slivers on the server, you can also do a listresources and sliverstatus for a sliver or two, compare that output to the output from before the upgrade. == Advisories == Here are some things that FOAM admins should be aware of and watch out for. === Rspecs that include match rules with no protocol specified === Rspecs that include match rules at one layer typically need to include match rules specifying the protocol at a lower layer. For example: * An rspec that specifies a layer 3 match, like nw_src=A.B.C.D/N, also needs to specify dl_type to indicate what the layer 3 protocol is (e.g. dl_type=0x800 for IP and/or 0x806 for ARP). * An rspec that specifies a layer 4 match, like tp_port=N, also needs to specify nw_proto to indicate what the layer 4 protocol is (e.g. nw_proto=1 for ICMP, or 6 for TCP, or 17 for UDP). The important thing is that if the protocol ''isn't'' specified, some switches will end up with flowtable entries that match more traffic than they should, interfering with other experiments at your site. FOAM admins should be careful not to approve rspecs that include a match at one layer without specifying the protocol at a lower layer. We don't currently think that there's a valid use case for an experimenter wanting to specify a match at one layer without specifying the protocol at a lower layer, but will amend this advisory if we encounter one. == Auto-approval == FOAM includes a simple analysis engine which can be used to inform auto-approval decisions. The analysis is done by default, but can be disabled if you really want to for some reason. === Auto-approval mode === The action taken on a newly created sliver depends on the geni.approval.approve-on-creation configuration option. It has three possible values: * Mode 0, "never": Newly created slivers are never automatically approved. * Mode 1, "always": Newly created slivers are always automatically approved. * Mode 2, "analysis": Newly created slivers are automatically approved if they pass all the checks in the analysis engine. The default value is "0" on new FOAM installs, so no slivers will be auto-approved. The analysis engine runs regardless of the geni.approval.approve-on-creation setting, with the results visible in the 'foam_pend_reason' field (visible in 'foamctl geni:show-sliver' and in the GENI AM API !SliverStatus call). If the sliver passed the checks in the analysis engine, this field will be "null"; otherwise, it will contain more information about what test it failed. (More precisely, it'll include one test that the sliver didn't pass; if it failed multiple tests, only one will be listed.) You can use foamctl to check your current auto-approval mode: {{{ foamctl config:get-value --key="geni.approval.approve-on-creation" --passwd-file=/etc/foam.passwd }}} And to set it: {{{ foamctl config:set-value --key="geni.approval.approve-on-creation" --value="2" --passwd-file=/etc/foam.passwd }}} === Turning on auto-approval === We recommend that GENI sites turn on analysis-based auto-approval. If your site's topology doesn't contain any loops, and doesn't connect to other sites in a way that could form a loop, this is generally safe to do. The FOAM aggregates in the GENI racks have analysis-based auto-approval turned on by default. If your topology does contain a loop, or if you connect to other sites in a way that could form a loop (e.g. you connect to two sites who also connect to each other), you can use FOAM's "port groups" feature to allow auto-approval of safe topolgies only. This can be somewhat complicated; talk to the GPO (gpo-infra@geni.net) about the details of your site, and we'll help you set it up. (Loops are dangerous because they enable naive experimenters to generate huge amounts of traffic easily (and accidentally), and current !OpenFlow implementations on switches generally don't isolate different experimenters' traffic from each other very well, meaning that one accident of this sort can fairly easily take down entire networks for many people.) === Analysis and VLANs === In principle, traffic in GENI is sliced by VLAN, such that each slice has one or more private VLANs that carry traffic only for that slice. In practice, the current GENI mesoscale OpenFlow network consists mainly of VLAN-hybrid switches, and thus uses a single shared VLAN (colloquially "1750", the VLAN ID number used by many (but not all) sites), where traffic is then sliced by other features, such as MAC address, ethertype, or IP subnet. The [wiki:NetworkCore GENI network core page] has more details about that. (FIXME: No it doesn't. But it really should. Or maybe there's some other page that does, or should.) Prior to FOAM 0.12, the analysis engine only supports VLAN-hybrid datapaths, and could not auto-approve any sliver that included a VLAN-based match (the dl_vlan match attribute). As of FOAM 0.12, the analysis engine also supports dl_vlan matches, and differentiates between "exclusive" VLANs, "reserved" VLANs, and "shared" VLANs. The shared mesoscale VLAN (and other VLAN-hybrid datapaths) is effectively a special case of a shared VLAN, one in which no dl_vlan attribute is specified ("dl_vlan=none" is one way to think about it). By default, FOAM treates each VLAN as exclusive, unless a FOAM admin has added the VLAN to the list of reserved or shared VLANs. (A VLAN can be on both lists, although there's not really any reason to do that; but if you want to change a VLAN from one to the other, for example, you need to both remove it from the old list and add it to the new one.) === Analysis checks === The analysis engine performs the following checks: 1. If any port groups are defined, the request must include at most one of the ports in each port group. 2. The request must not include any VLANs on the reserved list. 3. If the request includes any exclusive VLANs: Those VLANs must not overlap with the VLANs of any already-approved slivers. 4. If the request (a) includes any shared VLANs; and/or (b) includes any matches with no dl_vlan attribute; then the sliver's other matches must pass at least one of the following additional tests, and not fail any of them: a. If the request includes matches based on IP space: That IP space must not overlap with the IP space matches of any already-approved slivers. b. If the request includes matches based on MAC addresses: Those MAC addresses must not overlap with the MAC address matches of any already-approved slivers. c. If the request includes matches based on ethertypes (other than IP and ARP): Those ethertypes must not overlap with the ethertype matches of any already-approved slivers. For each of those tests, the result could be "pass", "fail", or "n/a" (e.g. if the request doesn't include IP space). In that last set of tests, overlap is only considered within the context of a VLAN (or the "none" VLAN). For example, if you have three shared VLANs, and an already-approved sliver matches 172.16.0.0/16 in one shared VLAN, another sliver which matches 172.16.0.0/16 in a different shared VLAN will not be considered to overlap with the first sliver. For slivers with only exclusive VLANs, if the request includes any '''other''' match criteria, those criteria will appear in the resulting !FlowVisor flowspace rules; but they're entirely irrelevant to auto-approval for exclusive VLANs, which only considers whether the VLAN overlaps with an already-approved sliver. Experimenters are encouraged not to include any other match criteria in slivers with exclusive VLANs. == Sliver approval workflow == This section describes our workflow at BBN for approving slivers which aren't approved automatically. FOAM sends e-mail about new slivers to the FOAM admin e-mail address that you configured when you set up FOAM. If further communication about a sliver request is needed, we copy that address on the e-mail, so that everyone will see it. We also send mail to to that address when we approve or reject the sliver (or if we review the request and we're not sure whether to approve or reject it), so everyone knows who did it. If a sliver is automatically approved, FOAM will also send mail when that happens. If the automatic approval fails (e.g. because !FlowVisor is down, because the new sliver uses the same controller as an existing sliver, etc), the admin *won't* get mail; https://openflow.stanford.edu/bugs/browse/FOAM-288 is tracking that bug in FOAM. If a sliver isn't automatically approved, it'll show up in the pending queue, and have pend_reason set to some text explaining why the sliver wasn't automatically approved. There's also a nightly cron job that sends a reminder message to the FOAM admin address about the slivers in the pending queue; you should generally either approve them, reject them, or contact the experimenter if you need more information before making a decision. Some reasons to reject a sliver that wasn't automatically approved: * It includes multiple cross-connects. * It includes an I2-connected MyPLC plnode (ganel, gardil, sardis) and an NLR cross-connect, or an NLR plnode (bain, navis) and an I2 cross-connect. * It overlaps with something, and the existing sliver is more legitimate than the new one (e.g. the new sliver is trying to use someone else's subnet). We generally follow up to the rejection e-mail message from FOAM, copying the experimenter, to explain why we rejected the sliver. Some reasons to approve a sliver that wasn't automatically approved: * It includes a VLAN tag, but is otherwise fine to approve. * It includes an entire DPID and no ports, but is otherwise fine to approve. * It's something that we don't want to automatically approve for everyone, but are making a specific exception for in the case of this sliver/experimenter. Since this is new, this list is still a little vague, and deciding whether to approve a non-auto-approved sliver is something of a judgment call, since in theory any safe sliver should be auto-approved, and thus any non-auto-approved sliver has a pretty high chance of being unsafe in some way. If we find a type of sliver that isn't auto-approved, but that is always safe to approve manually, we'll add more details here. Regardless, we always send e-mail to the admin address saying what we did, so everyone's in the loop. (One convenient way is to reply to the notification message about the sliver.) == Managing FOAM slivers == https://openflow.stanford.edu/display/FOAM/foamctl+Guide is the official guide to foamctl, and describes in detail everything that it can do. (NOTE: As of 2012-07-17, this guide was FOAM 0.6 specific, and marked as a "legacy document". [https://openflow.stanford.edu/bugs/browse/FOAM-160 FOAM-160] is tracking the task of updating it.) Here are some specific commands that we've found useful for performing common tasks. These commands all assume that you're running them on the FOAM server, and that you have a file /etc/foam.passwd, containing the FOAM admin password. === Get a list of slivers === Pending ones (this is essentially "the queue" of slivers that are awaiting admin action): {{{ foamctl geni:list-slivers -s Pending --passwd-file=/etc/foam.passwd }}} All active ones: {{{ foamctl geni:list-slivers --passwd-file=/etc/foam.passwd }}} Either of these will give you a sliver URN; if you do {{{ sliver_urn=urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbsstghosts:678fc69b-76e1-4a50-9fb2-ab5c4a5298d6 }}} (with the actual URN of course), the rest of these commands will then work as-is -- or you can find a sliver from a slice name, see below. Deleted ones: {{{ foamctl geni:list-slivers --deleted --passwd-file=/etc/foam.passwd }}} === Find a sliver from a slice name === If you know a user's slice name, you can grep for it: {{{ foamctl geni:list-slivers --passwd-file=/etc/foam.passwd | egrep sliver_urn.+exampleslice }}} You can use this to get a sliver URN and/or an FV slice name from a GENI slice name, assigned to $sliver_urn and $flowvisor_slice: {{{ slicename=exampleslice ; sliver_urn=$(foamctl geni:list-slivers --passwd-file=/etc/foam.passwd | egrep sliver_urn.+$slicename | sed -e 's/ *"sliver_urn": "\(.*\)".*/\1/') ; flowvisor_slice=$(echo $sliver_urn | awk -F : '{print $NF}') }}} Replace "exampleslice" with a string to egrep for. Choose something that's unique to the slice; note that if the string contains egrep regexp type characters, like * or +, you'll need to backslash those. The rest of these commands assume that you've used that (or something similar) to set $sliver_urn. === Show a sliver's basic info === {{{ foamctl geni:show-sliver -u $sliver_urn --passwd-file=/etc/foam.passwd }}} === Show a sliver's rspec === {{{ foamctl geni:show-sliver -r -u $sliver_urn --passwd-file=/etc/foam.passwd }}} === Show a sliver's flowspec === {{{ foamctl geni:show-sliver -s -u $sliver_urn --passwd-file=/etc/foam.passwd }}} === Show a sliver's flowspace === {{{ foamctl geni:show-sliver -f -u $sliver_urn --passwd-file=/etc/foam.passwd }}} === Approve a sliver === This marks a sliver in FOAM as Approved, and adds its FV slice and flowspace rules for it to the !FlowVisor (if it wasn't already). {{{ foamctl geni:approve-sliver -u $sliver_urn --passwd-file=/etc/foam.passwd }}} === Disable a sliver === This marks a sliver in FOAM as Pending, and removes its FV slice and flowspace rules for it from the !FlowVisor (if there were any). {{{ foamctl geni:disable-sliver -u $sliver_urn --passwd-file=/etc/foam.passwd }}} === Reject a sliver === This marks a sliver in FOAM as Rejected, and removes its FV slice and flowspace rules for it from the !FlowVisor (if there were any). {{{ foamctl geni:reject-sliver -u $sliver_urn --passwd-file=/etc/foam.passwd }}} === Delete a sliver === This disables a sliver, and marks it as deleted, just like the GENI AM API !DeleteSliver call, and removes its FV slice and flowspace rules for it from the !FlowVisor (if there were any). {{{ foamctl geni:delete-sliver -u $sliver_urn --passwd-file=/etc/foam.passwd }}} You should generally only do this with the experimenter's permission, and if the experimenter can't delete their own sliver for some reason, so they're not confused about where their sliver went. (If you disable or reject it, they can still see it; if you delete it, it's essentially gone forever from their point of view.) === Compare two slivers === This compares two slivers (called "old" and "new" here, since a common use case is to compare a new request from an experimenter to an existing old request). {{{ rm -f old.txt new.txt oldsliver= newsliver= foamctl geni:show-sliver -r -u $oldsliver --passwd-file=/etc/foam.passwd > old.txt foamctl geni:show-sliver -r -u $newsliver --passwd-file=/etc/foam.passwd > new.txt diff -u old.txt new.txt }}} In the case of someone who deleted and then re-created a sliver, you could get the sliver URNs from the e-mail from FOAM, for example. You can also use 'geni:show-sliver -s' to show flowspaces instead, which have the advantage of not including any comments or other formatting in the experimenter's rspec, but are a less familiar format for most admins. == Changing the !FlowVisor password == FOAM's database stores information about the !FlowVisor it manages, including its hostname and password. If you want to change that !FlowVisor's fvadmin password, you need to change it both in FV itself, and also in FOAM. First, change the password in Flowvisor: {{{ fvctl --passwd-file=/etc/flowvisor.passwd changePasswd fvadmin (enter the new password) }}} Then update /etc/flowvisor.passwd to contain the new password. Finally, change the password in FOAM: {{{ foamctl config:set-value --key="flowvisor.passwd" --value="$(cat /etc/flowvisor.passwd)" --passwd-file=/etc/foam.passwd }}} == Testing FOAM == We have a separate page describing our procedure for [wiki:OpenFlow/FOAM/Testing testing FOAM], e.g. after an upgrade.