Version 67 (modified by Josh Smift, 11 years ago) (diff)



FOAM is an OpenFlow aggregate manager, which sites in GENI use to allow experimenters to allocate OpenFlow resources. has more information about FOAM (from the official site at Stanford), including installation instructions, a FAQ with common error messages (for both experimenters and admins), etc.

Info for experimenters

The following sections are mostly of interest to GENI experimenters.


FOAM uses GENI v3 rspecs, with the OpenFlow v3 extensions. We have a page about how to write OF v3 rspecs, including some examples, information about differences from previous versions, etc. If you need a hand, just drop a note to

Getting your sliver approved

Most FOAM aggregates have enabled automatic approval of slivers whose flowspace doesn't overlap with any existing slivers. If your flowspace does overlap with someone else's, or if auto-approval isn't enabled, your reservation request may be held for approval, in which case a local FOAM admin at that site will need to approve your request before your sliver actually becomes live. You should get e-mail from FOAM when your sliver is created, and another message when it's been approved; if you don't hear back, you may be able to reach a FOAM admin by replying to that message (or to the contact information in the body of the message).

FOAM can read your e-mail address from your slice credential, so if that address is valid, then you don't need to include an e-mail address in your rspec. If the address in your slice credential isn't valid, make sure that you also provide a valid email address in your rspec so that you can get the notifications about status changes of your OpenFlow sliver. (This should generally not be a problem for GENI CH, Emulab, or PLC credentials.)

Info for admins

The following sections are mostly of interest to FOAM admins.

New FOAM admins

We have an intro page for new FOAM admins, with some introductory information about what FOAM is and how it works.

System requirements

FOAM is fairly lightweight, and can run on the same system as the FlowVisor that it uses, without additional hardware requirements. You can also run it on a separate system for isolation purposes, e.g. so that if you need to reboot the FOAM server, that doesn't affect FlowVisor.

We currently recommend running FOAM on Ubuntu 12.04, although it should also work on Debian and other versions of Ubuntu. NOTE, however, that FOAM expects sqlite to have write-ahead-logging, a feature that appears in sqlite 3.7 or later, and Ubuntu 10.04 in particular only has 3.6.22. FOAM will generally work ok with this, but if experimenters create many slivers at a time, the lack of write-ahead logging can lead to the database becoming locked, requiring a FOAM restart.


The GPO currently recommends version 0.12.3 (the latest stable release) for GENI sites.

In the very unlikely event that you're interested in versions of FOAM previous to 0.8, has the last version of this page with 0.6-specific information.

Slice Authority trust configuration

FOAM has a set of CA certificates that it uses for user authorization: If a user has a certificate signed by a CA that FOAM is configured to trust, then that user can talk to FOAM (and create FOAM slivers, which will be approved automatically if auto-approval is on, etc).

To configure what Slice Authorities FOAM trusts, install or remove the CA cert for the Slice Authority in a file in /opt/foam/etc/gcf-ca-certs, and then rebuild the nginx CA cert bundle and restart FOAM and nginx:

sudo foamctl admin:bundle-certs
sudo service foam restart
sudo service nginx restart

The GPO recommends that GENI FOAM aggregates trust the authorities in the GENI cert bundle; see below for detailed instructins on how to do that.

Initial configuration

The GPO has a variety of recommendations for configuring FOAM. We strongly encourage GENI sites to do all of these things, and encourage you to contact us ( if you have any questions or concerns.

Administrative e-mail

When you install FOAM for the first time (i.e. not an upgrade to an existing installation), you should configure its ability to send you administrative e-mail; those steps are part of the official FOAM 0.10 install guide.

Slice Authorities

The GPO recommends that GENI FOAM aggregates trust the authorities in the GENI cert bundle. To do that, first download the bundle and check its MD5 checksum:

wget -O geni-cert-bundle.tar.gz
md5sum geni-cert-bundle.tar.gz

See for the expected value of the checksum. If the checksum doesn't match, contact and we'll take a look.

If the checksum on the tar.gz file matches, unpack the bundle and check the sums on the files in it:

tar xfz geni-cert-bundle.tar.gz
cd geni-cert-bundle
md5sum --check MD5SUMS 

That should produce output like: OK OK OK OK
plc.pem: OK

If any of them don't say "OK", or you get any other errors, contact

If those do all check out ok, install the certs in the directory that FOAM uses, rebuild the cert file that nginx uses, and restart FOAM and nginx:

sudo cp *.pem /opt/foam/etc/gcf-ca-certs
sudo foamctl admin:bundle-certs
sudo service foam restart
sudo service nginx restart

Experimenters should then be able to use FOAM with credentials signed by any of those authorities; you can test this yourself if you have such credentials handy.

Admin password

We typically put the FOAM admin password into /etc/foam.passwd, so you don't have to type it every time you run a foamctl command. Make sure that it's only readable by people who should have full admin access to FOAM on your system! (World-readable might be fine if only FOAM admins have accounts on your FOAM server; or you might want to make it only group-readable by a group that includes the FOAM admins.) The examples on the rest of this page assume that you've done that.

FlowVisor configuration

You'll then need to configure FOAM to point to your FlowVisor, which you can do interactively with foamctl config:set-flowvisor-info, or non-interactively like this:

foamctl config:set-value --key="flowvisor.passwd" --value="$(cat /etc/flowvisor.passwd)" --passwd-file=/etc/foam.passwd
foamctl config:set-value --key="flowvisor.hostname" --value="localhost" --passwd-file=/etc/foam.passwd

This example works if your FlowVisor is on the same host as FOAM; if it's not, use values that work for your installation.

Site tag

In order to work smoothly with GENI monitoring, you'll also need to set your site tag to a name that resolves, in DNS, to the system where your FOAM server runs (i.e. the hostname part of the GENI AM API URL). It defaults to the system's FQDN, and that's fine if you don't want to change it. You can also change it to something more abstract, like a CNAME such as "", which is what we do. To change it, use a command like this, replacing the "value=" part with a good value for your site:

foamctl config:set-value --key="" --value="" --passwd-file=/etc/foam.passwd

NOTE that changing later this will require experimenters to change their rspecs for your FOAM server, and require anyone who connects to your network to update their attachment point information, AND require changes on the GMOC monitoring DB, so (a) it's really, really good to get this right the first time; (b) you should discuss it with the GPO before you change it on a server that anyone's using or connected to.


You should then install and configure the FOAM monitoring package; we have a separate guide for that.

Other settings

For GENI purposes, we recommend also configuring your admin e-mail (which causes it to show up in sliverstatus), and max lease:

  • The admin e-mail setting should generally be the same address you used when you set up e-mail. We use (a list with a couple of admins on it) at BBN.
  • The max lease setting controls how the maximum value of the sliver expiration date (e.g. when an experimenter does a renewsliver). We use 9000 hours (about a year) at BBN, and recommend that; making this short just means that experimenters who want a long-lived sliver have to do a renewsliver more frequently.

To set those with foamctl (replace the "value" part with the values you want to use):

foamctl config:set-value --key="" --value="" --passwd-file=/etc/foam.passwd
foamctl config:set-value --key="geni.max-lease" --value=9000 --passwd-file=/etc/foam.passwd

Version holding

We also recommend telling dpkg to hold FOAM and FlowVisor at their current versions, so that if you do a general upgrade of all of the packages on your system, they won't get updated along with that, but will only update when you explicitly say so. To do that on Ubunutu:

/bin/echo "foam hold" | sudo /usr/bin/dpkg --set-selections
/bin/echo "flowvisor hold" | sudo /usr/bin/dpkg --set-selections

If you run FlowVisor on a different server, you'd want to run the second command there instead.

Attachment information

Finally, we recommend configuring FOAM with information about what other devices its switches are connected to, which is really useful in helping experimenters figure out the topology of the GENI infrastructure. We have a separate page with more information about configuring attachment information.

NOTE that you'll want to update this if you add or remove connections!


We recommend backing up your FOAM databases, which includes both configuration information and FOAM sliver data.


This sequence of commands will save your various FOAM .db files as .sql files. It can be run while FOAM is running or stopped, as you like.

First, set $DUMPDIR to a directory where you (or the user you're running this as, e.g. if you're running it out of cron) have write permissions, and make sure the directory exists:

mkdir -p $DUMPDIR

Then, dump all of the .db files in /opt/foam/db into .sql files in your dump directory:

cd /opt/foam/db
for dbfile in $(ls *.db)
  dbname=$(basename $dbfile .db)
  sqlite3 $dbfile .dump > $DUMPDIR/$dbname.sql

Now you can back up those .sql files with whatever method you prefer for backing up normal text files.


NOTE that this completely and irrevocably overwrites your existing FOAM databases! Use with caution.

NOTE that this should only be done when FOAM is not running. First, do:

sudo service foam stop

You need to run the restore as the 'www-data' user, since that user has the right permissions to create the DB files.

Put the .sql files to restore into an otherwise-empty directory, change into that directory, and run:

for sqlfile in $(ls *.sql)
  dbname=$(basename $sqlfile .sql)
  sudo -u www-data rm -f /opt/foam/db/$dbname.db
  sudo -u www-data sqlite3 /opt/foam/db/$dbname.db ".read $dbname.sql"

Then start up FOAM:

sudo service foam start

You should then do some sanity checks to make sure things look like you expect.


Here are some before-and-after tests you can do when you upgrade FOAM, to confirm that your state hasn't changed unexpectedly.

On the FOAM server, get a "before" list of slivers, active and deleted:

version=<put the version number here>

rm -rf ~/tmp/foam-upgrade/$version
mkdir -p ~/tmp/foam-upgrade/$version/before
cd ~/tmp/foam-upgrade/$version/before

foamctl geni:list-slivers --passwd-file=/etc/foam.passwd > list-slivers-active.txt
until [ "$(grep -v "HTTP Error 504: Gateway Time-out" list-slivers-deleted.txt)" != "" ] ; do foamctl geni:list-slivers --deleted --passwd-file=/etc/foam.passwd >| list-slivers-deleted.txt ; done

egrep '(email|desc|slice_urn|sliver_urn)' list-slivers-active.txt | sort > list-slivers-active-cooked.txt
egrep '(email|desc|slice_urn|sliver_urn)' list-slivers-deleted.txt | sort > list-slivers-deleted-cooked.txt

The 'until ...' command should print "grep: list-slivers-deleted.txt: No such file or directory" once, and then take some time to dump out a list of deleted slivers. It might not take long at all, depending on how many deleted slivers you have on your FOAM server, but it can take a few minutes if you have a lot of them.

If you own any slivers on the server, you can also do a listresources and sliverstatus for a sliver or two at this point, and save that output for later comparison.

Then do the upgrade, as usual. For simple upgrades:

sudo apt-get update
sudo service foam stop
sudo apt-get install --ignore-hold --allow-unauthenticated foam
sudo service foam start

More complex upgrades might have additional steps, which will generally be described on a separate page, in the announcement of the upgrade, and/or in the release notes.

Then, on the FOAM server, get an "after" list of slivers, active and deleted:

rm -rf ~/tmp/foam-upgrade/$version/after
mkdir -p ~/tmp/foam-upgrade/$version/after
cd ~/tmp/foam-upgrade/$version/after

foamctl geni:list-slivers --passwd-file=/etc/foam.passwd > list-slivers-active.txt
until [ "$(grep -v "HTTP Error 504: Gateway Time-out" list-slivers-deleted.txt)" != "" ] ; do foamctl geni:list-slivers --deleted --passwd-file=/etc/foam.passwd >| list-slivers-deleted.txt ; done

egrep '(email|desc|slice_urn|sliver_urn)' list-slivers-active.txt | sort > list-slivers-active-cooked.txt
egrep '(email|desc|slice_urn|sliver_urn)' list-slivers-deleted.txt | sort > list-slivers-deleted-cooked.txt

Diff the results:

cd ../before
for file in *cooked.txt ; do diff -u $file ../after/$file ; done

You should typically expect no differences.

If you own any slivers on the server, you can also do a listresources and sliverstatus for a sliver or two, compare that output to the output from before the upgrade.


Here are some things that FOAM admins should be aware of and watch out for.

Rspecs that include match rules with no protocol specified

Rspecs that include match rules at one layer typically need to include match rules specifying the protocol at a lower layer. For example:

  • An rspec that specifies a layer 3 match, like nw_src=A.B.C.D/N, also needs to specify dl_type to indicate what the layer 3 protocol is (e.g. dl_type=0x800 for IP and/or 0x806 for ARP).
  • An rspec that specifies a layer 4 match, like tp_port=N, also needs to specify nw_proto to indicate what the layer 4 protocol is (e.g. nw_proto=1 for ICMP, or 6 for TCP, or 17 for UDP).

The important thing is that if the protocol isn't specified, some switches will end up with flowtable entries that match more traffic than they should, interfering with other experiments at your site. FOAM admins should be careful not to approve rspecs that include a match at one layer without specifying the protocol at a lower layer.

We don't currently think that there's a valid use case for an experimenter wanting to specify a match at one layer without specifying the protocol at a lower layer, but will amend this advisory if we encounter one.


FOAM includes a simple analysis engine which can be used to inform auto-approval decisions. The analysis is done by default (whether or not it's acted on), but can be disabled if you really want to for some reason.

Auto-approval mode

The action taken on a newly created sliver depends on the geni.approval.approve-on-creation configuration option. It has three possible values:

  • Mode 0, "never": Newly created slivers are never automatically approved.
  • Mode 1, "always": Newly created slivers are always automatically approved.
  • Mode 2, "analysis": Newly created slivers are automatically approved if they pass all the checks in the analysis engine.

The default value is "0" on new FOAM installs, so no slivers will be auto-approved.

The analysis engine runs regardless of the geni.approval.approve-on-creation setting, with the results visible in the 'foam_pend_reason' field (visible in 'foamctl geni:show-sliver' and in the GENI AM API SliverStatus call). If the sliver passed the checks in the analysis engine, this field will be "null"; otherwise, it will contain more information about what test it failed. (More precisely, it'll include one test that the sliver didn't pass; if it failed multiple tests, only one will be listed.)

You can use foamctl to check your current auto-approval mode:

foamctl config:get-value --key="geni.approval.approve-on-creation" --passwd-file=/etc/foam.passwd

And to set it:

foamctl config:set-value --key="geni.approval.approve-on-creation" --value="2" --passwd-file=/etc/foam.passwd

Turning on auto-approval

We recommend that GENI sites turn on analysis-based auto-approval. If your site's topology doesn't contain any loops, and doesn't connect to other sites in a way that could form a loop, this is generally safe to do. The FOAM aggregates in the GENI racks have analysis-based auto-approval turned on by default.

If your topology does contain a loop, or if you connect to other sites in a way that could form a loop (e.g. you connect to two sites who also connect to each other), you can use FOAM's "port groups" feature to allow auto-approval of safe topolgies only. This can be somewhat complicated; talk to the GPO ( about the details of your site, and we'll help you set it up.

(Loops are dangerous because they enable naive experimenters to generate huge amounts of traffic easily (and accidentally), and current OpenFlow implementations on switches generally don't isolate different experimenters' traffic from each other very well, meaning that one accident of this sort can fairly easily take down entire networks for many people.)

Analysis and VLANs

In principle, traffic in GENI is sliced by VLAN, such that each slice has one or more private VLANs that carry traffic only for that slice. In practice, the current GENI mesoscale OpenFlow network consists mainly of VLAN-hybrid switches, and thus uses a single shared VLAN (colloquially "1750", the VLAN ID number used by many (but not all) sites), where traffic is then sliced by other features, such as MAC address, ethertype, or IP subnet. The GENI network core page has more details about that.

The analysis engine differentiates between "exclusive" VLANs, "reserved" VLANs, and "shared" VLANs. The shared mesoscale VLAN (and other VLAN-hybrid datapaths) is effectively a special case of a shared VLAN, one in which no dl_vlan attribute is specified ("dl_vlan=none" is one way to think about it).

By default, FOAM treates each VLAN as exclusive, unless a FOAM admin has added the VLAN to the list of reserved or shared VLANs. (A VLAN can be on both lists, although there's not really any reason to do that; but if you want to change a VLAN from one to the other, for example, you need to both remove it from the old list and add it to the new one.)

Analysis checks

The analysis engine performs the following checks:

  1. If any port groups are defined, the request must include at most one of the ports in each port group.
  2. The request must not include any VLANs on the reserved list.
  3. If the request includes any exclusive VLANs: Those VLANs must not overlap with the VLANs of any already-approved slivers.
  4. If the request (a) includes any shared VLANs; and/or (b) includes any matches with no dl_vlan attribute; then the sliver's other matches must pass at least one of the following additional tests, and not fail any of them:
    1. If the request includes matches based on IP space: That IP space must not overlap with the IP space matches of any already-approved slivers.
    2. If the request includes matches based on MAC addresses: Those MAC addresses must not overlap with the MAC address matches of any already-approved slivers.
    3. If the request includes matches based on ethertypes (other than IP and ARP): Those ethertypes must not overlap with the ethertype matches of any already-approved slivers. For each of those tests, the result could be "pass", "fail", or "n/a" (e.g. if the request doesn't include IP space).

In that last set of tests, overlap is only considered within the context of a VLAN (or the "none" VLAN). For example, if you have three shared VLANs, and an already-approved sliver matches in one shared VLAN, another sliver which matches in a different shared VLAN will not be considered to overlap with the first sliver.

For slivers with only exclusive VLANs, if the request includes any other match criteria, those criteria will appear in the resulting FlowVisor flowspace rules; but they're entirely irrelevant to auto-approval for exclusive VLANs, which only considers whether the VLAN overlaps with an already-approved sliver. Experimenters are encouraged not to include any other match criteria in slivers with exclusive VLANs.

Sliver approval workflow

This section describes our workflow at BBN for approving slivers which aren't approved automatically.

FOAM sends e-mail about new slivers to the FOAM admin e-mail address that you configured when you set up FOAM. If further communication about a sliver request is needed, we copy that address on the e-mail, so that everyone will see it. We also send mail to to that address when we approve or reject the sliver (or if we review the request and we're not sure whether to approve or reject it), so everyone knows who did it.

If a sliver is automatically approved, FOAM will also send mail when that happens. If the automatic approval fails (e.g. because FlowVisor is down, because the new sliver uses the same controller as an existing sliver, etc), the admin *won't* get mail; is tracking that bug in FOAM.

If a sliver isn't automatically approved, it'll show up in the pending queue, and have pend_reason set to some text explaining why the sliver wasn't automatically approved. There's also a nightly cron job that sends a reminder message to the FOAM admin address about the slivers in the pending queue; you should generally either approve them, reject them, or contact the experimenter if you need more information before making a decision.

Some reasons to reject a sliver that wasn't automatically approved:

  • It includes multiple cross-connects.
  • It includes an I2-connected MyPLC plnode (ganel, gardil, sardis) and an NLR cross-connect, or an NLR plnode (bain, navis) and an I2 cross-connect.
  • It overlaps with something, and the existing sliver is more legitimate than the new one (e.g. the new sliver is trying to use someone else's subnet).

We generally follow up to the rejection e-mail message from FOAM, copying the experimenter, to explain why we rejected the sliver.

Some reasons to approve a sliver that wasn't automatically approved:

  • It includes an entire DPID and no ports, but is otherwise fine to approve.
  • It's something that we don't want to automatically approve for everyone, but are making a specific exception for in the case of this sliver/experimenter.

Since this is new, this list is still a little vague, and deciding whether to approve a non-auto-approved sliver is something of a judgment call, since in theory any safe sliver should be auto-approved, and thus any non-auto-approved sliver has a pretty high chance of being unsafe in some way. If we find a type of sliver that isn't auto-approved, but that is always safe to approve manually, we'll add more details here.

Regardless, we always send e-mail to the admin address saying what we did, so everyone's in the loop. (One convenient way is to reply to the notification message about the sliver.)

Managing FOAM slivers is the official guide to foamctl, and describes in detail everything that it can do. (NOTE: As of 2012-07-17, this guide was FOAM 0.6 specific, and marked as a "legacy document". FOAM-160 is tracking the task of updating it.)

Here are some specific commands that we've found useful for performing common tasks.

These commands all assume that you're running them on the FOAM server, and that you have a file /etc/foam.passwd, containing the FOAM admin password.

Re-approve all previously-approved slivers

If your FlowVisor dies badly or otherwise gets into a bad state, you can now re-approve all FOAM slivers that were previously approved, with one command:

foamctl geni:reapprove-slivers

Get a list of slivers

Pending ones (this is essentially "the queue" of slivers that are awaiting admin action):

foamctl geni:list-slivers -s Pending --passwd-file=/etc/foam.passwd

All active ones:

foamctl geni:list-slivers --passwd-file=/etc/foam.passwd

Either of these will give you a sliver URN; if you do

(with the actual URN of course), the rest of these commands will then work as-is -- or you can find a sliver from a slice name, see below.

Deleted ones:

foamctl geni:list-slivers --deleted --passwd-file=/etc/foam.passwd

Find a sliver from a slice name

If you know a user's slice name, you can grep for it:

foamctl geni:list-slivers --passwd-file=/etc/foam.passwd | egrep sliver_urn.+exampleslice

You can use this to get a sliver URN and/or an FV slice name from a GENI slice name, assigned to $sliver_urn and $flowvisor_slice:

slicename=exampleslice ; sliver_urn=$(foamctl geni:list-slivers --passwd-file=/etc/foam.passwd | egrep sliver_urn.+$slicename | sed -e 's/ *"sliver_urn": "\(.*\)".*/\1/') ; flowvisor_slice=$(echo $sliver_urn | awk -F : '{print $NF}')

Replace "exampleslice" with a string to egrep for. Choose something that's unique to the slice; note that if the string contains egrep regexp type characters, like * or +, you'll need to backslash those.

The rest of these commands assume that you've used that (or something similar) to set $sliver_urn.

Show a sliver's basic info

foamctl geni:show-sliver -u $sliver_urn --passwd-file=/etc/foam.passwd

Show a sliver's rspec

foamctl geni:show-sliver -r -u $sliver_urn --passwd-file=/etc/foam.passwd

Show a sliver's flowspec

foamctl geni:show-sliver -s -u $sliver_urn --passwd-file=/etc/foam.passwd

Show a sliver's flowspace

foamctl geni:show-sliver -f -u $sliver_urn --passwd-file=/etc/foam.passwd

Approve a sliver

This marks a sliver in FOAM as Approved, and adds its FV slice and flowspace rules for it to the FlowVisor (if it wasn't already).

foamctl geni:approve-sliver -u $sliver_urn --passwd-file=/etc/foam.passwd

Disable a sliver

This marks a sliver in FOAM as Pending, and removes its FV slice and flowspace rules for it from the FlowVisor (if there were any).

foamctl geni:disable-sliver -u $sliver_urn --passwd-file=/etc/foam.passwd

Reject a sliver

This marks a sliver in FOAM as Rejected, and removes its FV slice and flowspace rules for it from the FlowVisor (if there were any).

foamctl geni:reject-sliver -u $sliver_urn --passwd-file=/etc/foam.passwd

Delete a sliver

This disables a sliver, and marks it as deleted, just like the GENI AM API DeleteSliver call, and removes its FV slice and flowspace rules for it from the FlowVisor (if there were any).

foamctl geni:delete-sliver -u $sliver_urn --passwd-file=/etc/foam.passwd

You should generally only do this with the experimenter's permission, and if the experimenter can't delete their own sliver for some reason, so they're not confused about where their sliver went. (If you disable or reject it, they can still see it; if you delete it, it's essentially gone forever from their point of view.)

Compare two slivers

This compares two slivers (called "old" and "new" here, since a common use case is to compare a new request from an experimenter to an existing old request).

rm -f old.txt new.txt
oldsliver=<old sliver URN>
newsliver=<new sliver URN>
foamctl geni:show-sliver -r -u $oldsliver --passwd-file=/etc/foam.passwd > old.txt
foamctl geni:show-sliver -r -u $newsliver --passwd-file=/etc/foam.passwd > new.txt
diff -u old.txt new.txt

In the case of someone who deleted and then re-created a sliver, you could get the sliver URNs from the e-mail from FOAM, for example.

You can also use 'geni:show-sliver -s' to show flowspaces instead, which have the advantage of not including any comments or other formatting in the experimenter's rspec, but are a less familiar format for most admins.

Changing the FlowVisor password

FOAM's database stores information about the FlowVisor it manages, including its hostname and password. If you want to change that FlowVisor's fvadmin password, you need to change it both in FV itself, and also in FOAM.

First, change the password in Flowvisor:

fvctl --passwd-file=/etc/flowvisor.passwd changePasswd fvadmin
(enter the new password)

Then update /etc/flowvisor.passwd to contain the new password.

Finally, change the password in FOAM:

foamctl config:set-value --key="flowvisor.passwd" --value="$(cat /etc/flowvisor.passwd)" --passwd-file=/etc/foam.passwd

Testing FOAM

We have a separate page describing our procedure for testing FOAM, e.g. after an upgrade.