wiki:OpenFlow/FOAM

Version 44 (modified by Josh Smift, 11 years ago) (diff)

--

FOAM

FOAM is an OpenFlow aggregate manager, which sites in GENI use to allow experimenters to allocate OpenFlow resources.

https://openflow.stanford.edu/display/FOAM has more information about FOAM (from the official site at Stanford), including installation instructions, a FAQ with common error messages (for both experimenters and admins), etc.

Info for experimenters

The following sections are mostly of interest to GENI experimenters.

Rspecs

FOAM uses GENI v3 rspecs, with the OpenFlow v3 extensions. We have a page about how to write OF v3 rspecs, including some examples, information about differences from previous versions, etc. If you need a hand, just drop a note to help@geni.net.

Getting your sliver approved

If you allocate a shared resource that connects to an OpenFlow aggregate (e.g. a MyPLC plnode or ProtoGENI host), you'll typically also need to reserve some OpenFlow resources. When you do this, your reservation request may be held for approval, in which case a local FOAM admin will need to approve your request before your sliver actually becomes live. You should get e-mail from FOAM when your sliver is created, and another message when it's been approved; if you don't hear back, you may be able to reach a FOAM admin by replying to that message (or to the contact information in the body of the message).

As of FOAM 0.8, FOAM can read your e-mail address from your slice credential. If that address is valid, then you don't need to include an e-mail address in your rspec, but make sure it is! If it isn't, make sure that you also provide a valid email address in your rspec so that you can get the notifications about status changes of your OpenFlow sliver.

If you're setting up a multi-campus topology, note that your sliver will need to be approved separately at each FOAM aggregate.

Info for admins

The following sections are mostly of interest to FOAM admins.

System requirements

FOAM is fairly lightweight, and can run on the same system as the FlowVisor that it uses, without additional hardware requirements. You can also run it on a separate system for isolation purposes, e.g. so that if you need to reboot the FOAM server, that doesn't affect FlowVisor.

Version

The GPO currently recommends version 0.8.2 (the latest stable release) for GENI sites.

Note that there were significant changes between FOAM 0.6 and FOAM 0.8; http://groups.geni.net/geni/wiki/OpenFlow/FOAM?version=22 has the last version of this page with 0.6-specific information.

Initial configuration

The GPO has a variety of recommendations for configuring FOAM. We strongly encourage GENI sites to do all of these things.

Administrative e-mail

When you install FOAM for the first time (i.e. not an upgrade to an existing installation), you should configure its ability to send you administrative e-mail; those steps are part of the official FOAM 0.8 install guide.

Admin password

We typically put the FOAM admin password into /etc/foam.passwd, so you don't have to type it every time you run a foamctl command. Make sure that it's only readable by people who should have full admin access to FOAM on your system! (World-readable might be fine if only FOAM admins have accounts on your FOAM server; or you might want to make it only group-readable by a group that includes the FOAM admins.) The examples on the rest of this page assume that you've done that.

FlowVisor configuration

You'll then need to configure FOAM to point to your FlowVisor, which you can do interactively with foamctl config:set-flowvisor-info, or non-interactively like this (with values that work for your installation):

foamctl config:set-value --key="flowvisor.hostname" --value="localhost" --passwd-file=/etc/foam.passwd
foamctl config:set-value --key="flowvisor.passwd" --value="$(cat /etc/flowvisor.passwd)" --passwd-file=/etc/foam.passwd

Site tag

In order to work smoothly with GENI monitoring, you'll also need to set your site tag to a name that resolves, in DNS, to the system where your FOAM server runs (i.e. the hostname part of the GENI AM API URL). It defaults to the system's FQDN, and that's fine if you don't want to change it. You can also change it to something more abstract, like a CNAME such as "foam.gpolab.bbn.com", which is what we do. To change it, use a command like this, replacing the "value=" part with a good value for your site:

foamctl config:set-value --key="geni.site-tag" --value="foam.gpolab.bbn.com" --passwd-file=/etc/foam.passwd

Monitoring

You should then install and configure the FOAM monitoring package; we have a separate guide for that.

Other settings

For GENI purposes, we recommend also configuring your admin e-mail (which causes it to show up in sliverstatus), and max lease:

  • The admin e-mail setting should generally be the same address you used when you set up e-mail. We use foam-admin@gpolab.bbn.com (a list with a couple of admins on it) at BBN.
  • The max lease setting controls how the maximum value of the sliver expiration date (e.g. when an experimenter does a renewsliver). We use 9000 hours (about a year) at BBN, and recommend that; making this short just means that experimenters who want a long-lived sliver have to do a renewsliver more frequently.

To set those with foamctl (replace the "value" part with the values you want to use):

foamctl config:set-value --key="site.admin.email" --value="foam-admin@gpolab.bbn.com" --passwd-file=/etc/foam.passwd
foamctl config:set-value --key="geni.max-lease" --value=9000 --passwd-file=/etc/foam.passwd

Version holding

Finally, we also recommend telling dpkg to hold FOAM and FlowVisor at their current versions, so that if you do a general upgrade of all of the packages on your system, they won't get updated along with that, but will only update when you explicitly say so. To do that on Ubunutu:

/bin/echo "foam hold" | /usr/bin/dpkg --set-selections 
/bin/echo "flowvisor hold" | /usr/bin/dpkg --set-selections 

Upgrading

Here are some before-and-after tests you can do when you upgrade FOAM, to confirm that your state hasn't changed unexpectedly.

On the FOAM server, get a "before" list of slivers, active and deleted:

cd 
rm -rf ~/tmp/foam-upgrade 
mkdir -p ~/tmp/foam-upgrade/before 
cd ~/tmp/foam-upgrade/before 

foamctl geni:list-slivers --passwd-file=/etc/foam.passwd > list-slivers-active.txt 
foamctl geni:list-slivers --deleted --passwd-file=/etc/foam.passwd > list-slivers-deleted.txt 

egrep '(email|desc|slice_urn|sliver_urn)' list-slivers-active.txt | sort > list-slivers-active-cooked.txt 
egrep '(email|desc|slice_urn|sliver_urn)' list-slivers-deleted.txt | sort > list-slivers-deleted-cooked.txt 

If you own any slivers on the server, you can also do a listresources and sliverstatus for a sliver or two.

Then do the upgrade, as usual. For simple upgrades:

sudo apt-get update
sudo service foam stop
sudo apt-get install -y --force-yes foam
sudo service foam start

More complex upgrades might have additional steps, which will generally be described on a separate page, in the announcement of the upgrade, and/or in the release notes.

Then, on the FOAM server, get an "after" list of slivers, active and deleted:

cd 
rm -rf ~/tmp/foam-upgrade/after 
mkdir -p ~/tmp/foam-upgrade/after 
cd ~/tmp/foam-upgrade/after 

foamctl geni:list-slivers --passwd-file=/etc/foam.passwd > list-slivers-active.txt 
foamctl geni:list-slivers --deleted --passwd-file=/etc/foam.passwd > list-slivers-deleted.txt 

egrep '(email|desc|slice_urn|sliver_urn)' list-slivers-active.txt | sort > list-slivers-active-cooked.txt 
egrep '(email|desc|slice_urn|sliver_urn)' list-slivers-deleted.txt | sort > list-slivers-deleted-cooked.txt 

Diff the results:

for file in *cooked.txt ; do diff -u ../before/$file $file ; done 

You should typically expect no differences.

Advisories

Here are some things that FOAM admins should be aware of and watch out for.

Rspecs that include both cross-connect ports

In the current standard campus topology, a local OpenFlow-controlled VLAN (often VLAN 1750) is cross-connected to multiple inter-campus VLANs, and a given sliver typically only includes one cross-connect port. FOAM admins should be careful not to approve rspecs that include both cross-connect ports (either explicitly or implicitly, e.g. by requesting the entire local DPID for VLAN 1750).

Some experimenters may have good reasons for wanting to use both cross-connects; if they do, make sure they understand that this is a risky topology before you approve their sliver.

Rspecs that include match rules with no protocol specified

Rspecs that include match rules at one layer typically need to include match rules specifying the protocol at a lower layer. For example:

  • An rspec that specifies a layer 3 match, like nw_src=A.B.C.D/N, also needs to specify dl_type to indicate what the layer 3 protocol is (e.g. dl_type=0x800 for IP and/or 0x806 for ARP).
  • An rspec that specifies a layer 4 match, like tp_port=N, also needs to specify nw_proto to indicate what the layer 4 protocol is (e.g. nw_proto=1 for ICMP, or 6 for TCP, or 17 for UDP).

The important thing is that if the protocol isn't specified, some switches will end up with flowtable entries that match more traffic than they should, interfering with other experiments at your site. FOAM admins should be careful not to approve rspecs that include a match at one layer without specifying the protocol at a lower layer.

We don't currently think that there's a valid use case for an experimenter wanting to specify a match at one layer without specifying the protocol at a lower layer, but will amend this advisory if we encounter one.

Auto-approval

FOAM 0.8 includes a simple analysis engine which can be used to inform auto-approval decisions. The analysis is done by default, but can be disabled if you really want to for some reason.

The action taken on a newly created sliver depends on the geni.approval.approve-on-creation configuration option. It has three possible values:

  • Mode 0, "never": Newly created slivers are never automatically approved.
  • Mode 1, "always": Newly created slivers are always automatically approved.
  • Mode 2, "analysis": Newly created slivers are automatically approved if they pass all the checks in the analysis engine.

The default value is "0", for safety, so no slivers will be auto-approved.

The rest of this section is only really relevant if you use mode 2. If you've read the rest of this section and want to do that, you can do it with foamctl:

foamctl config:set-value --key="geni.approval.approve-on-creation" --value="2" --passwd-file=/etc/foam.passwd

The analysis engine performs three tests related to the sliver's flowspace:

  1. If the sliver's flowspace includes IP space: That IP space must not overlap with the IP space of any already-approved slivers.
  2. If the sliver's flowspace includes MAC addresses: Those MAC addresses must not overlap with the MAC addresses of any already-approved slivers.
  3. If the sliver's flowspace includes ethertypes (other than IP and ARP): Those ethertypes must not overlap with the ethertypes of any already-approved slivers.

For each of those tests, the result could be "pass", "fail", or "n/a" (e.g. if the sliver's flowspace doesn't include IP space, or there are no port groups defined).

It also performs three tests related to datapaths, ports, and VLANs:

  1. The sliver's flowspace must include at least one datapath.
  2. The sliver's flowspace must not include more than one of the ports in any port group.
  3. The sliver's flowspace must not include any flowspace defined by VLAN. (no 'dl_vlan')

In mode 2, the sliver is automatically approved if at least one test in the first group passes, and no test fails (in either group).

In practice, this implements a policy somewhat like: "Auto-approve a sliver that includes IP, MAC, or ethertype flowspace, doesn't overlap with anything else, and doesn't include more than one port in these port groups". Port groups might include things like cross-connects, or any other collection of ports where you only want a given sliver to have one port in the group.

If a sliver is not auto-approved, the 'pend_reason' field (visible in 'foamctl geni:show-sliver') should explain why. (More precisely, it'll include one test that the sliver didn't pass; if it failed multiple tests, only one will be listed.)

A more flexible policy is expected in future versions of FOAM.

Sliver approval workflow

This section describes our workflow for approving slivers at BBN.

FOAM sends e-mail about new slivers to the FOAM admin e-mail address that you configured when you set up FOAM. If further communication about a sliver request is needed, we copy that address on the e-mail, so that everyone will see it. We also send mail to to that address when we approve or reject the sliver (or if we review the request and we're not sure whether to approve or reject it), so everyone knows who did it.

If a sliver is automatically approved, FOAM will also send mail when that happens. If the automatic approval fails (e.g. because FlowVisor is down, because the new sliver uses the same controller as an existing sliver, etc), the admin *won't* get mail; https://openflow.stanford.edu/bugs/browse/FOAM-288 is tracking that bug in FOAM.

If a sliver isn't automatically approved, it'll show up in the pending queue, and have pend_reason set to some text explaining why the sliver wasn't automatically approved. There's also a nightly cron job that sends a reminder message to the FOAM admin address about the slivers in the pending queue; you should generally either approve them, reject them, or contact the experimenter if you need more information before making a decision.

Some reasons to reject a sliver that wasn't automatically approved:

  • It includes multiple cross-connects.
  • It includes an I2-connected MyPLC plnode (ganel, gardil, sardis) and an NLR cross-connect, or an NLR plnode (bain, navis) and an I2 cross-connect.
  • It overlaps with something, and the existing sliver is more legitimate than the new one (e.g. the new sliver is trying to use someone else's subnet).

We generally follow up to the rejection e-mail message from FOAM, copying the experimenter, to explain why we rejected the sliver.

Some reasons to approve a sliver that wasn't automatically approved:

  • It includes a VLAN tag, but is otherwise fine to approve.
  • It includes an entire DPID and no ports, but is otherwise fine to approve.
  • It's something that we don't want to automatically approve for everyone, but are making a specific exception for in the case of this sliver/experimenter.

Since this is new, this list is still a little vague, and deciding whether to approve a non-auto-approved sliver is something of a judgment call, since in theory any safe sliver should be auto-approved, and thus any non-auto-approved sliver has a pretty high chance of being unsafe in some way. If we find a type of sliver that isn't auto-approved, but that is always safe to approve manually, we'll add more details here.

Regardless, we always send e-mail to the admin address saying what we did, so everyone's in the loop. (One convenient way is to reply to the notification message about the sliver.)

Managing FOAM slivers

https://openflow.stanford.edu/display/FOAM/foamctl+Guide is the official guide to foamctl, and describes in detail everything that it can do. (NOTE: As of 2012-07-17, this guide was FOAM 0.6 specific, and marked as a "legacy document". FOAM-160 is tracking the task of updating it.)

Here are some specific commands that we've found useful for performing common tasks.

These commands all assume that you're running them on the FOAM server, and that you have a file /etc/foam.passwd, containing the FOAM admin password.

Get a list of slivers

Pending ones (this is essentially "the queue" of slivers that are awaiting admin action):

foamctl geni:list-slivers -s Pending --passwd-file=/etc/foam.passwd

All active ones:

foamctl geni:list-slivers --passwd-file=/etc/foam.passwd

Either of these will give you a sliver URN; if you do

sliver_urn=urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbsstghosts:678fc69b-76e1-4a50-9fb2-ab5c4a5298d6

(with the actual URN of course), the rest of these commands will then work as-is -- or you can find a sliver from a slice name, see below.

Deleted ones:

foamctl geni:list-slivers --deleted --passwd-file=/etc/foam.passwd

Find a sliver from a slice name

If you know a user's slice name, you can grep for it:

foamctl geni:list-slivers --passwd-file=/etc/foam.passwd | egrep sliver_urn.+exampleslice

You can use this to get a sliver URN and/or an FV slice name from a GENI slice name, assigned to $sliver_urn and $flowvisor_slice:

slicename=exampleslice ; sliver_urn=$(foamctl geni:list-slivers --passwd-file=/etc/foam.passwd | egrep sliver_urn.+$slicename | sed -e 's/ *"sliver_urn": "\(.*\)".*/\1/') ; flowvisor_slice=$(echo $sliver_urn | awk -F : '{print $NF}')

Replace "exampleslice" with a string to egrep for. Choose something that's unique to the slice; note that if the string contains egrep regexp type characters, like * or +, you'll need to backslash those.

The rest of these commands assume that you've used that (or something similar) to set $sliver_urn.

Show a sliver's basic info

foamctl geni:show-sliver -u $sliver_urn --passwd-file=/etc/foam.passwd

Show a sliver's rspec

foamctl geni:show-sliver -r -u $sliver_urn --passwd-file=/etc/foam.passwd

Show a sliver's flowspec

foamctl geni:show-sliver -s -u $sliver_urn --passwd-file=/etc/foam.passwd

Show a sliver's flowspace

foamctl geni:show-sliver -f -u $sliver_urn --passwd-file=/etc/foam.passwd

Approve a sliver

This marks a sliver in FOAM as Approved, and adds its FV slice and flowspace rules for it to the FlowVisor (if it wasn't already).

foamctl geni:approve-sliver -u $sliver_urn --passwd-file=/etc/foam.passwd

Disable a sliver

This marks a sliver in FOAM as Pending, and removes its FV slice and flowspace rules for it from the FlowVisor (if there were any).

foamctl geni:disable-sliver -u $sliver_urn --passwd-file=/etc/foam.passwd

Reject a sliver

This marks a sliver in FOAM as Rejected, and removes its FV slice and flowspace rules for it from the FlowVisor (if there were any).

foamctl geni:reject-sliver -u $sliver_urn --passwd-file=/etc/foam.passwd

Delete a sliver

This disables a sliver, and marks it as deleted, just like the GENI AM API DeleteSliver call, and removes its FV slice and flowspace rules for it from the FlowVisor (if there were any).

foamctl geni:delete-sliver -u $sliver_urn --passwd-file=/etc/foam.passwd

You should generally only do this with the experimenter's permission, and if the experimenter can't delete their own sliver for some reason, so they're not confused about where their sliver went. (If you disable or reject it, they can still see it; if you delete it, it's essentially gone forever from their point of view.)

Compare two slivers

This compares two slivers (called "old" and "new" here, since a common use case is to compare a new request from an experimenter to an existing old request).

rm -f old.txt new.txt
oldsliver=<old sliver URN>
newsliver=<new sliver URN>
foamctl geni:show-sliver -r -u $oldsliver --passwd-file=/etc/foam.passwd > old.txt
foamctl geni:show-sliver -r -u $newsliver --passwd-file=/etc/foam.passwd > new.txt
diff -u old.txt new.txt

In the case of someone who deleted and then re-created a sliver, you could get the sliver URNs from the e-mail from FOAM, for example.

You can also use 'geni:show-sliver -s' to show flowspaces instead, which have the advantage of not including any comments or other formatting in the experimenter's rspec, but are a less familiar format for most admins.

Slice Authority trust configuration

FOAM has a set of CA certificates that it uses for user authorization: If a user has a certificate signed by a CA that FOAM is configured to trust, then that user can talk to FOAM (and create FOAM slivers, which will be approved automatically if auto-approval is on, etc).

To configure what Slice Authorities FOAM trusts, install or remove the CA cert for the Slice Authority in a file in /opt/foam/etc/gcf-ca-certs, and then rebuild the nginx CA cert bundle and restart FOAM and nginx:

sudo foamctl geni:bundle-certs
sudo service foam restart
sudo service nginx restart

GENI deployments should trust the pgeni.gpolab.bbn.com SA; the official FOAM installation guide includes this step, or you can get the cert from http://www.pgeni.gpolab.bbn.com/ca-cert/pgeni.gpolab.bbn.com.pem if you need it.

Changing the FlowVisor password

FOAM's database stores information about the FlowVisor it manages, including its hostname and password. If you want to change that FlowVisor's fvadmin password, you need to change it both in FV itself, and also in FOAM.

First, change the password in Flowvisor:

fvctl --passwd-file=/etc/flowvisor.passwd changePasswd fvadmin
(enter the new password)

Then update /etc/flowvisor.passwd to contain the new password.

Finally, change the password in FOAM:

foamctl config:set-value --key="flowvisor.passwd" --value="$(cat /etc/flowvisor.passwd)" --passwd-file=/etc/foam.passwd

Testing FOAM

We have a separate page describing our procedure for testing FOAM, e.g. after an upgrade.