wiki:OpenFlow/FOAM

Version 33 (modified by Josh Smift, 12 years ago) (diff)

--

FOAM

FOAM is an OpenFlow aggregate manager, which sites in GENI use to allow experimenters to allocate OpenFlow resources.

https://openflow.stanford.edu/display/FOAM has more information about FOAM (from the official site at Stanford), including installation instructions, a FAQ with common error messages (for both experimenters and admins), etc.

Info for experimenters

The following sections are mostly of interest to GENI experimenters.

Rspecs

FOAM uses GENI v3 rspecs, with the OpenFlow v3 extensions. We have a page about how to write OF v3 rspecs, including some examples, information about differences from previous versions, etc. If you need a hand, just drop a note to help@geni.net.

Getting your sliver approved

If you allocate a shared resource that connects to an OpenFlow aggregate (e.g. a MyPLC plnode or ProtoGENI host), you'll typically also need to reserve some OpenFlow resources. When you do this, your reservation request may be held for approval, in which case a local FOAM admin will need to approve your request before your sliver actually becomes live. You should get e-mail from FOAM when your sliver is created, and another message when it's been approved; if you don't hear back, you may be able to reach a FOAM admin by replying to that message (or to the contact information in the body of the message).

As of FOAM 0.8, FOAM can read your e-mail address from your slice credential. If that address is valid, then you don't need to include an e-mail address in your rspec, but make sure it is! If it isn't, make sure that you also provide a valid email address in your rspec so that you can get the notifications about status changes of your OpenFlow sliver.

If you're setting up a multi-campus topology, note that your sliver will need to be approved separately at each FOAM aggregate.

Info for admins

The following sections are mostly of interest to FOAM admins.

Version

The GPO currently recommends version 0.8.2 (the latest stable release) for GENI sites. There are many changes from 0.8 to 0.6, and we recommend working with the GPO to help the upgrade process go smoothly. Contact us at gpo-infra@geni.net and we'll be happy to help!

NOTE that much of this page is FOAM 0.8 specific. http://groups.geni.net/geni/wiki/OpenFlow/FOAM?version=22 has the last 0.6 specific version.

Initial configuration

When you install FOAM for the first time (i.e. not an upgrade to an existing installation), you should configure e-mail; those steps are part of the official FOAM 0.8 install guide.

You'll need to configure FOAM to point to your FlowVisor, which you can do interactively with foamctl config:set-flowvisor-info, or non-interactively like this (with values that work for your installation):

foamctl config:set-value --key="flowvisor.hostname" --value="localhost" --passwd-file=/opt/foam/etc/foampasswd
foamctl config:set-value --key="flowvisor.passwd" --value="$(cat /etc/flowvisor/fvpasswd)" --passwd-file=/opt/foam/etc/foampasswd

For GENI purposes, we recommend also configuring your site tag, admin e-mail (which causes it to show up in sliverstatus), and max lease:

  • Your site tag should be something that uniquely identifies this FOAM instance. We use foam.gpolab.bbn.com (a CNAME that points to our FOAM server) at BBN.
  • The admin e-mail setting should generally be the same address you used when you set up e-mail. We use foam-admin@gpolab.bbn.com (a list with a couple of admins on it) at BBN.
  • The max lease setting controls how the maximum value of the sliver expiration date (e.g. when an experimenter does a renewsliver). We use 9000 hours (about a year) at BBN, and recommend that; making this short just means that experimenters who want a long-lived sliver have to do a renewsliver more frequently.

We also typically put the FOAM admin password into /opt/foam/etc/foampasswd, so you don't have to type it every time you run a foamctl command. Make sure that it's only readable by people who should have full admin access to FOAM on your system! (World-readable might be fine if only FOAM admins have accounts on your FOAM server; or you might want to make it only group-readable by a group that includes the FOAM admins.)

To set those with foamctl (replace the "value" part with the values you want to use):

foamctl config:set-value --key="geni.site-tag" --value="foam.gpolab.bbn.com" --passwd-file=/opt/foam/etc/foampasswd
foamctl config:set-value --key="site.admin.email" --value="foam-admin@gpolab.bbn.com" --passwd-file=/opt/foam/etc/foampasswd
foamctl config:set-value --key="geni.max-lease" --value=9000 --passwd-file=/opt/foam/etc/foampasswd

Upgrading

Here are some before-and-after tests you can do when you upgrade FOAM, to confirm that your state hasn't changed unexpectedly.

Note that if you're upgrading from FOAM 0.6 to 0.8, you'll need to modify the foamctl commands in the "before" section, changing "geni:list-slivers" to "list-slivers".

On the FOAM server, get a "before" list of slivers, active and deleted:

cd 
rm -rf ~/tmp/foam-upgrade 
mkdir -p ~/tmp/foam-upgrade/before 
cd ~/tmp/foam-upgrade/before 

foamctl geni:list-slivers --passwd-file=/opt/foam/etc/foampasswd > list-slivers-active.txt 
foamctl geni:list-slivers --deleted --passwd-file=/opt/foam/etc/foampasswd > list-slivers-deleted.txt 

egrep '(email|desc|slice_urn|sliver_urn)' list-slivers-active.txt | sort > list-slivers-active-cooked.txt 
egrep '(email|desc|slice_urn|sliver_urn)' list-slivers-deleted.txt | sort > list-slivers-deleted-cooked.txt 

If you own any slivers on the server, you can also do a listresources and sliverstatus for a sliver or two.

Then do the upgrade, as usual.

On the FOAM server, get an "after" list of slivers, active and deleted:

cd 
rm -rf ~/tmp/foam-upgrade/after 
mkdir -p ~/tmp/foam-upgrade/after 
cd ~/tmp/foam-upgrade/after 

foamctl geni:list-slivers --passwd-file=/opt/foam/etc/foampasswd > list-slivers-active.txt 
foamctl geni:list-slivers --deleted --passwd-file=/opt/foam/etc/foampasswd > list-slivers-deleted.txt 

egrep '(email|desc|slice_urn|sliver_urn)' list-slivers-active.txt | sort > list-slivers-active-cooked.txt 
egrep '(email|desc|slice_urn|sliver_urn)' list-slivers-deleted.txt | sort > list-slivers-deleted-cooked.txt 

Diff the results:

for file in *cooked.txt ; do diff -u ../before/$file $file ; done 

Advisories

Here are some things that FOAM admins should be aware of and watch out for.

Rspecs that include both cross-connect ports

In the current standard campus topology, a local OpenFlow-controlled VLAN (often VLAN 1750) is cross-connected to multiple inter-campus VLANs, and a given sliver typically only includes one cross-connect port. FOAM admins should be careful not to approve rspecs that include both cross-connect ports (either explicitly or implicitly, e.g. by requesting the entire local DPID for VLAN 1750).

Some experimenters may have good reasons for wanting to use both cross-connects; if they do, make sure they understand that this is a risky topology before you approve their sliver.

Rspecs that include match rules with no protocol specified

Rspecs that include match rules at one layer typically need to include match rules specifying the protocol at a lower layer. For example:

  • An rspec that specifies a layer 3 match, like nw_src=A.B.C.D/N, also needs to specify dl_type to indicate what the layer 3 protocol is (e.g. dl_type=0x800 for IP and/or 0x806 for ARP).
  • An rspec that specifies a layer 4 match, like tp_port=N, also needs to specify nw_proto to indicate what the layer 4 protocol is (e.g. nw_proto=1 for ICMP, or 6 for TCP, or 17 for UDP).

The important thing is that if the protocol isn't specified, some switches will end up with flowtable entries that match more traffic than they should, interfering with other experiments at your site. FOAM admins should be careful not to approve rspecs that include a match at one layer without specifying the protocol at a lower layer.

We don't currently think that there's a valid use case for an experimenter wanting to specify a match at one layer without specifying the protocol at a lower layer, but will amend this advisory if we encounter one.

Administrative list-resources blocks other FOAM operations

In FOAM 0.6, an administrative list-resources call locks the FOAM database, and thus blocks other FOAM operations. If an experimenter tries to use FOAM while this is happening, they'll get a "database is locked" error; if they retry, it should work. (If the error persists, something else is probably wrong.)

The FOAM monitoring code (from the 'tango-monitor-foam' package) does an administrative list-slivers whenever it runs, and it runs once per minute by default, so there's small window each minute (a second or two) when this can happen.

This issue is fixed in FOAM 0.8.

Auto-approval

FOAM 0.8 adds a simple analysis engine which can be used to inform auto-approval decisions. The analysis is done by default, but can be disabled if you really want to for some reason.

The action taken on a newly created sliver depends on the geni.approval.approve-on-creation configuration option. It has three possible values:

  • Mode 0, "never": Newly created slivers are never automatically approved.
  • Mode 1, "always": Newly created slivers are always automatically approved.
  • Mode 2, "analysis": Newly created slivers are automatically approved if they pass all the checks in the analysis engine.

The default value is "0", for safety, so no slivers will be auto-approved.

The rest of this section is only really relevant if you use mode 2. If you've read the rest of this section and want to do that, you can do it with foamctl:

foamctl config:set-value --key="geni.approval.approve-on-creation" --value="2" --passwd-file=/opt/foam/etc/foampasswd

The analysis engine performs three tests related to the sliver's flowspace:

  1. If the sliver's flowspace includes IP space: That IP space must not overlap with the IP space of any already-approved slivers.
  2. If the sliver's flowspace includes MAC addresses: Those MAC addresses must not overlap with the MAC addresses of any already-approved slivers.
  3. If the sliver's flowspace includes ethertypes (other than IP and ARP): Those ethertypes must not overlap with the ethertypes of any already-approved slivers.

For each of those tests, the result could be "pass", "fail", or "n/a" (e.g. if the sliver's flowspace doesn't include IP space, or there are no port groups defined).

It also performs three tests related to datapaths, ports, and VLANs:

  1. The sliver's flowspace must include at least one datapath.
  2. The sliver's flowspace must not include more than one of the ports in any port group.
  3. The sliver's flowspace must not include any flowspace defined by VLAN. (no 'dl_vlan')

In mode 2, the sliver is automatically approved if at least one test in the first group passes, and no test fails (in either group).

In practice, this implements a policy somewhat like: "Auto-approve a sliver that includes IP, MAC, or ethertype flowspace, doesn't overlap with anything else, and doesn't include more than one port in these port groups". Port groups might include things like cross-connects, or any other collection of ports where you only want a given sliver to have one port in the group.

If a sliver is not auto-approved, the 'pend_reason' field (visible in 'foamctl geni:show-sliver') should explain why. (More precisely, it'll include one test that the sliver didn't pass; if it failed multiple tests, only one will be listed.)

A more flexible policy is expected in future versions of FOAM.

Sliver approval workflow

This section describes our workflow for approving slivers at BBN.

FOAM sends e-mail about new slivers to the FOAM admin e-mail address that you configured when you set up FOAM. If further communication about a sliver request is needed, we copy that address on the e-mail, so that everyone will see it. We also send mail to to that address when we approve or reject the sliver (or if we review the request and we're not sure whether to approve or reject it), so everyone knows who did it.

If a sliver is automatically approved, FOAM will also send mail when that happens. If the automatic approval fails (e.g. because FlowVisor is down, because the new sliver uses the same controller as an existing sliver, etc), the admin *won't* get mail; https://openflow.stanford.edu/bugs/browse/FOAM-288 is tracking that bug in FOAM.

If a sliver isn't automatically approved, it'll show up in the pending queue, and have pend_reason set to some text explaining why the sliver wasn't automatically approved.

Some reasons to reject a sliver that wasn't automatically approved:

  • It includes multiple cross-connects.
  • It includes an I2-connected MyPLC plnode (ganel, gardil, sardis) and an NLR cross-connect, or an NLR plnode (bain, navis) and an I2 cross-connect.
  • It overlaps with something, and the existing sliver is more legitimate than the new one (e.g. someone's trying to use someone else's subnet).

We generally follow up to the rejection e-mail message from FOAM, copying the experimenter, to explain why we rejected the sliver.

Some reasons to approve a sliver that wasn't automatically approved:

  • It includes a VLAN tag, but is otherwise fine to approve.
  • It includes an entire DPID and no ports, but is otherwise fine to approve.
  • It's something that we don't want to automatically approve for everyone, but are making a specific exception for in the case of this sliver/experimenter.

Since this is new, this list is still a little vague, and deciding whether to approve a non-auto-approved sliver is something of a judgment call, since in theory any safe sliver should be auto-approved, and thus any non-auto-approved sliver has a pretty high chance of being unsafe in some way. If we find a type of sliver that isn't auto-approved, but that is always safe to approve manually, we'll add more details here.

Regardless, we always send e-mail to the admin address saying what we did, so everyone's in the loop. (One convenient way is to reply to the notification message about the sliver.)

Managing FOAM slivers

https://openflow.stanford.edu/display/FOAM/foamctl+Guide is the official guide to foamctl, and describes in detail everything that it can do. (NOTE: As of 2012-07-17, this guide was FOAM 0.6 specific, and marked as a "legacy document". FOAM-160 is tracking the task of updating it.)

Here are some specific commands that we've found useful for performing common tasks.

These commands all assume that you're running them on the FOAM server, and that you have a file /opt/foam/etc/foampasswd, containing the FOAM admin password.

Get a list of slivers

Pending ones (this is essentially "the queue" of slivers that are awaiting admin action):

foamctl geni:list-slivers -s Pending --passwd-file=/opt/foam/etc/foampasswd

All active ones:

foamctl geni:list-slivers --passwd-file=/opt/foam/etc/foampasswd

Either of these will give you a sliver URN; if you do

sliver_urn=urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbsstghosts:678fc69b-76e1-4a50-9fb2-ab5c4a5298d6

(with the actual URN of course), the rest of these commands will then work as-is -- or you can find a sliver from a slice name, see below.

Deleted ones:

foamctl geni:list-slivers --deleted --passwd-file=/opt/foam/etc/foampasswd

Find a sliver from a slice name

If you know a user's slice name, you can grep for it:

foamctl geni:list-slivers --passwd-file=/opt/foam/etc/foampasswd | egrep sliver_urn.+exampleslice

You can use this to get a sliver URN and/or an FV slice name from a GENI slice name, assigned to $sliver_urn and $flowvisor_slice:

slicename=exampleslice ; sliver_urn=$(foamctl geni:list-slivers --passwd-file=/opt/foam/etc/foampasswd | egrep sliver_urn.+$slicename | sed -e 's/ *"sliver_urn": "\(.*\)".*/\1/') ; flowvisor_slice=$(echo $sliver_urn | awk -F : '{print $NF}')

Replace "exampleslice" with a string to egrep for. Choose something that's unique to the slice; note that if the string contains egrep regexp type characters, like * or +, you'll need to backslash those.

The rest of these commands assume that you've used that (or something similar) to set $sliver_urn.

Show a sliver's basic info

foamctl geni:show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

Show a sliver's rspec

foamctl geni:show-sliver -r -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

Show a sliver's flowspec

foamctl geni:show-sliver -s -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

Show a sliver's flowspace

foamctl geni:show-sliver -f -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

Approve a sliver

This marks a sliver in FOAM as Approved, and adds its FV slice and flowspace rules for it to the FlowVisor (if it wasn't already).

foamctl geni:approve-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

Disable a sliver

This marks a sliver in FOAM as Pending, and removes its FV slice and flowspace rules for it from the FlowVisor (if there were any).

foamctl geni:disable-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

Reject a sliver

This marks a sliver in FOAM as Rejected, and removes its FV slice and flowspace rules for it from the FlowVisor (if there were any).

foamctl geni:reject-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

Delete a sliver

This disables a sliver, and marks it as deleted, just like the GENI AM API DeleteSliver call, and removes its FV slice and flowspace rules for it from the FlowVisor (if there were any).

foamctl geni:delete-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

You should generally only do this with the experimenter's permission, and if the experimenter can't delete their own sliver for some reason, so they're not confused about where their sliver went. (If you disable or reject it, they can still see it; if you delete it, it's essentially gone forever from their point of view.)

Compare two slivers

This compares two slivers (called "old" and "new" here, since a common use case is to compare a new request from an experimenter to an existing old request).

rm -f old.txt new.txt
oldsliver=<old sliver URN>
newsliver=<new sliver URN>
foamctl geni:show-sliver -r -u $oldsliver --passwd-file=/opt/foam/etc/foampasswd > old.txt
foamctl geni:show-sliver -r -u $newsliver --passwd-file=/opt/foam/etc/foampasswd > new.txt
diff -u old.txt new.txt

In the case of someone who deleted and then re-created a sliver, you could get the sliver URNs from the e-mail from FOAM, for example.

You can also use 'geni:show-sliver -s' to show flowspaces instead, which have the advantage of not including any comments or other formatting in the experimenter's rspec, but are a less familiar format for most admins.

Slice Authority trust configuration

You may want to configure FOAM to trust user certificates signed by additional Slice Authorities. To do that, install the CA cert for the Slice Authority in a file in /opt/foam/etc/gcf-ca-certs, and then rebuild the nginx CA cert bundle and restart FOAM and nginx:

sudo foamctl geni:bundle-certs
sudo service foam restart
sudo service nginx restart

In particular, GENI mesoscale deployments should trust the pgeni.gpolab.bbn.com SA; the official FOAM installation guide includes this step, or you can get the cert from http://www.pgeni.gpolab.bbn.com/ca-cert/pgeni.gpolab.bbn.com.pem if you need it.

Changing the FlowVisor password

FOAM's database stores information about the FlowVisor it manages, including its hostname and password. If you want to change that FlowVisor's password, you need to change it both in FV itself, and also in FOAM.

  • Change the password in FlowVisor: this assumes /etc/flowvisor/config.xml FlowVisor's config, and is owned by user openflow:
    • Change the password in FlowVisor's config:
      $ sudo service flowvisor stop
      $ sudo -u openflow fvconfig chpasswd /etc/flowvisor/config.xml fvadmin
      Enter password for account 'fvadmin' on the flowvisor:
      Enter password again
      $ sudo service flowvisor start  
      
    • If you store the password in /etc/flowvisor/fvpasswd, change it there too by hand:
      $ sudo vi /etc/flowvisor/fvpasswd
      
  • Change the password in FOAM:
    foamctl config:set-value --key="flowvisor.passwd" --value="$(cat /etc/flowvisor/fvpasswd)" --passwd-file=/opt/foam/etc/foampasswd
    

Testing FOAM

We have a separate page describing our procedure for testing FOAM, e.g. after an upgrade.