wiki:OpenFlow/FOAM

Version 9 (modified by Josh Smift, 8 years ago) (diff)

--

FOAM

FOAM is an OpenFlow aggregate manager, which sites in GENI use to allow experimenters to allocate OpenFlow resources.

OpenFlow aggregates in GENI are currently in the process of migrating to FOAM from Expedient.

https://openflow.stanford.edu/display/DOCS/FAQ has more information, including common error messages (for both experimenters and admins).

Info for experimenters

The following sections are mostly of interest to GENI experimenters.

Rspec changes

Expedient used OpenFlow v1 and v2 rspecs; FOAM uses GENI v3 with the OpenFlow v3 extensions. We have a page about how to write OF v3 rspecs, including some examples, information about differences from OF v2, etc. If you need a hand, just drop a note to help@geni.net.

Aggregate Manager URL changes

The Aggregate Manager URL for FOAM is generally different from the one for Expedient. FOAM URL typically look like this:

https://foam.gpolab.bbn.com:3626/foam/gapi/1

In particular, note port 3626, "/foam/gapi/1" as the pathname (the one on the end indicates GENI AM API v1), and no trailing slash.

Expedient URL typically looked like this:

https://expedient.gpolab.bbn.com:1443/openflow/gapi/

In particular, note port 1443, "/openflow/gapi/" as the pathname, and a trailing slash.

Getting your sliver approved

If you allocate a shared resource that connects to an OpenFlow aggregate (e.g. a MyPLC plnode or ProtoGENI host), you'll typically also need to reserve some OpenFlow resources. When you do this, your reservation request may be held for approval, and a local FOAM admin needs to approve your request before your sliver actually becomes live. You should get e-mail from FOAM when your sliver is created, and another message when it's been approved; if you don't hear back, you may be able to reach a FOAM admin by replying to that message.

Note that you don't need to send a separate email to the administrator, like you did for Expedient, FOAM will automatically send the email to the administrator of FOAM. Make sure that you also provide a valid email address in your rspec so that you can get the notifications about status changes of your OpenFlow sliver.

If you're setting up a multi-campus topology, note that your sliver will need to be approved separately at each FOAM aggregate.

Info for admins

The following sections are mostly of interest to FOAM admins.

Advisories

Here are some things that FOAM admins should be aware of and watch out for.

Rspecs that include both cross-connect ports

In the current standard campus topology, a local OpenFlow-controlled VLAN (often VLAN 1750) is cross-connected to multiple inter-campus VLANs, and a given sliver typically only includes one cross-connect port. FOAM admins should be careful not to approve rspecs that include both cross-connect ports (either explicitly or implicitly, e.g. by requesting the entire local DPID for VLAN 1750).

Some experimenters may have good reasons for wanting to use both cross-connects; if they do, make sure they understand that this is a risky topology before you approve their sliver.

Rspecs that include match rules with no protocol specified

Rspecs that include match rules at one layer typically need to include match rules specifying the protocol at a lower layer. For example:

  • An rspec that specifies a layer 3 match, like nw_src=A.B.C.D/N, also needs to specify dl_type to indicate what the layer 3 protocol is (e.g. dl_type=0x800 for IP and/or 0x806 for ARP).
  • An rspec that specifies a layer 4 match, like tp_port=N, also needs to specify nw_proto to indicate what the layer 4 protocol is (e.g. nw_proto=1 for ICMP, or 6 for TCP, or 17 for UDP).

The important thing is that if the protocol isn't specified, some switches will end up with flowtable entries that match more traffic than they should, interfering with other experiments at your site. FOAM admins should be careful not to approve rspecs that include a match at one layer without specifying the protocol.

We don't currently think that there's a valid use case for an experimenter wanting to specify a match at one layer without specifying the protocol at a lower layer, but will amend this advisory if we encounter one.

Switching from Expedient to FOAM

https://openflow.stanford.edu/display/DOCS/Install+Guide is the official FOAM installation guide, and following those steps will generally work. Before you get started, here are some specific notes for mesoscale campuses who are migrating from Expedient.

As an overview, this will involve:

  • identifying where you want to run FOAM, and its FlowVisor
  • choosing a site tag
  • configuring e-mail
  • disabling Expedient
  • informing experimenters@geni.net and response-team@geni.net about the change
  • helping experimenters migrate orphaned Expedient slivers

You should be able to run FOAM on the same system where you're currently running Expedient, if you'd like. Conversely, if you'd like to switch to a different system, the migration is an opportune time to do that.

We recommend having a dedicated FOAM-controlled FlowVisor; this isn't strictly necessary, but it keeps things pleasantly simple. We recommend using the same FlowVisor that Expedient was using, disabling Expedient once FOAM is running, and migrating orphaned Expedient slivers to FOAM, which allows experiments with existing slivers to keep running while you upgrade.

FlowVisor and FOAM can run on the same host, which should ideally have at least 3 GB of RAM and at least two CPUs. Fast disks also help, as FlowVisor (as of 0.8.1) can be I/O intensive. These requirements may increase for larger scale deployments.

The binary package works well with the current Long Term Stable release of Ubuntu (10.04), and we recommend that if you don't have another preference. We're working on creating RPMs, for sites who prefer Red Hat type distributions.

You can choose whatever you'd like as your site tag; we recommend something that looks more like a hostname (e.g. "foam.gpolab.bbn.com") than like a single word (e.g. "BBN"). Using the fully-qualified domain name of your FOAM server is convenient, as it allows you to use a different site tag on different servers, if you have more than one (e.g. one for testing purposes). You can also use a CNAME; for example, the GPO Lab FOAM server's hostname is tulum.gpolab.bbn.com, but we use foam.gpolab.bbn.com as our site tag (and then other tags for staging and testing instances).

FOAM will send e-mail to experimenters and to admins when various things happen: When a sliver is created, approved, disabled, rejected, or deleted; and when a sliver is within a week of expiring, a day of expiring, and actually expires. You'll also get a message once a day with a list of slivers that are currently in the pending queue.

We recommend using an e-mail address for the "From" setting, at installation time, which actually receives mail, so that experimenters can reach you by replying to messages from FOAM, if they need help. You don't need to set Reply-to (just hit return when prompted) unless you specifically want replies from experimenters to go somewhere other than the address in the From line. We use foam-admin@gpolab.bbn.com for both "From" and "Admin email", and don't set reply-to.

You can customize the text of the messages that FOAM sends: The default templates are in /opt/foam/etc/templates/default, and if you create a file in /opt/foam/etc/templates/custom with the same name as one of the files in .../default, the one in .../custom will take precedence. The files in .../default will be replaced when you install new versions of FOAM; anything you create in .../custom will persist through upgrades.

We recommend disabling Expedient once FOAM is up and running, to avoid confusion. This is how we removed it in the GPO Lab:

sudo apt-get remove python-optin-manager python-expedient apache2
sudo apt-get autoremove
sudo dpkg --purge apache2.2-common python-expedient
sudo rm -rf /usr/lib/python2.6/dist-packages/{openflow,geni,expedient,sfa,expedient_geni} /etc/{expedient,optin_manager} /etc/apt/sources.list.d/expedient.list

You should drop a note to experimenters@geni.net and response-team@geni.net once you're done, to inform others of the change.

If you had any existing Expedient slivers, you should contact those sliver owners about migrating to FOAM; see below for more about that.

If you have any trouble migrating, or suggestions for how to improve these docs, just drop a note to gpo-infra@bbn.com, or stop by the foam@conference.j.ir.bbn.com Jabber channel.

If you encounter any bugs, or have improvement or feature requests, https://openflow.stanford.edu/bugs/browse/FOAM is the JIRA issue tracker for FOAM. (There's also one for FlowVisor, and other Stanford OpenFlow projects.)

Managing FOAM slivers

https://openflow.stanford.edu/display/DOCS/foamctl+Guide is the official guide to foamctl, and describes in detail everything that it can do. Here are some specific commands that we've found useful for performing common tasks.

These commands all assume that you're running them on the FOAM server, and that you have a file /opt/foam/etc/foampasswd, containing the FOAM admin password.

Get a list of slivers

Pending ones:

foamctl list-slivers -s Pending --passwd-file=/opt/foam/etc/foampasswd

All active ones:

foamctl list-slivers --passwd-file=/opt/foam/etc/foampasswd

Either of these will give you a sliver URN; if you do

sliver_urn=urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbsstghosts:678fc69b-76e1-4a50-9fb2-ab5c4a5298d6

(with the actual URN of course), the rest of these commands will then work as-is.

Deleted ones:

foamctl list-slivers -d --passwd-file=/opt/foam/etc/foampasswd

Find a sliver from a slice name

If you know a user's slice name, you can grep for it:

foamctl list-slivers --passwd-file=/opt/foam/etc/foampasswd | egrep sliver_urn.+exampleslice

You can use this to get a sliver URN and/or an FV slice name from a GENI slice name, assigned to $sliver_urn and $flowvisor_slice:

slicename=exampleslice ; sliver_urn=$(foamctl list-slivers --passwd-file=/opt/foam/etc/foampasswd | egrep sliver_urn.+$slicename | sed -e 's/ *"sliver_urn": "\(.*\)".*/\1/') ; flowvisor_slice=$(echo $sliver_urn | awk -F : '{print $NF}')

The rest of these commands assume that you've used that (or something similar) to set $sliver_urn.

Show a sliver's basic info

foamctl show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

Show a sliver's rspec

foamctl show-sliver -r -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

Show a sliver's flowspec

foamctl show-sliver -s -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

Show a sliver's flowspace

foamctl show-sliver -f -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

Approve a sliver

This marks a sliver in FOAM as Approved, and adds a FV slice and flowspace rules for it to the FlowVisor.

foamctl approve-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

Disable a sliver

This marks a sliver in FOAM as Pending, and removes a FV slice and flowspace rules for it from the FlowVisor.

foamctl disable-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

Reject a sliver

This marks a sliver in FOAM as Rejected, and removes a FV slice and flowspace rules for it from the FlowVisor.

foamctl reject-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

Delete a sliver

This disables a sliver, and marks it as deleted, just like the GENI AM API DeleteSliver call.

foamctl delete-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

You should generally only do this with the experimenter's permission, and if the experimenter can't delete their own sliver for some reason, so they're not confused about where their sliver went. (If you disable or reject it, they can still see it; if you delete it, it's essentially gone forever from their point of view.)

Moving orphaned Expedient-created FV slices to FOAM slivers

If you've shut down Expedient and brought up FOAM, you may have FlowVisor slices that were created by Expedient, which are now essentially orphaned. For each of those FV slices, its owner should create an OpenFlow v3 rspec (the format FOAM uses), and create a new sliver in FOAM. You can then verify that the new sliver looks right, manually delete their old sliver, and approve the new one. (If a sliver owner needs help writing their new rspec, feel free to encourage them to contact help@geni.net.)

Here's an example, showing how we migrated the jbs15 and jbs16 slivers at BBN, after the owner had been contacted about the change, and created their new slivers in FOAM.

Check to see that the new slivers are there in FOAM, awaiting approval:

+$ foamctl list-slivers -s Pending --passwd-file=/opt/foam/etc/foampasswd
{
 "slivers": [
  {
   "status": "Pending", 
   "flowvisor_slice": "8d32974c-5a1b-4ebf-8c5f-097c9c64cf8d", 
   "deleted": "False", 
   "slice_urn": "urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs15", 
   "enabled": false, 
   "id": 1, 
   "expiration": "2011-10-19 02:56:32.798032", 
   "sliver_urn": "urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs15:8d32974c-5a1b-4ebf-8c5f-097c9c64cf8d", 
   "email": "jbs@bbn.com", 
   "desc": "The controller on naxos:33015, for jbs15."
  }, 
  {
   "status": "Pending", 
   "flowvisor_slice": "d82dae58-5de5-4caa-b458-46ee130462d0", 
   "deleted": "False", 
   "slice_urn": "urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs16", 
   "enabled": false, 
   "id": 2, 
   "expiration": "2011-10-19 02:56:51.263455", 
   "sliver_urn": "urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs16:d82dae58-5de5-4caa-b458-46ee130462d0", 
   "email": "jbs@bbn.com", 
   "desc": "The controller on naxos:33016, for jbs16."
  }
 ]
}

Identify the Expedient-created FV slices:

+$ fvctl --passwd-file=/etc/flowvisor/fvpasswd listSlices | grep jbs15
Slice 8: jbs15-naxos-33015_ID__tulum_gpolab_bbn_com_133

+$ fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo jbs15-naxos-33015_ID__tulum_gpolab_bbn_com_133
Got reply:
connection_2=06:d6:00:24:a8:c4:b9:00-->/192.1.249.23:45621-->naxos.gpolab.bbn.com/192.1.249.133:33015
connection_1=06:d6:00:12:e2:b8:a5:d0-->/192.1.249.23:45563-->naxos.gpolab.bbn.com/192.1.249.133:33015
contact_email=jbs@bbn.com
controller_hostname=naxos.gpolab.bbn.com
controller_port=33015
creator=fvadmin

+$ fvctl --passwd-file=/etc/flowvisor/fvpasswd listSlices | grep jbs16
Slice 16: jbs16-naxos-33016_ID__tulum_gpolab_bbn_com_106

+$ fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo jbs16-naxos-33016_ID__tulum_gpolab_bbn_com_106
Got reply:
connection_2=06:d6:00:24:a8:c4:b9:00-->/192.1.249.23:36076-->naxos.gpolab.bbn.com/192.1.249.133:33016
connection_1=06:d6:00:12:e2:b8:a5:d0-->/192.1.249.23:36018-->naxos.gpolab.bbn.com/192.1.249.133:33016
contact_email=jbs@bbn.com
controller_hostname=naxos.gpolab.bbn.com
controller_port=33016
creator=fvadmin

Delete those:

+$ fvctl --passwd-file=/etc/flowvisor/fvpasswd deleteSlice jbs16-naxos-33016_ID__tulum_gpolab_bbn_com_106
success!

+$ fvctl --passwd-file=/etc/flowvisor/fvpasswd deleteSlice jbs15-naxos-33015_ID__tulum_gpolab_bbn_com_133
success!

Approve the new FOAM slivers:

+$ slicename=jbs15 ; sliver_urn=$(foamctl list-slivers --passwd-file=/opt/foam/etc/foampasswd | egrep sliver_urn.+$slicename | sed -e 's/ *"sliver_urn": "\(.*\)".*/\1/') ; flowvisor_slice=$(echo $sliver_urn | awk -F : '{print $NF}')
+$ foamctl approve-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

+$ slicename=jbs16 ; sliver_urn=$(foamctl list-slivers --passwd-file=/opt/foam/etc/foampasswd | egrep sliver_urn.+$slicename | sed -e 's/ *"sliver_urn": "\(.*\)".*/\1/') ; flowvisor_slice=$(echo $sliver_urn | awk -F : '{print $NF}')
+$ foamctl approve-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

And voila.

Slice Authority trust configuration

You may want to configure FOAM to trust user certificates signed by additional Slice Authorities. To do that, install the CA cert for the Slice Authority in a file in /opt/foam/etc/gcf-ca-certs, and then rebuild the nginx CA cert bundle and restart FOAM and nginx:

sudo foamctl bundle-certs
sudo service foam restart
sudo service nginx restart

In particular, GENI mesoscale deployments should trust the pgeni.gpolab.bbn.com SA; the official FOAM installation guide includes this step, or you can get the cert from http://www.pgeni.gpolab.bbn.com/ca-cert/pgeni.gpolab.bbn.com.pem if you need it.