[[PageOutline]] = !FlowVisor = The !FlowVisor is a special purpose OpenFlow controller that acts as a transparent proxy between OpenFlow switches and multiple OpenFlow controllers. (https://github.com/OPENNETWORKINGLAB/flowvisor/wiki has more information about !FlowVisor (from the official site at ONL), including [https://github.com/OPENNETWORKINGLAB/flowvisor/wiki/Installation-from-Binary installation instructions], [https://github.com/OPENNETWORKINGLAB/flowvisor/wiki/FAQ a FAQ] with common error messages (for both experimenters and admins), etc. = Info for experimenters = The following sections are mostly of interest to GENI experimenters. == !FlowVisor and ARP == !FlowVisor can slice ARP traffic, but there are some caveats to be aware of. The [https://github.com/OPENNETWORKINGLAB/flowvisor/wiki/FAQ FlowVisor FAQ] has more details. = Info for admins = The following sections are mostly of interest to !FlowVisor admins. == Issue reporting == If you have a !FlowVisor issue, such as an unexpected FV crash, we'd like to work with you and ONL to try to identify the cause and resolve it. First, send mail to flowvisor@onlab.us and gpo-infra@geni.net, so we can discuss and research the root cause of the outage. If the issue was a !FlowVisor crash, include any FV-related lines from syslog at the time of the crash, and the contents of /var/log/flowvisor/flowvisor-stderr.log. Once the cause is identified, make a !FlowVisor ticket to track the issue; https://github.com/OPENNETWORKINGLAB/flowvisor/wiki/Filing-New-Issues-or-bugs describes how to do that most effectively. ONL will then prioritize and address the ticket accordingly, and work towards a resolution. You can also, of course, make tickets to track other !FlowVisor issues, bugs, feature requests, etc. == System requirements == Existing sites are running their !FlowVisor on a variety of hardware (including in VMs); ONL currently recommends that a !FlowVisor system should have at least 3 GB of RAM and at least two CPUs. These requirements may increase for larger scale deployments. It's also advantageous to have a robust control network interface to !FlowVisor. ONL currently recommends two gigabit interfaces, one to communicate with switches ("downstream") and one to communicate with controllers ("upstream"), if your network design permits. If your upstream and downstream traffic need to share an interface, it might be prudent to have a 10 GB NIC if possible. == Version == The GPO had recommended version 1.0.8 for GENI sites, but we've since found some showstopper issues with it, and are not recommending uprading at this time to sites running 0.8.17. Sites that haven't yet upgraded are mostly running 0.8.17; existing GENI racks may be running older versions of 0.8. === GENI deployment process === The GPO generally aims to have a scheduled deployment of a new version of !FlowVisor immediately after GEC. There may also be unscheduled deployments of bugfix versions between GECs. Here's an outline of the process for scheduled deployments, with timeframes (relative to "ask sites to upgrade") for when to start each step: * T minus 12 weeks: Evaluate possible next target versions. * T minus 11 weeks: Select the next target version down to the minor release number (henceforth "Version X.Y"), start basic "does it work at all" testing. * T minus 10 weeks: Go/no-go decision about Version X.Y; make tickets for any known issues. (If "no-go", select a new Version X.Y, do basic testing, etc.) * T minus 9 weeks: Install and test at BBN on dev systems; make tickets for any issues. * T minus 6 weeks: Go/no-go decision about Version X.Y: * If "go", select the next target version down to the maintenance release number (henceforth "Version X.Y.z"), and install Version X.Y.z on staging and then production at BBN. * If "no-go", select a new Version X.Y, install and test at BBN, etc -- two weeks until final go/no-go. * T minus 4 weeks: Final go/no-go decision. * T minus 1 week: At GEC, announce Version X.Y.z as the new GENI-recommended version. * T: Ask sites to upgrade to Version X.Y.z; follow up with sites as needed. Here's an outline of the process for unscheduled deployments: * Install and test at BBN on dev systems; make tickets for any issues. * When there are no remaining issues, install and test on staging and then production at BBN. * When we're confident that there aren't any previously-undetected issues, announce the new version as the new GENI-recommended version. * Ask sites to upgrade; follow up with sites as needed. It's much the same as the scheduled deployment process, but without the steps related to selecting a version, since this process only applies to an already-identified bugfix version (typically something that fixes bugs that are sufficiently critical that they can't wait until the next scheduled deployment). == !FlowVisor APIs == As of FV 1.0, there are two APIs for talking to !FlowVisor: The old XMLRPC API, and the new JSON API. Since FOAM, ORCA, and other various GENI things only speak the XMLRPC API, we're continuing to use it, even though it's officially deprecated. There are two `fvctl` programs that come with !FlowVisor, one for each API, in `/usr/bin/fvctl-xml` and `/usr/bin/fvctl-json`, which have very different input and output formats. The package also comes with a symlink in `/usr/bin/fvctl` that points to `fvctl-json` by default; on GENI sites, we recommend changing it to point to `fvctl-xml`, like so: {{{ sudo rm /usr/bin/fvctl sudo ln -s fvctl-xml /usr/bin/fvctl sudo sed -i -e 's/echo "Warning: The XMLRPC interface is deprecated/#echo "Warning: The XMLRPC interface is deprecated/' /usr/bin/fvctl-xml }}} (That last line is just to remove the warning about the XMLRPC interface being deprecated; we know, and don't need a reminder every time we run fvctl.) The rest of this page assumes that you've done this, and that `fvctl` runs the XML version of fvctl. == Initial configuration == The default configuration of !FlowVisor is generally suitable for GENI sites, except that we recommend the XMLRPC API for now. (See above for more info, including commands to make `/usr/bin/fvctl` point to the XML version.) === Admin password === We typically put the !FlowVisor fvadmin password into /etc/flowvisor.passwd, so you don't have to type it every time you run an fvctl command. Make sure that it's only readable by people who should have full admin access to !FlowVisor on your system! (World-readable might be fine if only FV admins have accounts on your FV server; or you might want to make it only group-readable by a group that includes the FV admins.) The examples on the rest of this page assume that you've done that. One easy way to generate a fairly secure password: {{{ sudo apt-get install pwgen test -f /etc/flowvisor.passwd || sudo sh -c 'pwgen -sB 24 > /etc/flowvisor.passwd' sudo apt-get remove pwgen }}} (And then adjust the permissions on /etc/flowvisor.passwd as necessary.) The rest of this page assumes that you've done that. === Version holding === We also recommend telling your package management system to hold !FlowVisor at its current version, so that if you do a general upgrade of all of the packages on your system, it won't get updated along with that, but will only update when you explicitly say so. To do that on systems that use dpkg type packages: {{{ echo "flowvisor hold" | sudo dpkg --set-selections }}} To do that on systems that use RPM type packages: {{{ sudo sed -i -e 's/enabled=1/enabled=0/' /etc/yum.repos.d/onlab.repo }}} (To re-enable the repo when you really do want to upgrade, just change back to `enabled=1`.) We also recommend [wiki:OpenFlow/FOAM#Versionholding doing this for FOAM]. == Backups == We recommend backing up your !FlowVisor database, which includes both configuration information and non-transient FV state (FV slices, flowspace rules, etc). === Backup === There's an fvctl command that saves a JSON version of the DB: {{{ fvctl --passwd-file=/etc/flowvisor.passwd dumpConfig config.json }}} You can then back up this file however you back up other plain files at your site. === Restore === NOTE that fvconfig can only be run when !FlowVisor is '''not running'''. NOTE that this completely and irrevocably overwrites your existing !FlowVisor database! Use with caution. You need to run the restore as the 'flowvisor' user, since that user has the right permissions to create the DB files: {{{ sudo -u flowvisor fvconfig load config.json }}} The last argument is the input file to use, and can be any file name as long as it's readable by the 'flowvisor' user. This is generally sufficient, but if your database is in a very bad state, you may need to delete it completely and re-create it before you can load the JSON file: {{{ sudo rm -rf /usr/share/db/flowvisor/FlowVisorDB sudo -u flowvisor fvconfig generate /dev/null sudo -u flowvisor fvconfig load config.json }}} In that second step, just hit return when prompted for a password (this will be overwritten by the next step); and ignore the "keytool error" error, it's just telling you that you already have a key pair. == Upgrading == Here are some before-and-after tests you can do when you upgrade !FlowVisor, to confirm that your state hasn't changed unexpectedly. First, get a "before" dump of the flowspace and slice info: {{{ cd rm -rf ~/tmp/flowvisor-upgrade mkdir -p ~/tmp/flowvisor-upgrade/before cd ~/tmp/flowvisor-upgrade/before fvctl --passwd-file=/etc/flowvisor.passwd dumpConfig config.json fvctl --passwd-file=/etc/flowvisor.passwd listFlowSpace | sed -re 's/id=\[[0-9\-]+\]/[id number stripped]/' > flowspace for slice in $(fvctl --passwd-file=/etc/flowvisor.passwd listSlices | awk '{print $3}' | sort); do echo "Info for slice: $slice" ; fvctl --passwd-file=/etc/flowvisor.passwd getSliceInfo $slice | egrep -v ^connection_[0-9]+= | sort ; done > sliceinfo }}} Next do the upgrade, as usual. For simple upgrades, on systems that use dpkg type packages: {{{ sudo apt-get update sudo service flowvisor stop sudo apt-get -y install flowvisor=1.0.8-1 sudo service flowvisor start }}} On systems that use RPM type packages: {{{ sudo service flowvisor stop sudo yum -y install flowvisor-1.0.8-1 sudo service flowvisor start }}} More complex upgrades might have additional steps, which will generally be described on a separate page, in the announcement of the upgrade, and/or in the release notes. Then make sure `/usr/bin/fvctl` points to `fvctl-xml`: {{{ sudo ln -f -s fvctl-xml /usr/bin/fvctl sudo sed -i -e 's/echo "Warning: The XMLRPC interface is deprecated/#echo "Warning: The XMLRPC interface is deprecated/' /usr/bin/fvctl-xml }}} (That last line is just to remove the warning about the XMLRPC interface being deprecated; we know, and don't need a reminder every time we run fvctl.) Then, get an "after" dump of the flowspace and slice info: {{{ cd rm -rf ~/tmp/flowvisor-upgrade/after mkdir -p ~/tmp/flowvisor-upgrade/after cd ~/tmp/flowvisor-upgrade/after fvctl --passwd-file=/etc/flowvisor.passwd dumpConfig config.json fvctl --passwd-file=/etc/flowvisor.passwd listFlowSpace | sed -re 's/id=\[[0-9\-]+\]/[id number stripped]/' > flowspace for slice in $(fvctl --passwd-file=/etc/flowvisor.passwd listSlices | awk '{print $3}' | sort); do echo "Info for slice: $slice" ; fvctl --passwd-file=/etc/flowvisor.passwd getSliceInfo $slice | egrep -v ^connection_[0-9]+= | sort ; done > sliceinfo }}} Finally, diff the results against the "before" results: {{{ cd ~/tmp/flowvisor-upgrade/before for file in * ; do diff -u $file ../after/$file ; done }}} There generally shouldn't be any diffs. After that, a good way to test that things are working properly is to do some FOAM operations (create or delete or approve or disable some slivers) on the FOAM server that manages that !FlowVisor. (Note that when upgrading from FV 0.8 to FV 1.0, the config.json file will show a great many diffs, because the output is aesthetically different, e.g. things appear in a different order, are indented differently, etc. If the flowspace and sliceinfo files don't show any differences, that's good enough.) == Useful commands == Here are some useful fvctl and fvconfig commands; run these on the system that runs the !FlowVisor you want to configure, with the fvadmin password in /etc/flowvisor.passwd. NOTE that fvctl can only be run when !FlowVisor is '''running''', and fvconfig can only be run when !FlowVisor is '''not running'''. === Dump the database to a JSON file === {{{ fvctl --passwd-file=/etc/flowvisor.passwd dumpConfig /tmp/config.json }}} This creates a JSON version of the !FlowVisor DB, including state (like slice and flowspace rule information) and configuration settings. The last argument is the output file to use, and can be something other than /tmp/config.json, as long as it's writable by the 'flowvisor' user. === Load a JSON file into the DB === NOTE that fvconfig can only be run when !FlowVisor is '''not running'''. NOTE that this completely and irrevocably overwrites your existing !FlowVisor database! Use with caution. {{{ sudo -u flowvisor fvconfig load /tmp/config.json }}} The last argument is the input file to use, and can be something other than /tmp/config.json, as long as it's readable by the 'flowvisor' user. === Change a configuration parameter === Most configuration parameters can only be changed by dumping the DB, editing the resulting JSON file, and loading the changed file into the DB, which requires you to stop and restart !FlowVisor, causing an outage. https://github.com/OPENNETWORKINGLAB/flowvisor/issues/212 and https://github.com/OPENNETWORKINGLAB/flowvisor/issues/213 are tracking the task of making it possible to get and set all configuration parameters via fvctl. Meanwhile, to (for example) turn on checkpointing, you can do this sequence: {{{ fvctl --passwd-file=/etc/flowvisor.passwd dumpConfig /tmp/config.json sudo service flowvisor stop sudo -u flowvisor sed -i -e 's/"checkpointing": false/"checkpointing": true/' /tmp/config.json sudo -u flowvisor fvconfig load /tmp/config.json sudo service flowvisor start }}} (We don't actually recommend that you turn on checkpointing, but if you wanted to, this is how you'd do it.) === Show the flowspace === {{{ fvctl --passwd-file=/etc/flowvisor.passwd listFlowSpace }}} === Show the slices === {{{ fvctl --passwd-file=/etc/flowvisor.passwd listSlices }}} === Get info about a slice === {{{ fvctl --passwd-file=/etc/flowvisor.passwd getSliceInfo default }}} Replace 'default' with the slice you want info about. === Show the controller for every slice === {{{ for slice in $(fvctl --passwd-file=/etc/flowvisor.passwd listSlices 2>&1 | grep Slice | grep -v fvadmin | awk '{ print $3; }') ; do echo $slice ; fvctl --passwd-file=/etc/flowvisor.passwd getSliceInfo $slice | grep controller ; done }}} === Remove all existing flowspace entries === {{{ for id in $(fvctl --passwd-file=/etc/flowvisor.passwd listFlowSpace 2>&1 | sed -e 's/rule.*id=\[\([0-9]*\)\].*/\1/' | egrep ^[0-9]*$) ; do fvctl --passwd-file=/etc/flowvisor.passwd removeFlowSpace $id ; done }}} === Remove flowspace entries matching a pattern === The first of the two 'for' lines just echoes what it's going to do, the second actually does it. {{{ fvpattern=00:0c:29:82:59:5b for id in $(fvctl --passwd-file=/etc/flowvisor.passwd listFlowSpace | egrep -i $fvpattern 2>&1 | sed -e 's/rule.*id=\[\([0-9]*\)\].*/\1/' | egrep ^[0-9]*$) ; do echo fvctl --passwd-file=/etc/flowvisor.passwd removeFlowSpace $id ; done for id in $(fvctl --passwd-file=/etc/flowvisor.passwd listFlowSpace | egrep -i $fvpattern 2>&1 | sed -e 's/rule.*id=\[\([0-9]*\)\].*/\1/' | egrep ^[0-9]*$) ; do fvctl --passwd-file=/etc/flowvisor.passwd removeFlowSpace $id ; done }}} Replace the pattern after "fvpattern=" with a regexp that you want to match. === Remove all existing slices === {{{ for slice in $(fvctl --passwd-file=/etc/flowvisor.passwd listSlices 2>&1 | grep Slice | grep -v fvadmin | awk '{ print $3; }') ; do fvctl --passwd-file=/etc/flowvisor.passwd deleteSlice $slice ; done }}} === Create one slice and one flowspace rule === {{{ fvctl --passwd-file=/etc/flowvisor.passwd createSlice default tcp:localhost:36633 user@example.net fvctl --passwd-file=/etc/flowvisor.passwd addFlowSpace all 1000 any "Slice:default=4" }}} You'll need to enter a password for the slice; make it something secure, but nothing ever uses it, so don't worry about recording it.