wiki:OpenFlow/FOAM/Testing

Version 26 (modified by Josh Smift, 7 years ago) (diff)

--

Testing FOAM

NOTE that all the foamctl commands on this page now use the new 0.8 syntax. If you want an older version, with the 0.6 syntax, try http://groups.geni.net/geni/wiki/OpenFlow/FOAM/Testing?version=6 instead.

This page describes a couple of ways to test FOAM:

  • Testing that slivers can be created, and then managed in various ways, including:
    • Simple tests that slivers appear and disappear in FlowVisor.
    • More complex tests for other sliver management operations.
    • Tests for sliver expiration.
  • Testing auto-approval, in the various approve-on-creation modes.
  • Testing that the software complies with the GENI AM API, using the test suite that comes with GCF.

Sliver creation and management

This section contains multiple tests, and includes a Setup subsection to set some variables, a Creation subsection to create a sliver, and then additional subsections to do various tests with it.

You only need to do the Setup subsection once.

You need to repeat the Creation subsection before doing any of the other subsections, because the Creation subsection sets some variables that you'll need for the other subsections, and each of the other subsections concludes by deleting the sliver.

Pre-requisites/assumptions:

  • You're set up to use Omni in general, and have it in your path as 'omni', i.e. you've successfully used Omni to reserve GENI resources (and have the latest version of the software, a user credential signed by a CA that's trusted by the FOAM server you intend to test, a working omni_config file, etc).
  • You have a valid request rspec file to test with. We plan to make some sample rspecs available.
  • You can log in non-interactively to the FOAM and FV servers, and have admin privs there, and have the FOAM and fvadmin password stored in files (/opt/foam/etc/passwd and /etc/flowvisor/fvpasswd). (This generally can't be used as-is to test FOAM at other sites.)
  • You know what output to expect when common things succeed. (Expected output is described here only in cases where an error is the expected result, or where it otherwise might not be obvious; if you think the expected output isn't obvious for any given step, let us know.)
  • The URL of the FOAM server is also a name that you can use to log in to it. (Very likely, but might be untrue with virtualhosts or some other odd thing.)
  • On the FOAM server you're testing, when you create a sliver, it will be automatically approved. (More specifically, that geni.openflow.analysis-engine = True (analyze slivers for overlap), and geni.approval.approve-on-creation = 2 (approve slivers at creation time if they don't fail any overlap-analysis tests), and the test sliver's rspec describes a flowspace that will pass overlap analysis.)
  • On the FOAM server you're testing, when the 'www-data' user runs a cron job that has output, it sends mail to 'www-data', and you can receive this mail.
  • On the FOAM server you're testing, you have the foam-confirm-sliver, flowvisor-confirm-fvslice-exists, and flowvisor-confirm-flowspace-rule-count scripts. (These are currently GPO-only; GPO folks, they're in the syseng repo, under gposw.)

You can run most of these tests wherever you usually use Omni; they don't need to run on the FOAM/FV server, except as noted below.

Setup

Set the FOAM server name (modify this to use the DNS name of the server that you want to test):

foamserver=foam.example.edu

Set the name of the FlowVisor server to be the same (if it's the same, or something else if it isn't):

fvserver=$foamserver

Set the AM API URL:

am=https://${foamserver}:3626/foam/gapi/1

Identify the rspec you plan to use (modify this if you want to use a different rspec):

rspec=~/rspecs/request/foam-testing/$foamserver/openflow-mac-acedcodefood.rspec

Set the slicename to something with your username and "foamtest":

slicename=${USER}foamtest

Set the full slice URN (modify this if you're not using pgeni.gpolab.bbn.com as your Slice Authority), and a version of it for handing off to egrep later:

slice_urn=urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+$slicename
slice_urn_egrep=$(echo $slice_urn | sed -e 's/+/\\+'/g)

Make sure FOAM is listening at all:

omni -a $am getversion
omni -a $am listresources

Now you're ready to create a sliver.

Here's all of those commands in one block for easy of copy-and-paste:

foamserver=foam.example.edu

fvserver=$foamserver
am=https://${foamserver}:3626/foam/gapi/1
rspec=~/rspecs/request/foam-testing/$foamserver/openflow-mac-acedcodefood.rspec
slicename=${USER}foamtest
slice_urn=urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+$slicename
slice_urn_egrep=$(echo $slice_urn | sed -e 's/+/\\+'/g)
omni -a $am getversion
omni -a $am listresources

Creation

Create a slice:

omni createslice $slicename

Create a sliver:

omni -a $am createsliver $slicename $rspec

(That should generate a "sliver created" and a "sliver approved" e-mail message, one of each to the experimenter and one of each to the FOAM admin.)

Confirm that it looks right via Omni:

omni -a $am listresources $slicename
omni -a $am sliverstatus $slicename

Confirm that it shows up in FOAM:

ssh $foamserver foamctl geni:list-slivers --passwd-file=/opt/foam/etc/foampasswd | egrep slice_urn.+$slice_urn_egrep

Get the sliver URN and FlowVisor slicename from FOAM, and put them into $sliver_urn and $fv_slicename:

sliver_urn=$(ssh $foamserver foamctl geni:list-slivers --passwd-file=/opt/foam/etc/foampasswd | egrep sliver_urn.+$slice_urn_egrep | sed -e 's/ *"sliver_urn": "\(.*\)".*/\1/')
fv_slicename=$(echo $sliver_urn | awk -F : '{print $NF}')

Now you're ready to do additional tests with this sliver.

Simple management

NOTE: Do the Creation subsection immediately before starting this subsection.

NOTE: This subsection needs to be revamped to match the new assumption that when you create a sliver, it will be automatically approved.

The steps in this subsection look at a newly-created sliver, approve it, confirm that it shows up in the FlowVisor, delete it, and confirm that it vanishes from the FlowVisor.

Show the sliver, and confirm that 'status' is "Pending":

ssh $foamserver foamctl geni:show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

Confirm that the sliver doesn't yet show up in FlowVisor:

ssh $fvserver fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo $fv_slicename

(Expect a "slice does not exist" error.)

Show the sliver's rspec:

ssh $foamserver foamctl geni:show-sliver -r -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

Show the sliver's flowspec:

ssh $foamserver foamctl geni:show-sliver -s -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

Show the sliver's flowspace rules:

ssh $foamserver foamctl geni:show-sliver -f -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

Approve the sliver, and confirm that 'status' is now "Approved":

ssh $foamserver foamctl geni:approve-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
ssh $foamserver foamctl geni:show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

(That should generate a "sliver approved" e-mail message, one to the experimenter and one to the FOAM admin.)

Confirm that the sliver now shows up in FlowVisor, with two flowspace rules:

ssh $fvserver fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo $fv_slicename
ssh $fvserver fvctl --passwd-file=/etc/flowvisor/fvpasswd listFlowSpace | grep $fv_slicename

(Don't worry about the ouptut saying that it can't connect to the controller; this is expected if your rspec didn't include a controller that's actually running, which the example one doesn't.)

Delete the sliver:

omni -a $am deletesliver $slicename

(That should generate a "sliver deleted" e-mail message, one to the experimenter and one to the FOAM admin.)

Confirm that Omni thinks it's gone:

omni -a $am listresources $slicename
omni -a $am sliverstatus $slicename

(Expect an "unknown slice" error for each of those.)

Confirm that the sliver no longer shows up in FOAM:

ssh $foamserver foamctl geni:show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
ssh $foamserver foamctl geni:list-slivers --passwd-file=/opt/foam/etc/foampasswd | grep $slice_urn

(Expect 'deleted' to be "True" for the first, and no output for the second. For the first, note that 'status' will still be "Approved", which is fine, it's just tracking what the status was when it was deleted.)

Confirm that the sliver no longer shows up in FlowVisor:

ssh $fvserver fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo $fv_slicename
ssh $fvserver fvctl --passwd-file=/etc/flowvisor/fvpasswd listFlowSpace | grep $fv_slicename

(Expect a "slice does not exist" error for the first, and no output for the second.)

Complex management

NOTE: Do the Creation subsection immediately before starting this subsection.

The steps in this subsection look at a newly-created sliver, run through a sequence of approve/disable/reject actions to confirm that it can transition from any of those states to the others, and check the FlowVisor after each change; then delete the sliver, and confirm that it vanishes from the FlowVisor.

Confirm that the sliver was approved at creation time: Check that 'status' is "Approved", and that the sliver shows up in FlowVisor, with two flowspace rules:

ssh $foamserver foam-confirm-sliver $sliver_urn status approved
ssh $fvserver flowvisor-confirm-fvslice-exists $fv_slicename true
ssh $fvserver flowvisor-confirm-flowspace-rule-count $fv_slicename 2

Add a temporary cron job to tell FOAM to generate e-mail with a list of pending slivers:

ssh -t $foamserver "sudo sh -c \"echo $(date +'%M %H' -d 'now + 1 min') '* * *  www-data  /opt/foam/bin/daily-queue ; echo foamtest cron job ran at $(date)' > /etc/cron.d/foamtest\""

(Within a minute (when the cron job fires), that should generate a message to www-data saying that the cron job ran, and one of two other things will happen: (a) If there are *other* pending slivers, it should generate a message to the FOAM admin listing only those slivers, and not your test sliver; OR (b) If there are no other pending slivers, it should not send any message about pending slivers to the FOAM admin.)

Disable the sliver, to put it back into the pending queue:

ssh $foamserver foamctl geni:disable-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd

(That should generate a "sliver disabled" e-mail message, one to the experimenter and one to the FOAM admin.)

Confirm that 'status' is now "Pending", and that the sliver now does not show up in FlowVisor, and has no flowspace rules:

ssh $foamserver foam-confirm-sliver $sliver_urn status pending
ssh $fvserver flowvisor-confirm-fvslice-exists $fv_slicename false
ssh $fvserver flowvisor-confirm-flowspace-rule-count $fv_slicename 0

Add a temporary cron job to tell FOAM to generate e-mail with a list of pending slivers:

ssh -t $foamserver "sudo sh -c \"echo $(date +'%M %H' -d 'now + 1 min') '* * *  www-data  /opt/foam/bin/daily-queue ; echo foamtest cron job ran at $(date)' > /etc/cron.d/foamtest\""

(Within a minute (when the cron job fires), that should generate a message to www-data saying that the cron job ran; and e-mail to the FOAM admin with a list of pending slivers, including your test sliver.)

Clean up the cron job:

ssh -t $foamserver sudo rm /etc/cron.d/foamtest

This next set of tests will generate a bunch of e-mail; check at the end that it all showed up as expected.

Approve the sliver, confirm that 'status' is now "Approved", and confirm that the sliver now shows up in FlowVisor, with two flowspace rules:

ssh $foamserver foamctl geni:approve-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
ssh $foamserver foam-confirm-sliver $sliver_urn status approved
ssh $fvserver flowvisor-confirm-fvslice-exists $fv_slicename true
ssh $fvserver flowvisor-confirm-flowspace-rule-count $fv_slicename 2

Reject the sliver, confirm that 'status' is now "Rejected", and confirm that the sliver now does not show up in FlowVisor, and has no flowspace rules:

ssh $foamserver foamctl geni:reject-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
ssh $foamserver foam-confirm-sliver $sliver_urn status rejected
ssh $fvserver flowvisor-confirm-fvslice-exists $fv_slicename false
ssh $fvserver flowvisor-confirm-flowspace-rule-count $fv_slicename 0

Disable the sliver, confirm that 'status' is now "Pending", and confirm that the sliver now does not show up in FlowVisor, and has no flowspace rules:

ssh $foamserver foamctl geni:disable-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
ssh $foamserver foam-confirm-sliver $sliver_urn status pending
ssh $fvserver flowvisor-confirm-fvslice-exists $fv_slicename false
ssh $fvserver flowvisor-confirm-flowspace-rule-count $fv_slicename 0

Reject the sliver, confirm that 'status' is now "Rejected", and confirm that the sliver now does not show up in FlowVisor, and has no flowspace rules:

ssh $foamserver foamctl geni:reject-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
ssh $foamserver foam-confirm-sliver $sliver_urn status rejected
ssh $fvserver flowvisor-confirm-fvslice-exists $fv_slicename false
ssh $fvserver flowvisor-confirm-flowspace-rule-count $fv_slicename 0

Approve the sliver, confirm that 'status' is now "Approved", and confirm that the sliver now shows up in FlowVisor, with two flowspace rules:

ssh $foamserver foamctl geni:approve-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
ssh $foamserver foam-confirm-sliver $sliver_urn status approved
ssh $fvserver flowvisor-confirm-fvslice-exists $fv_slicename true
ssh $fvserver flowvisor-confirm-flowspace-rule-count $fv_slicename 2

Delete the sliver, confirm that 'deleted' is now "True", and confirm that the sliver now does not show up in FlowVisor, and has no flowspace rules:

ssh $foamserver foamctl geni:delete-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
ssh $foamserver foam-confirm-sliver $sliver_urn deleted true
ssh $fvserver flowvisor-confirm-fvslice-exists $fv_slicename false
ssh $fvserver flowvisor-confirm-flowspace-rule-count $fv_slicename 0

That set of tests should generate six e-mail messages to the experimenter and six to the FOAM admin: There should be a "sliver approved" message, then "sliver rejected", then "sliver disabled", then rejected, approved, and deleted. Probably in that order, although e-mail is sometimes delivered out of sequence, so don't worry too much if they're not in order.

Expiration

NOTE: Do the Creation subsection immediately before starting this subsection.

The steps in this subsection look at a newly-created sliver, and test expiration in FOAM, by changing its expiration date and, then running the scripts that expire slivers and notify about soon-to-expire ones.

Confirm that the sliver was approved at creation time: Check that 'status' is "Approved", and that the sliver shows up in FlowVisor, with two flowspace rules:

ssh $foamserver foam-confirm-sliver $sliver_urn status approved
ssh $fvserver flowvisor-confirm-fvslice-exists $fv_slicename true
ssh $fvserver flowvisor-confirm-flowspace-rule-count $fv_slicename 2

Now, do some tests to make sure FOAM notifies you about a sliver that will expire within a day, but only once.

Change the expiration date of your slice to be nine days away, since otherwise you can't extend the sliver expiration date very far:

omni renewslice $slicename "$(date -d 'now + 9 days')" -f gpolab

Change the expiration date of your sliver to be twelve hours away, and confirm that the sliver has the new expiration date:

omni -a $am renewsliver $slicename "$(date -d 'now + 12 hours')"
omni -a $am sliverstatus $slicename

(That should generate a "sliver renewed" e-mail message, one to the experimenter and one to the FOAM admin.)

Add a temporary cron job to tell FOAM to send mail about slivers that are expiring soon and remove any expired slivers:

ssh -t $foamserver "sudo sh -c \"echo $(date +'%M %H' -d 'now + 1 min') '* * *  www-data  /opt/foam/bin/expire ; /opt/foam/bin/expire-emails ; echo foamtest cron job ran at $(date)' > /etc/cron.d/foamtest\""

(Within a minute (when the cron job fires), that should generate a "sliver expiring shortly" e-mail message, one to the experimenter and one to the FOAM admin; no messages about expired slivers being deleted; and one message to www-data saying that the cron job ran.)

Add the cron job again, to confirm that it doesn't send a second notification:

ssh -t $foamserver "sudo sh -c \"echo $(date +'%M %H' -d 'now + 1 min') '* * *  www-data  /opt/foam/bin/expire ; /opt/foam/bin/expire-emails ; echo foamtest cron job ran at $(date)' > /etc/cron.d/foamtest\""

(Within a minute (when the cron job fires), that should generate no messages about slivers expiring in the future; no messages about expired slivers being deleted; and one message to www-data saying that the cron job ran.)

Next, test to make sure FOAM notifies you again if you change the expiration date.

Change the expiration date of your sliver to be one day away, and confirm that the sliver has the new expiration date:

omni -a $am renewsliver $slicename "$(date -d 'now + 1 day')"
omni -a $am sliverstatus $slicename

(That should generate a "sliver renewed" e-mail message, one to the experimenter and one to the FOAM admin.)

Add the cron job again, to confirm that it does notify you again now that the expiration date has changed:

ssh -t $foamserver "sudo sh -c \"echo $(date +'%M %H' -d 'now + 1 min') '* * *  www-data  /opt/foam/bin/expire ; /opt/foam/bin/expire-emails ; echo foamtest cron job ran at $(date)' > /etc/cron.d/foamtest\""

(Within a minute (when the cron job fires), that should generate a "sliver expiring shortly" e-mail message, one to the experimenter and one to the FOAM admin; no messages about expired slivers being deleted; and one message to www-data saying that the cron job ran.)

Next, do some tests to make sure that FOAM notifies you about a sliver that will expire within a week, but only once.

Change the expiration date of your sliver to be two days away, and confirm that the sliver has the new expiration date:

omni -a $am renewsliver $slicename "$(date -d 'now + 2 days')"
omni -a $am sliverstatus $slicename

(That should generate a "sliver renewed" e-mail message, one to the experimenter and one to the FOAM admin.)

Add the cron job again, to confirm that it now sends a "within a week" message:

ssh -t $foamserver "sudo sh -c \"echo $(date +'%M %H' -d 'now + 1 min') '* * *  www-data  /opt/foam/bin/expire ; /opt/foam/bin/expire-emails ; echo foamtest cron job ran at $(date)' > /etc/cron.d/foamtest\""

(Within a minute (when the cron job fires), that should generate a "sliver expiring in about a week" e-mail message, one to the experimenter and one to the FOAM admin; no messages about expired slivers being deleted; and one message to www-data saying that the cron job ran.)

Add the cron job again, to confirm that it doesn't send a second notification:

ssh -t $foamserver "sudo sh -c \"echo $(date +'%M %H' -d 'now + 1 min') '* * *  www-data  /opt/foam/bin/expire ; /opt/foam/bin/expire-emails ; echo foamtest cron job ran at $(date)' > /etc/cron.d/foamtest\""

(Within a minute (when the cron job fires), that should generate no messages about slivers expiring in the future; no messages about expired slivers being deleted; and one message to www-data saying that the cron job ran.)

Next, test to make sure FOAM doesn't notify you at all if your sliver is expiring more than a week from now.

Change the expiration date of your sliver to be eight days away, and confirm that the sliver has the new expiration date:

omni -a $am renewsliver $slicename "$(date -d 'now + 8 days')"
omni -a $am sliverstatus $slicename

(That should generate a "sliver renewed" e-mail message, one to the experimenter and one to the FOAM admin.)

Add the cron job again, to confirm that it doesn't send a notification:

ssh -t $foamserver "sudo sh -c \"echo $(date +'%M %H' -d 'now + 1 min') '* * *  www-data  /opt/foam/bin/expire ; /opt/foam/bin/expire-emails ; echo foamtest cron job ran at $(date)' > /etc/cron.d/foamtest\""

(Within a minute (when the cron job fires), that should generate no messages about slivers expiring in the future; no messages about expired slivers being deleted; and one message to www-data saying that the cron job ran.)

Finally, test to make sure FOAM deletes a sliver if its expiration date is in the past.

Change the expiration date of your sliver to be right now, and confirm that the sliver has the new expiration date:

omni -a $am renewsliver $slicename "$(date -d 'now')"
omni -a $am sliverstatus $slicename

(That should generate a "sliver renewed" e-mail message, one to the experimenter and one to the FOAM admin.)

Add the cron job again, to confirm that it deletes the sliver:

ssh -t $foamserver "sudo sh -c \"echo $(date +'%M %H' -d 'now + 1 min') '* * *  www-data  /opt/foam/bin/expire ; /opt/foam/bin/expire-emails ; echo foamtest cron job ran at $(date)' > /etc/cron.d/foamtest\""

(Within a minute (when the cron job fires), that should generate no messages about slivers expiring in the future; a "sliver expired" message, one to the experimenter and one to the FOAM admin; and one message to www-data saying that the cron job ran.)

Confirm that the sliver was deleted, does not show up in FlowVisor, and has no flowspace rules::

ssh $foamserver foam-confirm-sliver $sliver_urn deleted true
ssh $fvserver flowvisor-confirm-fvslice-exists $fv_slicename false
ssh $fvserver flowvisor-confirm-flowspace-rule-count $fv_slicename 0

Clean up the last cron job:

ssh -t $foamserver sudo rm /etc/cron.d/foamtest

Auto-approval

This section contains multiple tests, and includes a "General setup" subsection to set some variables, and then additional subsections to do various tests.

Each of the subsections corresponds to one of the approve-on-creation modes; you should run the tests in the section that corresponds to the approve-on-creation mode of the FOAM server that you're testing. (The "General setup" subsection includes a step for determining this, each subsection includes a step to double-check it.)

You only need to do the "General setup" subsection once.

Pre-requisites/assumptions:

  • You're set up to use Omni in general, and have it in your path as 'omni', i.e. you've successfully used Omni to reserve GENI resources (and have the latest version of the software, a user credential signed by a CA that's trusted by the FOAM server you intend to test, a working omni_config file, etc).
  • You have a set of valid request rspec files to test with. We plan to make some sample rspecs available.
  • You can log in non-interactively to the FOAM and FV servers, and have admin privs there, and have the FOAM and fvadmin password stored in files (/opt/foam/etc/passwd and /etc/flowvisor/fvpasswd). (This generally can't be used as-is to test FOAM at other sites.)
  • The URL of the FOAM server is also a name that you can use to log in to it. (Very likely, but might be untrue with virtualhosts or some other odd thing.)
  • On the FOAM server you're testing, you have the foam-confirm-sliver, flowvisor-confirm-fvslice-exists, and flowvisor-confirm-flowspace-rule-count scripts.
  • On the system where you're running commands, you have the foam-create-and-verify-sliver script.

Note that the scripts mentioned in the last two items are currently GPO-only. (GPO folks, they're in the syseng repo, under gposw.)

You can run these tests wherever you usually use Omni; as written, they use SSH to run remote commands on the FOAM/FV server.

General setup

Set the FOAM server name (modify this to use the DNS name of the server that you want to test):

foamserver=foam.example.edu

Set the name of the FlowVisor server to be the same (if it's the same, or something else if it isn't):

fvserver=$foamserver

Set the AM API URL:

am=https://${foamserver}:3626/foam/gapi/1

Identify the rspecs you plan to use (modify this if you want to use different rspecs), in several categories:

  • Four that specify subnets: The first two don't overlap, the third overlaps with one of the first two, the fourth overlaps with all of the first three.
  • Four that specify MAC addresses: The first two don't overlap, the third overlaps with one of the first two, the fourth overlaps with all of the first three.
  • Four that specify ethertypes: The first two don't overlap, the third overlaps with one of the first two, the fourth overlaps with all of the first three.
  • One that specifies a subnet, a MAC address, and an ethertype, each of which overlaps with one of the rspecs in the corresponding one of the three groups above.
  • One that includes a more complex flowspace: A subnet that doesn't overlap with anything else, but also an entire VLAN (which should not be auto-approved).
  • One that includes two ports that are in a you-can-only-have-one-of-these port group. (FIXME: What does FOAM actually call this? And for that matter, how do you set it?)
rspec_10_3_5_0_24=~/rspecs/request/foam-testing/$foamserver/openflow-ip-10-3-5-0-24.rspec
rspec_10_7_9_0_24=~/rspecs/request/foam-testing/$foamserver/openflow-ip-10-7-9-0-24.rspec
rspec_10_3_0_0_16=~/rspecs/request/foam-testing/$foamserver/openflow-ip-10-3-0-0-16.rspec
rspec_10_0_0_0_12=~/rspecs/request/foam-testing/$foamserver/openflow-ip-10-0-0-0-12.rspec

rspec_cafebeefface=~/rspecs/request/foam-testing/$foamserver/openflow-mac-cafebeefface.rspec
rspec_deadbeadfeed=~/rspecs/request/foam-testing/$foamserver/openflow-mac-deadbeadfeed.rspec
rspec_aced_and_cafe=~/rspecs/request/foam-testing/$foamserver/openflow-mac-aced-and-cafe.rspec
rspec_aced_and_cafe_and_dead=~/rspecs/request/foam-testing/$foamserver/openflow-mac-aced-and-cafe-and-dead.rspec

rspec_ethertype_3171=~/rspecs/request/foam-testing/$foamserver/openflow-ethertype-3171.rspec
rspec_ethertype_3172=~/rspecs/request/foam-testing/$foamserver/openflow-ethertype-3172.rspec
rspec_ethertype_3173_and_3171=~/rspecs/request/foam-testing/$foamserver/openflow-ethertype-3173-and-3171.rspec
rspec_ethertype_3173_and_3172_and_3171=~/rspecs/request/foam-testing/$foamserver/openflow-ethertype-3173-and-3172-and-3171.rspec

rspec_one_of_each=~/rspecs/request/foam-testing/$foamserver/openflow-one-each-subnet-mac-ethertype.rspec

rspec_dpid_aa_port_3_and_36=~/rspecs/request/foam-testing/$foamserver/openflow-dpid-aa-port-3-and-36.rspec
rspec_dpid_aa_port_3_and_11=~/rspecs/request/foam-testing/$foamserver/openflow-dpid-aa-port-3-and-11.rspec
rspec_dpid_aa_port_5_dpid_ff_port_21=~/rspecs/request/foam-testing/$foamserver/openflow-dpid-aa-port-5-dpid-ff-port-21.rspec
rspec_dpid_aa_port_9_dpid_ff_port_21=~/rspecs/request/foam-testing/$foamserver/openflow-dpid-aa-port-9-dpid-ff-port-21.rspec

rspec_subnet_and_vlan=~/rspecs/request/foam-testing/$foamserver/openflow-subnet-and-vlan.rspec

Set four slicenames, using something with your username and "foamtest":

slicename1=${USER}foamtest1
slicename2=${USER}foamtest2
slicename3=${USER}foamtest3
slicename4=${USER}foamtest4

Make sure FOAM is listening at all:

omni -a $am getversion
omni -a $am listresources

Check which approve-on-creation mode FOAM is currently set to use:

ssh $foamserver foamctl config:get-value --key="geni.approval.approve-on-creation" --passwd-file=/opt/foam/etc/foampasswd

The possible values are 2 for "analysis", 1 for "always", and 0 for "never". (You can change this if you want to test a different value, but you may not want to do that, e.g. if it's a server that anyone else is using.)

OPTIONALLY, turn off e-mail notification for creation, approval, and deletion:

ssh $foamserver foamctl config:set-value --key="email.event.createsliver.admin" --value="false" --passwd-file=/opt/foam/etc/foampasswd
ssh $foamserver foamctl config:set-value --key="email.event.approvesliver.admin" --value="false" --passwd-file=/opt/foam/etc/foampasswd
ssh $foamserver foamctl config:set-value --key="email.event.gapi-deletesliver.admin" --value="false" --passwd-file=/opt/foam/etc/foampasswd

If you do that, you may want to turn notification back on when you're done testing:

ssh $foamserver foamctl config:set-value --key="email.event.createsliver.admin" --value="true" --passwd-file=/opt/foam/etc/foampasswd
ssh $foamserver foamctl config:set-value --key="email.event.approvesliver.admin" --value="true" --passwd-file=/opt/foam/etc/foampasswd
ssh $foamserver foamctl config:set-value --key="email.event.gapi-deletesliver.admin" --value="true" --passwd-file=/opt/foam/etc/foampasswd

Now you're ready to run the tests in one or more of the following subections.

Approval mode 2 ("analysis")

This subsection tests the behavior of FOAM when new slivers are created and geni.approval.approve-on-creation is set to 2 (analysis). The expected behavior is that slivers will be automatically approved if they pass all of the checks in the analysis engine, i.e. they don't overlap with any existing slivers; and don't contain any flowspace that can't be handled by the analysis engine.

Double-check the FOAM server's current configuration:

ssh $foamserver foamctl config:get-value --key="geni.approval.approve-on-creation" --passwd-file=/opt/foam/etc/foampasswd

If the value isn't "2", then you shouldn't run this section on this FOAM server. (You can first change the configuration of this FOAM server, but you may not want to do that, e.g. if it's a server that anyone else is using.)

You can go through all of these tests in sequence; they're further divided into sets of tests for convenience.

Four slivers

This set of tests uses four rspecs that specify a single feature (IP subnet, MAC address, or ethertype). In each set of four rspecs, the first two (A and B) don't overlap, the third (C) overlaps with one of the first two (A), and the fourth (D) overlaps with all of the first three.

Each test (one per block below) creates the slivers in a different order, to confirm that the anlysis engine detects overlap regardless of whether a more specific or less specific sliver is created first. Each test deletes any lingering slivers before running, just to make sure the slate is clean. Each test should generate four "sliver created" e-mail messages, plus one "sliver approved" message per sliver that's expected to be automatically approved; and also four "sliver deleted" messages if any slivers were there to be deleted.

These tests attempt to verify these principles:

  • Whichever of A and C gets created first should be approved, and will then block the other.
  • B should always be approved as long as it's created before D, since it isn't blocked by either A or C.
  • D should only be approved if it's created first, because otherwise it's blocked by any of A, B, or C; and if it's created first, it blocks all of them.
  • If something would block other things if it was approved, it doesn't block those things if it's pending.

There's a "Four-sliver setup" section to set the A/B/C/D variables, and then a tests section to run the tests. You can re-run the "Four-sliver setup" section, and then re-run the tests section, e.g. to test each of the three features.

Four-sliver setup

Do ONE of these, depending on whether you want to test IP, MAC, or ethertype, to define which rspecs to use, and how many FlowVisor flowspace rules to expect if that rspec is approved.

IP:

rspec_A=$rspec_10_3_5_0_24
rspec_B=$rspec_10_7_9_0_24
rspec_C=$rspec_10_3_0_0_16
rspec_D=$rspec_10_0_0_0_12

rules_A=2
rules_B=2
rules_C=2
rules_D=2

MAC:

rspec_A=$rspec_cafebeefface
rspec_B=$rspec_deadbeadfeed
rspec_C=$rspec_aced_and_cafe
rspec_D=$rspec_aced_and_cafe_and_dead

rules_A=2
rules_B=2
rules_C=4
rules_D=6

Ethertype:

rspec_A=$rspec_ethertype_3171
rspec_B=$rspec_ethertype_3172
rspec_C=$rspec_ethertype_3173_and_3171
rspec_D=$rspec_ethertype_3173_and_3172_and_3171

rules_A=1
rules_B=1
rules_C=2
rules_D=3

Now you're ready to run the four-sliver tests.

Four-sliver tests

ABCD - A and B should be approved, C and D should be pending:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_A approved true $rules_A
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_B approved true $rules_B
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_C pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_D pending false 0

ACBD - A should be approved, C pending, B approved, and D pending:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_A approved true $rules_A
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_C pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_B approved true $rules_B
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_D pending false 0

BACD - B and A should be approved, C and D pending should be pending:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_B approved true $rules_B
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_A approved true $rules_A
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_C pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_D pending false 0

BCAD - B and C should be approved, A and D should be pending:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_B approved true $rules_B
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_C approved true $rules_C
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_A pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_D pending false 0

CABD - C should be approved, A pending, B approved, and D pending:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_C approved true $rules_C
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_A pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_B approved true $rules_B
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_D pending false 0

CBAD - C and B should be approved, A and D should be pending:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_C approved true $rules_C
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_B approved true $rules_B
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_A pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_D pending false 0

DABC - D should be approved, A and B and C should be pending:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_D approved true $rules_D
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_A pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_B pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_C pending false 0

ADBC - A should be approved, D pending, B approved, and C pending:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_A approved true $rules_A
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_D pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_B approved true $rules_B
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_C pending false 0

ADCB - A should be approved, D pending, C pending, and B approved:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_A approved true $rules_A
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_D pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_C pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_B approved true $rules_B

Delete any remaining slivers:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done

Three-feature slivers

This set of tests uses an rspec that reserves all three features (an IP subnet, a MAC address, and an Ethertype), and tests situations where the three-way rspec does and doesn't overlap with other rspecs that only reserve one of those three things.

Each test (one per block below) creates the slivers in a different order, to confirm that the anlysis engine detects overlap regardless of whether a more specific or less specific sliver is created first. Each test deletes any lingering slivers before running, just to make sure the slate is clean. Each test should generate four "sliver created" e-mail messages, plus one "sliver approved" message per sliver that's expected to be automatically approved; and also four "sliver deleted" messages if any slivers were there to be deleted.

These tests attempt to verify these principles:

  • If the three-way sliver doesn't overlap with any of the one-way slivers, they all get approved.
  • If the three-way sliver overlaps with the others:
    • It should be approved if it's created first, and the others should be pending.
    • It should be pending if it's created last, and the others should be approved.
    • If any one-way sliver is created first, and the three-way sliver is created second, the three-way sliver should be pending, and the others should all be approved.

This set of tests doesn't need any additional setup.

No overlap, three-way first:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_one_of_each approved true 5
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_10_7_9_0_24 approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_deadbeadfeed approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_ethertype_3172 approved true 1

No overlap, three-way last:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_10_7_9_0_24 approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_deadbeadfeed approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_ethertype_3172 approved true 1
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_one_of_each approved true 5

Overlap, three-way first:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_one_of_each approved true 5
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_10_3_5_0_24 pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_cafebeefface pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_ethertype_3171 pending false 0

Overlap three-way last:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_10_3_5_0_24 approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_cafebeefface approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_ethertype_3171 approved true 1
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_one_of_each pending false 0

Overlap, three-way second, IP first:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_10_3_5_0_24 approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_one_of_each pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_cafebeefface approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_ethertype_3171 approved true 1

Overlap, three-way second, MAC first:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_cafebeefface approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_one_of_each pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_10_3_5_0_24 approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_ethertype_3171 approved true 1

Overlap, three-way second, ethertype first:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_ethertype_3171 approved true 1
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_one_of_each pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_10_3_5_0_24 approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_cafebeefface approved true 2

Delete any remaining slivers:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done

Slivers with port groups

This set of tests uses rspecs containing port groups, and tests various situations where a sliver does or doesn't include multiple ports in a port group.

Because the different combinations of ports are defined in the rspec, each test creates only one sliver, using a different rspec with the different combinations of ports that we want to test.

NOTE that this assumes that you have datapaths and port groups defined in your FOAM server, which correspond to the datapaths and port groups in the rspec. FIXME: Need to document how to do this with fake datapaths.

One port in a port group, and one that isn't - should be approved:

foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_dpid_aa_port_3_and_36 approved true 4
for slicename in $slicename1 ; do omni -a $am deletesliver $slicename ; done

Two ports in a port group on the same datapath - should be pending:

foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_dpid_aa_port_3_and_11 pending false 0
for slicename in $slicename1 ; do omni -a $am deletesliver $slicename ; done

Two ports, one on each datapath, not in a port group - should be approved:

foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_dpid_aa_port_5_dpid_ff_port_21 approved true 8
for slicename in $slicename1 ; do omni -a $am deletesliver $slicename ; done

Two ports, one on each datapath, in a port group - should be pending:

foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_dpid_aa_port_9_dpid_ff_port_21 pending false 0
for slicename in $slicename1 ; do omni -a $am deletesliver $slicename ; done

Forbidden slivers

This set of tests uses rspecs containing things that should never be auto-approved. (Currently, only one of them, an rspec containing an entire VLAN.)

Create and verify the sliver:

foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_subnet_and_vlan pending false 0

(That should generate one "sliver created" and no "sliver approved" e-mail messages, one set to the experimenter and one set to the FOAM admin.)

Delete the sliver:

for slicename in $slicename1 ; do omni -a $am deletesliver $slicename ; done

Approval mode 1 ("always")

This subsection tests the behavior of FOAM when new slivers are created and geni.approval.approve-on-creation is set to 1 (always). The expected behavior is that all slivers will automatically be approved after they're created.

Double-check the FOAM server's current configuration:

ssh $foamserver foamctl config:get-value --key="geni.approval.approve-on-creation" --passwd-file=/opt/foam/etc/foampasswd

If the value isn't "1", then you shouldn't run this section on this FOAM server. (You can first change the configuration of this FOAM server, but you may not want to do that, e.g. if it's a server that anyone else is using.)

You can go through all of these tests in sequence; they're divided into blocks of four because we only try creating four slivers at a time.

IP subnets, smaller subnets first:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_10_3_5_0_24 approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_10_7_9_0_24 approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_10_3_0_0_16 approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_10_0_0_0_12 approved true 2

IP subnets, larger subnets first:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_10_0_0_0_12 approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_10_3_5_0_24 approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_10_7_9_0_24 approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_10_3_0_0_16 approved true 2

MAC addresses, fewer addresses first:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_cafebeefface approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_deadbeadfeed approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_aced_and_cafe approved true 4
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_aced_and_cafe_and_dead approved true 6

MAC addresses, more addresses first:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_aced_and_cafe_and_dead approved true 6
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_aced_and_cafe approved true 4
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_deadbeadfeed approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_cafebeefface approved true 2

Ethertypes, fewer ethertypes first:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_ethertype_3171 approved true 1
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_ethertype_3172 approved true 1
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_ethertype_3173_and_3171 approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_ethertype_3173_and_3172_and_3171 approved true 3

Ethertypes, more ethertypes first:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_ethertype_3173_and_3172_and_3171 approved true 3
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_ethertype_3173_and_3171 approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_ethertype_3172 approved true 1
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_ethertype_3171 approved true 1

Three-feature slivers with no overlap, three-way first:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_one_of_each approved true 5
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_10_7_9_0_24 approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_deadbeadfeed approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_ethertype_3172 approved true 1

Three-feature slivers with no overlap, three-way last:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_ethertype_3172 approved true 1
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_deadbeadfeed approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_10_7_9_0_24 approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_one_of_each approved true 5

Three-feature slivers with overlap, three-way first:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_one_of_each approved true 5
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_10_3_5_0_24 approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_cafebeefface approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_ethertype_3171 approved true 1

Three-feature slivers with overlap, three-way last:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_ethertype_3171 approved true 1
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_cafebeefface approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_10_3_5_0_24 approved true 2
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_one_of_each approved true 5

Forbidden slivers:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_subnet_and_vlan approved true 3

Delete any remaining slivers:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done

Approval mode 0 ("never")

This subsection tests the behavior of FOAM when new slivers are created and geni.approval.approve-on-creation is set to 0 (never). The expected behavior is that all slivers will automatically be pending after they're created.

Double-check the FOAM server's current configuration:

ssh $foamserver foamctl config:get-value --key="geni.approval.approve-on-creation" --passwd-file=/opt/foam/etc/foampasswd

If the value isn't "0", then you shouldn't run this section on this FOAM server. (You can first change the configuration of this FOAM server, but you may not want to do that, e.g. if it's a server that anyone else is using.)

You can go through all of these tests in sequence; they're divided into blocks of four because we only try creating four slivers at a time.

IP subnets, smaller subnets first:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_10_3_5_0_24 pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_10_7_9_0_24 pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_10_3_0_0_16 pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_10_0_0_0_12 pending false 0

IP subnets, larger subnets first:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_10_0_0_0_12 pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_10_3_5_0_24 pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_10_7_9_0_24 pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_10_3_0_0_16 pending false 0

MAC addresses, fewer addresses first:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_cafebeefface pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_deadbeadfeed pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_aced_and_cafe pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_aced_and_cafe_and_dead pending false 0

MAC addresses, more addresses first:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_aced_and_cafe_and_dead pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_aced_and_cafe pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_deadbeadfeed pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_cafebeefface pending false 0

Ethertypes, fewer ethertypes first:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_ethertype_3171 pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_ethertype_3172 pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_ethertype_3173_and_3171 pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_ethertype_3173_and_3172_and_3171 pending false 0

Ethertypes, more ethertypes first:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_ethertype_3173_and_3172_and_3171 pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_ethertype_3173_and_3171 pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_ethertype_3172 pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_ethertype_3171 pending false 0

Three-feature slivers with no overlap, three-way first:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_one_of_each pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_10_7_9_0_24 pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_deadbeadfeed pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_ethertype_3172 pending false 0

Three-feature slivers with no overlap, three-way last:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_ethertype_3172 pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_deadbeadfeed pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_10_7_9_0_24 pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_one_of_each pending false 0

Three-feature slivers with overlap, three-way first:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_one_of_each pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_10_3_5_0_24 pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_cafebeefface pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_ethertype_3171 pending false 0

Three-feature slivers with overlap, three-way last:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_ethertype_3171 pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename2 $rspec_cafebeefface pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename3 $rspec_10_3_5_0_24 pending false 0
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename4 $rspec_one_of_each pending false 0

Forbidden slivers:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done
foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename1 $rspec_subnet_and_vlan pending false 0

Delete any remaining slivers:

for slicename in $slicename1 $slicename2 $slicename3 $slicename4 ; do omni -a $am deletesliver $slicename > /dev/null 2>&1 ; done

Automation

We have a wrapper script that does all of that automatically, but it's also GPO-only at the moment. For GPO people, it's foam-run-auto-approval-tests, and you run it like this:

foam-run-auto-approval-tests foam3.gpolab.bbn.com 0

where the first argument is the FOAM server, and the second is the approval mode to test. (You can test an approval mode other than the one that the server is in, if you want to confirm that you get lots of error output when the test results aren't as expected -- it just tells the script what mode to test, it doesn't change anything on the server.)

You may also want to disable some e-mail notifications temporarily, as above, and re-enable them afterwards; the script doesn't do that for you.

GENI AM API

This section tests whether a given FOAM server is compliant with the GENI AM API, using Omni's built-in acceptance test suite. Stanford/ONLab and the GPO generally run these tests when new versions of FOAM are released, but you can run them yourself too, if you want.

Pre-requisites/assumptions:

  • You're set up to use Omni in general, and have it in your path as 'omni', i.e. you've successfully used Omni to reserve GENI resources (and have the latest version of the software, a user credential signed by a CA that's trusted by the FOAM server you intend to test, a working omni_config file, etc).
  • You have four rspec files needed to test with (three valid request rspecs, and one valid manifest rspec). We plan to make some sample rspecs available.

You can run these tests wherever you usually use Omni; they don't need to run on the FOAM/FV server.

As of this writing, the current version of GCF is 1.6.2; some of these steps may need to be different if you have a more recent version than that.

Note that this is a streamlined summary of the full docs for running the GCF GENI AM API Acceptance Test Suite. The full docs for that suite are in .../gcf-1.6.2/acceptance_tests/AM_API/README-accept-AMAPI.txt; there's also much more information at http://trac.gpolab.bbn.com/gcf/wiki/AmApiAcceptanceTests.

Setup

Set the FOAM server name (modify this to use the DNS name of the server that you want to test):

foamserver=foam.example.edu

Set the AM API URL:

am=https://${foamserver}:3626/foam/gapi/1

Identify the rspecs you plan to use (modify this to use your actual rspecs):

rspec_request1=~/rspecs/request/foam-testing/$foamserver/openflow-mac-acedcodefood.rspec
rspec_request2=~/rspecs/request/foam-testing/$foamserver/openflow-mac-cafebeefface.rspec
rspec_request3=~/rspecs/request/foam-testing/$foamserver/openflow-mac-deadbeadfeed.rspec
rspec_manifest1=~/rspecs/manifest/jbs-gpolab-cafebeefface.rspec

Run the tests

If you don't already have the latest version of Omni, you can get it from http://www.gpolab.bbn.com/local-sw/.

The rest of these docs assume that you have GCF 1.6.2 in ~/src/gcf-1.6.2.

Change into that directory, and copy in your rspec files:

cd ~/src/gcf-1.6.2/acceptance_tests/AM_API
cp $rspec_request1 request1.xml
cp $rspec_request3 request2.xml
cp $rspec_request2 request3.xml
cp $rspec_manifest1 bad.xml
cp request1.xml request.xml

Set your Python path, and run the tests:

export PYTHONPATH=~/src/gcf-1.6.2/src
./am_api_accept.py -a $am

If you'd like more verbose details about what tests it's running, you can add '--v'.

When not in verbose mode, this will run for a few minutes, and should produce output like

.............
----------------------------------------------------------------------
Ran 13 tests in 202.744s

OK

This may generate e-mail with harmless "slice is busy" errors to the admins of the Slice Authority that you're using; this can happen if something tries to use a slice too quickly after it's been created. Omni should re-try in those circumstances, and you won't see any errors yourself (unless you happen to be an admin for the Slice Authority you're using).

Any other errors or failures are unexpected and should be investigated.

The test suite should also generate sixteen or twenty e-mail messages to the experimenter, and the same number to the FOAM admin. There should be a "sliver created" message, two "sliver renewed" messages, and a "sliver deleted" message, and might be a "sliver approved" message if the sliver is automatically approved, for each of four slices; the slices have names like acc${user}-${time}, where $user is your Unix username on the system where you're running the tests, and $time is a six-digit timestamp for uniqueness.