Changes between Initial Version and Version 1 of OpenFlow/FOAM/Testing

Timestamp:

03/07/12 14:54:59 (12 years ago)

Author:

Josh Smift

Comment:

--

OpenFlow/FOAM/Testing

@@ +1 @@
+[[PageOutline]]
+
+= Testing FOAM =
+
+This page describes a couple of ways to test FOAM:
+
+ * Testing that the software complies with the GENI AM API, using the test suite that comes with GCF.
+ * Testing that slivers can be created, and then managed in various ways, including:
+   * Simple tests that slivers appear and disappear in FlowVisor.
+   * More complex tests for other sliver management operations.
+   * Tests for sliver expiration.
+
+= GENI AM API =
+
+This tests whether a given FOAM server is compliant with the GENI AM API, using Omni's built-in acceptance test suite.
+
+Pre-requisites/assumptions:
+
+ * You're set up to use Omni in general, i.e. you've successfully used Omni to reserve GENI resources (and have the latest version of the software, a user credential signed by a CA that's trusted by the FOAM server you intend to test, a working omni_config file, etc).
+ * You have four rspec files needed to test with (three valid request rspecs, and one valid manifest rspec). We plan to make some sample rspecs available.
+
+You can run these tests wherever you usually use Omni; they don't need to run on the FOAM/FV server.
+
+As of this writing, the current version of GCF is 1.6.1; some of these steps may need to be different if you have a more recent version than that.
+
+Note that this is a streamlined summary of the full docs for running the GCF GENI AM API Acceptance Test Suite. The full docs for that suite are in .../gcf-1.6.1/acceptance_tests/AM_API/README-accept-AMAPI.txt.
+
+== Setup ==
+
+Set the FOAM server name (modify this to use the DNS name of the server that you want to test):
+
+{{{
+foamserver=foam.example.edu
+}}}
+
+Set the AM API URL:
+
+{{{
+am=https://${foamserver}:3626/foam/gapi/1
+}}}
+
+Identify the rspecs you plan to use (modify this to use your actual rspecs):
+
+{{{
+rspec_request1=~/rspecs/request/misc/openflow-any-acedcodefood.rspec
+rspec_request2=~/rspecs/request/misc/openflow-any-cafebeefface.rspec
+rspec_request3=~/rspecs/request/misc/openflow-any-deadbeadfeed.rspec
+rspec_manifest1=~/rspecs/manifest/jbs-gpolab-cafebeefface.rspec
+}}}
+
+== Run the tests ==
+
+If you don't already have the latest version of Omni, you can get it from http://www.gpolab.bbn.com/local-sw/.
+
+The rest of these docs assume that you have GCF 1.6.1 in ~/src/gcf-1.6.1.
+
+Change into that directory, and copy in your rspec files:
+
+{{{
+cd ~/src/gcf-1.6.1/acceptance_tests/AM_API
+cp $rspec_request1 request1.xml
+cp $rspec_request3 request2.xml
+cp $rspec_request2 request3.xml
+cp $rspec_manifest1 bad.xml
+cp request1.xml request.xml
+}}}
+
+Set your Python path, and run the tests:
+
+{{{
+export PYTHONPATH=~/src/gcf-1.6.1/acceptance_tests/AM_API
+./am_api_accept.py -a $am
+}}}
+
+If you'd like more verbose details about what tests it's running, you can add '--v'.
+
+When not in verbose mode, this will run for a few minutes, and should produce output like
+
+{{{
+.............
+----------------------------------------------------------------------
+Ran 13 tests in 202.744s
+
+OK
+}}}
+
+This may generate e-mail with harmless "slice is busy" errors to the admins of the Slice Authority that you're using; this can happen if something tries to use a slice too quickly after it's been created. Omni should re-try in those circumstances, and you won't see any errors yourself (unless you happen to be an admin for the Slice Authority you're using).
+
+Any other errors or failures are unexpected and should be investigated.
+
+The test suite should also generate sixteen e-mail messages to the experimenter and sixteen to the FOAM admin. There should be a "sliver created" message, two "sliver renewed" messages, and a "sliver deleted" message, for each of four slices; the slices have names like accjbs, accjbs0, accjbs1, and accjbs2, replacing 'jbs' with your Unix username on the system where you're running the tests.
+
+= Sliver creation and management =
+
+This set of tests includes a Setup section to set some variables, a Creation section to create a sliver, and then additional sections to do various tests with it.
+
+You only need to do the Setup section once.
+
+You need to repeat the Creation section before doing any of the other sections, because the Creation section sets some variables that you'll need for the other sections, and each of the other sections concludes by deleting the sliver.
+
+Pre-requisites/assumptions:
+
+ * You're set up to use Omni in general, and have it in your path as 'omni', i.e. you've successfully used Omni to reserve GENI resources (and have the latest version of the software, a user credential signed by a CA that's trusted by the FOAM server you intend to test, a working omni_config file, etc).
+ * You have a valid request rspec file to test with. We plan to make some sample rspecs available.
+ * You can log in non-interactively to the FOAM and FV servers, and have admin privs there, and have the FOAM and fvadmin password stored in files (/opt/foam/etc/passwd and /etc/flowvisor/fvpasswd). (This generally can't be used as-is to test FOAM at other sites.)
+ * You know what output to expect when common things succeed. (Expected output is described here only in cases where an error is the expected result, or where it otherwise might not be obvious; if you think the expected output isn't obvious for any given step, let us know.)
+ * The URL of the FOAM server is also a name that you can use to log in to it. (Very likely, but might be untrue with virtualhosts or some other odd thing.)
+ * On the FOAM server you're testing, when you create a sliver, it's marked as Pending, and isn't automatically approved.
+
+You can run most of these tests wherever you usually use Omni; they don't need to run on the FOAM/FV server, except as noted below.
+
+== Setup ==
+
+Set the FOAM server name (modify this to use the DNS name of the server that you want to test):
+
+{{{
+foamserver=foam.example.edu
+}}}
+
+Set the name of the FlowVisor server to be the same (if it's the same, or something else if it isn't):
+
+{{{
+fvserver=$foamserver
+}}}
+
+Set the AM API URL:
+
+{{{
+am=https://${foamserver}:3626/foam/gapi/1
+}}}
+
+Identify the rspec you plan to use (modify this if you want to use a different rspec):
+
+{{{
+rspec=~/rspecs/request/misc/openflow-any-acedcodefood.rspec
+}}}
+
+Set the slicename to something with your username and "foamtest":
+
+{{{
+slicename=${USER}foamtest
+}}}
+
+Set the full slice URN (modify this if you're not using pgeni.gpolab.bbn.com as your Slice Authority), and a version of it for handing off to egrep later:
+
+{{{
+slice_urn=urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+$slicename
+slice_urn_egrep=$(echo $slice_urn | sed -e 's/+/\\+'/g)
+}}}
+
+Make sure FOAM is listening at all:
+
+{{{
+omni -a $am getversion
+omni -a $am listresources
+}}}
+
+Now you're ready to create a sliver.
+
+== Creation ==
+
+Create a slice:
+
+{{{
+omni createslice $slicename
+}}}
+
+Create a sliver:
+
+{{{
+omni -a $am createsliver $slicename $rspec
+}}}
+
+(That should generate a "sliver created" e-mail message, one to the experimenter and one to the FOAM admin.)
+
+Confirm that it looks right via Omni:
+
+{{{
+omni -a $am listresources $slicename
+omni -a $am sliverstatus $slicename
+}}}
+
+Confirm that it shows up in FOAM:
+
+{{{
+ssh $foamserver foamctl list-slivers --passwd-file=/opt/foam/etc/foampasswd | egrep slice_urn.+$slice_urn_egrep
+}}}
+
+Get the sliver URN and FlowVisor slicename from FOAM, and put them into $sliver_urn and $fv_slicename:
+
+{{{
+sliver_urn=$(ssh $foamserver foamctl list-slivers --passwd-file=/opt/foam/etc/foampasswd | egrep sliver_urn.+$slice_urn_egrep | sed -e 's/ *"sliver_urn": "\(.*\)".*/\1/')
+fv_slicename=$(echo $sliver_urn | awk -F : '{print $NF}')
+}}}
+
+Now you're ready to do additional tests with this sliver.
+
+== Simple management ==
+
+NOTE: Do the [wiki:OpenFlow/FOAM/Testing#Creation Creation] section immediately before starting this section.
+
+The steps in this section look at a newly-created sliver, approve it, confirm that it shows up in the FlowVisor, delete it, and confirm that it vanishes from the FlowVisor.
+
+Show the sliver, and confirm that 'status' is "Pending":
+
+{{{
+ssh $foamserver foamctl show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+}}}
+
+Confirm that the sliver doesn't yet show up in FlowVisor:
+
+{{{
+ssh $fvserver fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo $fv_slicename
+}}}
+
+(Expect a "slice does not exist" error.)
+
+Show the sliver's rspec:
+
+{{{
+ssh $foamserver foamctl show-sliver -r -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+}}}
+
+Show the sliver's flowspec:
+
+{{{
+ssh $foamserver foamctl show-sliver -s -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+}}}
+
+Show the sliver's flowspace rules:
+
+{{{
+ssh $foamserver foamctl show-sliver -f -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+}}}
+
+Approve the sliver, and confirm that 'status' is now "Approved":
+
+{{{
+ssh $foamserver foamctl approve-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+ssh $foamserver foamctl show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+}}}
+
+(That should generate a "sliver approved" e-mail message, one to the experimenter and one to the FOAM admin.)
+
+Confirm that the sliver now shows up in FlowVisor, with two flowspace rules:
+
+{{{
+ssh $fvserver fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo $fv_slicename
+ssh $fvserver fvctl --passwd-file=/etc/flowvisor/fvpasswd listFlowSpace | grep $fv_slicename
+}}}
+
+(Don't worry about the ouptut saying that it can't connect to the controller; this is expected if your rspec didn't include a controller that's actually running, which the example one doesn't.)
+
+Delete the sliver:
+
+{{{
+omni -a $am deletesliver $slicename
+}}}
+
+(That should generate a "sliver deleted" e-mail message, one to the experimenter and one to the FOAM admin.)
+
+Confirm that Omni thinks it's gone:
+
+{{{
+omni -a $am listresources $slicename
+omni -a $am sliverstatus $slicename
+}}}
+
+(Expect an "unknown slice" error for each of those.)
+
+Confirm that the sliver no longer shows up in FOAM:
+
+{{{
+ssh $foamserver foamctl list-slivers --passwd-file=/opt/foam/etc/foampasswd | grep $slice_urn
+ssh $foamserver foamctl show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+}}}
+
+(Expect no output for the first, and 'deleted' to be "True" for the second; note that 'status' will still be "Approved", which is fine, it's just tracking what the status was when it was deleted.)
+
+Confirm that the sliver no longer shows up in FlowVisor:
+
+{{{
+ssh $fvserver fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo $fv_slicename
+ssh $fvserver fvctl --passwd-file=/etc/flowvisor/fvpasswd listFlowSpace | grep $fv_slicename
+}}}
+
+(Expect a "slice does not exist" error for the first, and no output for the second.)
+
+== Complex management ==
+
+NOTE: Do the [wiki:OpenFlow/FOAM/Testing#Creation Creation] section immediately before starting this section.
+
+The steps in this section look at a newly-created sliver, run through a sequence of approve/disable/reject actions to confirm that it can transition from any of those states to the others, and check the FlowVisor after each change; then delete the sliver, and confirm that it vanishes from the FlowVisor.
+
+NOTE: These steps are run directly on the FOAM/FlowVisor server, rather than on your Omni client system. If those aren't the same server, do the 'foamctl' commands on the FOAM server, and the 'fvctl' commands on the FV server.
+
+Repeat the setup steps from above, but on the server, so you've got the right variables set there:
+
+{{{
+slicename=${USER}foamtest
+slice_urn=urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+$slicename
+slice_urn_egrep=$(echo $slice_urn | sed -e 's/+/\\+'/g)
+sliver_urn=$(foamctl list-slivers --passwd-file=/opt/foam/etc/foampasswd | egrep sliver_urn.+$slice_urn_egrep | sed -e 's/ *"sliver_urn": "\(.*\)".*/\1/')
+fv_slicename=$(echo $sliver_urn | awk -F : '{print $NF}')
+}}}
+
+Show the sliver, and confirm that 'status' is "Pending":
+
+{{{
+foamctl show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+}}}
+
+Confirm that the sliver doesn't yet show up in FlowVisor:
+
+{{{
+fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo $fv_slicename
+}}}
+
+(Expect a "slice does not exist" error; and likewise every time you do a "confirm that the sliver doesn't show up in FlowVisor" step below.)
+
+Add a temporary cron job to tell FOAM to generate e-mail with a list of pending slivers:
+
+{{{
+sudo sh -c "echo $(date +'%M %H' -d 'now + 1 min') '* * *  www-data  /opt/foam/bin/daily-queue ; echo foamtest cron job ran at $(date)' > /etc/cron.d/foamtest"
+}}}
+
+(Within a minute (when the cron job fires), that should generate e-mail to the FOAM admin with a list of pending slivers, including your test sliver; and a message to gpo-ops saying that the cron job ran.)
+
+Approve the sliver, confirm that 'status' is now "Approved", and confirm that the sliver now shows up in FlowVisor, with two flowspace rules:
+
+{{{
+foamctl approve-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+foamctl show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo $fv_slicename
+fvctl --passwd-file=/etc/flowvisor/fvpasswd listFlowSpace | grep $fv_slicename
+}}}
+
+(Don't worry about the ouptut saying that it can't connect to the controller; this is expected if your rspec didn't include a controller that's actually running, which the example one doesn't. Likewise every time you confirm that the sliver shows up in Flowvisor below.)
+
+(That should generate a "sliver approved" e-mail message, one to the experimenter and one to the FOAM admin.)
+
+Add a temporary cron job to tell FOAM to generate e-mail with a list of pending slivers:
+
+{{{
+sudo sh -c "echo $(date +'%M %H' -d 'now + 1 min') '* * *  www-data  /opt/foam/bin/daily-queue ; echo foamtest cron job ran at $(date)' > /etc/cron.d/foamtest"
+}}}
+
+(Within a minute (when the cron job fires), either (a) if there were other slivers in the previous mail, that should generate a message to the FOAM admin listing only those slivers, and not your test sliver, or otherwise (b) not send any message to the FOAM admin; and a message to gpo-ops saying that the cron job ran.)
+
+Clean up the cron job:
+
+{{{
+sudo rm /etc/cron.d/foamtest
+}}}
+
+This next set of tests will generate a bunch of e-mail; check at the end that it all showed up as expected.
+
+Disable the sliver, confirm that 'status' is now "Pending", and confirm that the sliver now does not show up in FlowVisor, and has no flowspace rules:
+
+{{{
+foamctl disable-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+foamctl show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo $fv_slicename
+fvctl --passwd-file=/etc/flowvisor/fvpasswd listFlowSpace | grep $fv_slicename
+}}}
+
+Reject the sliver, confirm that 'status' is now "Rejected", and confirm that the sliver now does not show up in FlowVisor, and has no flowspace rules:
+
+{{{
+foamctl reject-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+foamctl show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo $fv_slicename
+fvctl --passwd-file=/etc/flowvisor/fvpasswd listFlowSpace | grep $fv_slicename
+}}}
+
+Approve the sliver, confirm that 'status' is now "Approved", and confirm that the sliver now shows up in FlowVisor, with two flowspace rules:
+
+{{{
+foamctl approve-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+foamctl show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo $fv_slicename
+fvctl --passwd-file=/etc/flowvisor/fvpasswd listFlowSpace | grep $fv_slicename
+}}}
+
+Reject the sliver, confirm that 'status' is now "Rejected", and confirm that the sliver now does not show up in FlowVisor, and has no flowspace rules:
+
+{{{
+foamctl reject-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+foamctl show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo $fv_slicename
+fvctl --passwd-file=/etc/flowvisor/fvpasswd listFlowSpace | grep $fv_slicename
+}}}
+
+Disable the sliver, confirm that 'status' is now "Pending", and confirm that the sliver now does not show up in FlowVisor, and has no flowspace rules:
+
+{{{
+foamctl disable-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+foamctl show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo $fv_slicename
+fvctl --passwd-file=/etc/flowvisor/fvpasswd listFlowSpace | grep $fv_slicename
+}}}
+
+Approve the sliver, confirm that 'status' is now "Approved", and confirm that the sliver now shows up in FlowVisor, with two flowspace rules:
+
+{{{
+foamctl approve-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+foamctl show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo $fv_slicename
+fvctl --passwd-file=/etc/flowvisor/fvpasswd listFlowSpace | grep $fv_slicename
+}}}
+
+Delete the sliver, confirm that 'deleted' is now "True", and confirm that the sliver now does not show up in FlowVisor, and has no flowspace rules:
+
+{{{
+foamctl delete-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+foamctl show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo $fv_slicename
+fvctl --passwd-file=/etc/flowvisor/fvpasswd listFlowSpace | grep $fv_slicename
+}}}
+
+That set of tests should generate seven e-mail messages to the experimenter and seven to the FOAM admin: There should be a "sliver disabled" message, then "sliver rejected", then approved, rejected, disabled, approved, and deleted. Probably in that order, although e-mail is sometimes delivered out of sequence, so don't worry too much if they're not in order.
+
+== Expiration ==
+
+NOTE: Do the [wiki:OpenFlow/FOAM/Testing#Creation Creation] section immediately before starting this section.
+
+The steps in this section look at a newly-created sliver, and test expiration in FOAM, by changing its expiration date and, then running the scripts that expire slivers and notify about soon-to-expire ones.
+
+First, review and approve the sliver.
+
+Show the sliver, and confirm that 'status' is "Pending":
+
+{{{
+ssh $foamserver foamctl show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+}}}
+
+Confirm that the sliver doesn't yet show up in FlowVisor:
+
+{{{
+ssh $fvserver fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo $fv_slicename
+}}}
+
+(Expect a "slice does not exist" error.)
+
+Show the sliver's rspec:
+
+{{{
+ssh $foamserver foamctl show-sliver -r -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+}}}
+
+Show the sliver's flowspec:
+
+{{{
+ssh $foamserver foamctl show-sliver -s -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+}}}
+
+Show the sliver's flowspace rules:
+
+{{{
+ssh $foamserver foamctl show-sliver -f -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+}}}
+
+Approve the sliver, and confirm that 'status' is now "Approved":
+
+{{{
+ssh $foamserver foamctl approve-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+ssh $foamserver foamctl show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+}}}
+
+(That should generate a "sliver approved" e-mail message, one to the experimenter and one to the FOAM admin.)
+
+Confirm that the sliver now shows up in FlowVisor, with two flowspace rules:
+
+{{{
+ssh $fvserver fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo $fv_slicename
+ssh $fvserver fvctl --passwd-file=/etc/flowvisor/fvpasswd listFlowSpace | grep $fv_slicename
+}}}
+
+(Don't worry about the ouptut saying that it can't connect to the controller; this is expected if your rspec didn't include a controller that's actually running, which the example one doesn't.)
+
+Now, do some tests to make sure FOAM notifies you about a sliver that will expire within a day, but only once.
+
+Change the expiration date of your slice to be nine days away, since otherwise you can't extend the sliver expiration date very far:
+
+{{{
+omni renewslice $slicename "$(date -d 'now + 9 days')" -f gpolab
+}}}
+
+Change the expiration date of your sliver to be twelve hours away, and confirm that the sliver has the new expiration date:
+
+{{{
+omni -a $am renewsliver $slicename "$(date -d 'now + 12 hours')"
+omni -a $am sliverstatus $slicename
+}}}
+
+(That should generate a "sliver renewed" e-mail message, one to the experimenter and one to the FOAM admin.)
+
+Add a temporary cron job to tell FOAM to send mail about slivers that are expiring soon and remove any expired slivers:
+
+{{{
+ssh -t $foamserver "sudo sh -c \"echo $(date +'%M %H' -d 'now + 1 min') '* * *  www-data  /opt/foam/bin/expire ; /opt/foam/bin/expire-emails ; echo foamtest cron job ran at $(date)' > /etc/cron.d/foamtest\""
+}}}
+
+(Within a minute (when the cron job fires), that should generate a "sliver expiring shortly" e-mail message, one to the experimenter and one to the FOAM admin; no messages about expired slivers being deleted; and one message to gpo-ops saying that the cron job ran.)
+
+Add the cron job again, to confirm that it doesn't send a second notification:
+
+{{{
+ssh -t $foamserver "sudo sh -c \"echo $(date +'%M %H' -d 'now + 1 min') '* * *  www-data  /opt/foam/bin/expire ; /opt/foam/bin/expire-emails ; echo foamtest cron job ran at $(date)' > /etc/cron.d/foamtest\""
+}}}
+
+(Within a minute (when the cron job fires), that should generate no messages about slivers expiring in the future; no messages about expired slivers being deleted; and one message to gpo-ops saying that the cron job ran.)
+
+Next, test to make sure FOAM notifies you again if you change the expiration date.
+
+Change the expiration date of your sliver to be one day away, and confirm that the sliver has the new expiration date:
+
+{{{
+omni -a $am renewsliver $slicename "$(date -d 'now + 1 day')"
+omni -a $am sliverstatus $slicename
+}}}
+
+(That should generate a "sliver renewed" e-mail message, one to the experimenter and one to the FOAM admin.)
+
+Add the cron job again, to confirm that it does notify you again now that the expiration date has changed:
+
+{{{
+ssh -t $foamserver "sudo sh -c \"echo $(date +'%M %H' -d 'now + 1 min') '* * *  www-data  /opt/foam/bin/expire ; /opt/foam/bin/expire-emails ; echo foamtest cron job ran at $(date)' > /etc/cron.d/foamtest\""
+}}}
+
+(Within a minute (when the cron job fires), that should generate a "sliver expiring shortly" e-mail message, one to the experimenter and one to the FOAM admin; no messages about expired slivers being deleted; and one message to gpo-ops saying that the cron job ran.)
+
+Next, do some tests to make sure that FOAM notifies you about a sliver that will expire within a week, but only once.
+
+Change the expiration date of your sliver to be two days away, and confirm that the sliver has the new expiration date:
+
+{{{
+omni -a $am renewsliver $slicename "$(date -d 'now + 2 days')"
+omni -a $am sliverstatus $slicename
+}}}
+
+(That should generate a "sliver renewed" e-mail message, one to the experimenter and one to the FOAM admin.)
+
+Add the cron job again, to confirm that it now sends a "within a week" message:
+
+{{{
+ssh -t $foamserver "sudo sh -c \"echo $(date +'%M %H' -d 'now + 1 min') '* * *  www-data  /opt/foam/bin/expire ; /opt/foam/bin/expire-emails ; echo foamtest cron job ran at $(date)' > /etc/cron.d/foamtest\""
+}}}
+
+(Within a minute (when the cron job fires), that should generate a "sliver expiring soon" e-mail message, one to the experimenter and one to the FOAM admin; no messages about expired slivers being deleted; and one message to gpo-ops saying that the cron job ran.)
+
+Add the cron job again, to confirm that it doesn't send a second notification:
+
+{{{
+ssh -t $foamserver "sudo sh -c \"echo $(date +'%M %H' -d 'now + 1 min') '* * *  www-data  /opt/foam/bin/expire ; /opt/foam/bin/expire-emails ; echo foamtest cron job ran at $(date)' > /etc/cron.d/foamtest\""
+}}}
+
+(Within a minute (when the cron job fires), that should generate no messages about slivers expiring in the future; no messages about expired slivers being deleted; and one message to gpo-ops saying that the cron job ran.)
+
+Next, test to make sure FOAM doesn't notify you at all if your sliver is expiring more than a week from now.
+
+Change the expiration date of your sliver to be eight days away, and confirm that the sliver has the new expiration date:
+
+{{{
+omni -a $am renewsliver $slicename "$(date -d 'now + 8 days')"
+omni -a $am sliverstatus $slicename
+}}}
+
+(That should generate a "sliver renewed" e-mail message, one to the experimenter and one to the FOAM admin.)
+
+Add the cron job again, to confirm that it doesn't send a notification:
+
+{{{
+ssh -t $foamserver "sudo sh -c \"echo $(date +'%M %H' -d 'now + 1 min') '* * *  www-data  /opt/foam/bin/expire ; /opt/foam/bin/expire-emails ; echo foamtest cron job ran at $(date)' > /etc/cron.d/foamtest\""
+}}}
+
+(Within a minute (when the cron job fires), that should generate no messages about slivers expiring in the future; no messages about expired slivers being deleted; and one message to gpo-ops saying that the cron job ran.)
+
+Finally, test to make sure FOAM deletes a sliver if its expiration date is in the past.
+
+Change the expiration date of your sliver to be right now, and confirm that the sliver has the new expiration date:
+
+{{{
+omni -a $am renewsliver $slicename "$(date -d 'now')"
+omni -a $am sliverstatus $slicename
+}}}
+
+(That should generate a "sliver renewed" e-mail message, one to the experimenter and one to the FOAM admin.)
+
+Add the cron job again, to confirm that it deletes the sliver:
+
+{{{
+ssh -t $foamserver "sudo sh -c \"echo $(date +'%M %H' -d 'now + 1 min') '* * *  www-data  /opt/foam/bin/expire ; /opt/foam/bin/expire-emails ; echo foamtest cron job ran at $(date)' > /etc/cron.d/foamtest\""
+}}}
+
+(Within a minute (when the cron job fires), that should generate a "sliver expiring shortly" e-mail message, one to the experimenter and one to the FOAM admin; a "sliver expired" message, one to the experimenter and one to the FOAM admin; and one message to gpo-ops saying that the cron job ran.)
+
+Confirm that the sliver no longer shows up in FOAM:
+
+{{{
+ssh $foamserver foamctl list-slivers --passwd-file=/opt/foam/etc/foampasswd | grep $slice_urn
+ssh $foamserver foamctl show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
+}}}
+
+(Expect no output for the first, and 'deleted' to be "True" for the second.)
+
+Confirm that the sliver no longer shows up in FlowVisor:
+
+{{{
+ssh $fvserver fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo $fv_slicename
+ssh $fvserver fvctl --passwd-file=/etc/flowvisor/fvpasswd listFlowSpace | grep $fv_slicename
+}}}
+
+(Expect a "slice does not exist" error for the first, and no output for the second.)
+
+Clean up the last cron job:
+
+{{{
+ssh -t $foamserver sudo rm /etc/cron.d/foamtest
+}}}