Changes between Version 30 and Version 31 of OpenFlow/FOAM/Testing


Ignore:
Timestamp:
06/29/12 11:40:34 (11 years ago)
Author:
Josh Smift
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • OpenFlow/FOAM/Testing

    v30 v31  
    55NOTE that all the foamctl commands on this page now use the new 0.8 syntax. If you want an older version, with the 0.6 syntax, try http://groups.geni.net/geni/wiki/OpenFlow/FOAM/Testing?version=6 instead.
    66
     7NOTE that we've steadily increased the extent to which these tests are automated, by scripts that are currently GPO-only. We're working on making them public, but in the meanwhile, this page may not be as useful for non-BBN sites.
     8
    79This page describes a couple of ways to test FOAM:
    810
    9  * Testing that slivers can be created, and then managed in various ways, including:
    10    * Simple tests that slivers appear and disappear in FlowVisor.
    11    * More complex tests for other sliver management operations.
     11 * Testing sliver administration, including:
     12   * Test for various sliver management operations.
    1213   * Tests for sliver expiration.
    13  * Testing auto-approval, in the various approve-on-creation modes.
     14 * Testing auto-approval, in each of the three approve-on-creation modes.
    1415 * Testing that the software complies with the GENI AM API, using the test suite that comes with GCF.
    1516
    16 = Sliver creation and management =
    17 
    18 This section contains multiple tests, and includes a Setup subsection to set some variables, a Creation subsection to create a sliver, and then additional subsections to do various tests with it.
    19 
    20 You only need to do the Setup subsection once.
    21 
    22 You need to repeat the Creation subsection before doing any of the other subsections, because the Creation subsection sets some variables that you'll need for the other subsections, and each of the other subsections concludes by deleting the sliver.
     17= Sliver administration =
     18
     19This section contains multiple tests, and includes a "General setup" subsection to set some variables, and then additional subsections to do various tests.
     20
     21You only need to do the "General setup" subsection once.
    2322
    2423Pre-requisites/assumptions:
    … …  
    3231 * On the FOAM server you're testing, when the 'www-data' user runs a cron job that has output, it sends mail to 'www-data', and you can receive this mail.
    3332 * On the FOAM server you're testing, you have the foam-confirm-sliver, flowvisor-confirm-fvslice-exists, and flowvisor-confirm-flowspace-rule-count scripts. (These are currently GPO-only; GPO folks, they're in the syseng repo, under gposw.)
     33 * On the system where you're running commands, you have the foam-create-and-verify-sliver script.
     34
     35Note that the scripts mentioned in the last two items are currently GPO-only. (GPO folks, they're in the syseng repo, under gposw.)
    3436
    3537You can run most of these tests wherever you usually use Omni; they don't need to run on the FOAM/FV server, except as noted below.
    3638
    37 == Setup ==
     39== General setup ==
    3840
    3941Set the FOAM server name (modify this to use the DNS name of the server that you want to test):
    … …  
    8183}}}
    8284
    83 Now you're ready to create a sliver.
    84 
    85 Here's all of those commands in one block for easy of copy-and-paste:
     85Here's all of those commands in one block for ease of copy-and-paste:
    8686
    8787{{{
    … …  
    9898}}}
    9999
    100 == Creation ==
    101 
    102 Create a slice:
    103 
    104 {{{
    105 omni createslice $slicename
    106 }}}
    107 
    108 Create a sliver:
    109 
    110 {{{
    111 omni -a $am createsliver $slicename $rspec
     100== Management ==
     101
     102This subsection runs creates a sliver, and runs through a sequence of approve/disable/reject/delete actions to confirm that it can transition from any of those states to the others, and checks the FlowVisor after each change to confirm that the sliver is or isn't enabled (as appropriate).
     103
     104Create and verify the sliver:
     105
     106{{{
     107foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename $rspec approved true 2
    112108}}}
    113109
    114110(That should generate a "sliver created" and a "sliver approved" e-mail message, one of each to the experimenter and one of each to the FOAM admin.)
    115 
    116 Confirm that it looks right via Omni:
    117 
    118 {{{
    119 omni -a $am listresources $slicename
    120 omni -a $am sliverstatus $slicename
    121 }}}
    122 
    123 Confirm that it shows up in FOAM:
    124 
    125 {{{
    126 ssh $foamserver foamctl geni:list-slivers --passwd-file=/opt/foam/etc/foampasswd | egrep slice_urn.+$slice_urn_egrep
    127 }}}
    128111
    129112Get the sliver URN and FlowVisor slicename from FOAM, and put them into $sliver_urn and $fv_slicename:
    … …  
    134117}}}
    135118
    136 Now you're ready to do additional tests with this sliver.
    137 
    138 == Simple management ==
    139 
    140 NOTE: Do the [wiki:OpenFlow/FOAM/Testing#Creation Creation] subsection immediately before starting this subsection.
    141 
    142 NOTE: This subsection needs to be revamped to match the new assumption that when you create a sliver, it will be automatically approved.
    143 
    144 The steps in this subsection look at a newly-created sliver, approve it, confirm that it shows up in the FlowVisor, delete it, and confirm that it vanishes from the FlowVisor.
    145 
    146 Show the sliver, and confirm that 'status' is "Pending":
    147 
    148 {{{
    149 ssh $foamserver foamctl geni:show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
    150 }}}
    151 
    152 Confirm that the sliver doesn't yet show up in FlowVisor:
    153 
    154 {{{
    155 ssh $fvserver fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo $fv_slicename
    156 }}}
    157 
    158 (Expect a "slice does not exist" error.)
    159 
    160 Show the sliver's rspec:
    161 
    162 {{{
    163 ssh $foamserver foamctl geni:show-sliver -r -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
    164 }}}
    165 
    166 Show the sliver's flowspec:
    167 
    168 {{{
    169 ssh $foamserver foamctl geni:show-sliver -s -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
    170 }}}
    171 
    172 Show the sliver's flowspace rules:
    173 
    174 {{{
    175 ssh $foamserver foamctl geni:show-sliver -f -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
    176 }}}
    177 
    178 Approve the sliver, and confirm that 'status' is now "Approved":
    179 
    180 {{{
    181 ssh $foamserver foamctl geni:approve-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
    182 ssh $foamserver foamctl geni:show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
    183 }}}
    184 
    185 (That should generate a "sliver approved" e-mail message, one to the experimenter and one to the FOAM admin.)
    186 
    187 Confirm that the sliver now shows up in FlowVisor, with two flowspace rules:
    188 
    189 {{{
    190 ssh $fvserver fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo $fv_slicename
    191 ssh $fvserver fvctl --passwd-file=/etc/flowvisor/fvpasswd listFlowSpace | grep $fv_slicename
    192 }}}
    193 
    194 (Don't worry about the ouptut saying that it can't connect to the controller; this is expected if your rspec didn't include a controller that's actually running, which the example one doesn't.)
    195 
    196 Delete the sliver:
    197 
    198 {{{
    199 omni -a $am deletesliver $slicename
    200 }}}
    201 
    202 (That should generate a "sliver deleted" e-mail message, one to the experimenter and one to the FOAM admin.)
    203 
    204 Confirm that Omni thinks it's gone:
    205 
    206 {{{
    207 omni -a $am listresources $slicename
    208 omni -a $am sliverstatus $slicename
    209 }}}
    210 
    211 (Expect an "unknown slice" error for each of those.)
    212 
    213 Confirm that the sliver no longer shows up in FOAM:
    214 
    215 {{{
    216 ssh $foamserver foamctl geni:show-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
    217 ssh $foamserver foamctl geni:list-slivers --passwd-file=/opt/foam/etc/foampasswd | grep $slice_urn
    218 }}}
    219 
    220 (Expect 'deleted' to be "True" for the first, and no output for the second. For the first, note that 'status' will still be "Approved", which is fine, it's just tracking what the status was when it was deleted.)
    221 
    222 Confirm that the sliver no longer shows up in FlowVisor:
    223 
    224 {{{
    225 ssh $fvserver fvctl --passwd-file=/etc/flowvisor/fvpasswd getSliceInfo $fv_slicename
    226 ssh $fvserver fvctl --passwd-file=/etc/flowvisor/fvpasswd listFlowSpace | grep $fv_slicename
    227 }}}
    228 
    229 (Expect a "slice does not exist" error for the first, and no output for the second.)
    230 
    231 == Complex management ==
    232 
    233 NOTE: Do the [wiki:OpenFlow/FOAM/Testing#Creation Creation] subsection immediately before starting this subsection.
    234 
    235 The steps in this subsection look at a newly-created sliver, run through a sequence of approve/disable/reject actions to confirm that it can transition from any of those states to the others, and check the FlowVisor after each change; then delete the sliver, and confirm that it vanishes from the FlowVisor.
    236 
    237 Confirm that the sliver was approved at creation time: Check that 'status' is "Approved", and that the sliver shows up in FlowVisor, with two flowspace rules:
    238 
    239 {{{
    240 ssh $foamserver foam-confirm-sliver $sliver_urn status approved
    241 ssh $fvserver flowvisor-confirm-fvslice-exists $fv_slicename true
    242 ssh $fvserver flowvisor-confirm-flowspace-rule-count $fv_slicename 2
    243 }}}
    244 
    245119Add a temporary cron job to tell FOAM to generate e-mail with a list of pending slivers:
    246120
    … …  
    249123}}}
    250124
    251 (Within a minute (when the cron job fires), that should generate a message to www-data saying that the cron job ran, and one of two other things will happen: (a) If there are *other* pending slivers, it should generate a message to the FOAM admin listing only those slivers, and not your test sliver; OR (b) If there are no other pending slivers, it should not send any message about pending slivers to the FOAM admin.)
    252 
    253 Disable the sliver, to put it back into the pending queue:
     125(Within a minute (when the cron job fires), that should generate a message to www-data saying that the cron job ran, and one of two other things will happen: (a) If there are *other* pending slivers, it should generate a message to the FOAM admin listing only those slivers, and not your test sliver; OR (b) If there are no other pending slivers, it should not send any message to the FOAM admin about pending slivers.)
     126
     127Disable the sliver (to put it back into the pending queue), confirm that 'status' is now "Pending", and confirm that the sliver now does not show up in FlowVisor, and has zero flowspace rules:
    254128
    255129{{{
    256130ssh $foamserver foamctl geni:disable-sliver -u $sliver_urn --passwd-file=/opt/foam/etc/foampasswd
    257 }}}
    258 
    259 (That should generate a "sliver disabled" e-mail message, one to the experimenter and one to the FOAM admin.)
    260 
    261 Confirm that 'status' is now "Pending", and that the sliver now does not show up in FlowVisor, and has no flowspace rules:
    262 
    263 {{{
    264131ssh $foamserver foam-confirm-sliver $sliver_urn status pending
    265132ssh $fvserver flowvisor-confirm-fvslice-exists $fv_slicename false
    266133ssh $fvserver flowvisor-confirm-flowspace-rule-count $fv_slicename 0
    267134}}}
     135
     136(That should generate a "sliver disabled" e-mail message, one to the experimenter and one to the FOAM admin.)
    268137
    269138Add a temporary cron job to tell FOAM to generate e-mail with a list of pending slivers:
    … …  
    292161}}}
    293162
    294 Reject the sliver, confirm that 'status' is now "Rejected", and confirm that the sliver now does not show up in FlowVisor, and has no flowspace rules:
     163Reject the sliver, confirm that 'status' is now "Rejected", and confirm that the sliver now does not show up in FlowVisor, and has zero flowspace rules:
    295164
    296165{{{
    … …  
    301170}}}
    302171
    303 Disable the sliver, confirm that 'status' is now "Pending", and confirm that the sliver now does not show up in FlowVisor, and has no flowspace rules:
     172Disable the sliver, confirm that 'status' is now "Pending", and confirm that the sliver now does not show up in FlowVisor, and has zero flowspace rules:
    304173
    305174{{{
    … …  
    310179}}}
    311180
    312 Reject the sliver, confirm that 'status' is now "Rejected", and confirm that the sliver now does not show up in FlowVisor, and has no flowspace rules:
     181Reject the sliver, confirm that 'status' is now "Rejected", and confirm that the sliver now does not show up in FlowVisor, and has zero flowspace rules:
    313182
    314183{{{
    … …  
    328197}}}
    329198
    330 Delete the sliver, confirm that 'deleted' is now "True", and confirm that the sliver now does not show up in FlowVisor, and has no flowspace rules:
     199Delete the sliver, confirm that 'deleted' is now "True", and confirm that the sliver now does not show up in FlowVisor, and has zero flowspace rules:
    331200
    332201{{{
    … …  
    337206}}}
    338207
    339 That set of tests should generate six e-mail messages to the experimenter and six to the FOAM admin: There should be a "sliver approved" message, then "sliver rejected", then "sliver disabled", then rejected, approved, and deleted. Probably in that order, although e-mail is sometimes delivered out of sequence, so don't worry too much if they're not in order.
     208That set of tests should generate six e-mail messages to the experimenter and six to the FOAM admin: There should be a "sliver approved" message, then "sliver rejected", then "sliver disabled", then "rejected", then "approved", and then "deleted". Probably in that order, although e-mail is sometimes delivered out of sequence, so don't worry too much if they're not in order.
    340209
    341210== Expiration ==
    342211
    343 NOTE: Do the [wiki:OpenFlow/FOAM/Testing#Creation Creation] subsection immediately before starting this subsection.
    344 
    345 The steps in this subsection look at a newly-created sliver, and test expiration in FOAM, by changing its expiration date and, then running the scripts that expire slivers and notify about soon-to-expire ones.
    346 
    347 Confirm that the sliver was approved at creation time: Check that 'status' is "Approved", and that the sliver shows up in FlowVisor, with two flowspace rules:
    348 
    349 {{{
    350 ssh $foamserver foam-confirm-sliver $sliver_urn status approved
    351 ssh $fvserver flowvisor-confirm-fvslice-exists $fv_slicename true
    352 ssh $fvserver flowvisor-confirm-flowspace-rule-count $fv_slicename 2
    353 }}}
    354 
    355 Now, do some tests to make sure FOAM notifies you about a sliver that will expire within a day, but only once.
     212This subsection runs creates a sliver, and tests the sliver expiration process in FOAM, by changing the sliver's expiration date, and then running the scripts that expire slivers and notify about soon-to-expire ones, for a variety of expiration dates.
     213
     214In this subsection, we explicitly identify how many e-mail messages of each type we expect to receive after each run of the expiration scripts, since those notifications are what we're testing here, not just a side effect. The sliver-related ones from FOAM should be sent to the sliver owner and to the FOAM admin; the cron job confirmations should be sent to the user that the cron job runs as ('www-data' in these examples).
     215
     216Create and verify the sliver:
     217
     218{{{
     219foam-create-and-verify-sliver -q $foamserver $fvserver $am $slicename $rspec approved true 2
     220}}}
     221
     222(That should generate a "sliver created" and a "sliver approved" e-mail message, one of each to the experimenter and one of each to the FOAM admin.)
     223
     224Get the sliver URN and FlowVisor slicename from FOAM, and put them into $sliver_urn and $fv_slicename:
     225
     226{{{
     227sliver_urn=$(ssh $foamserver foamctl geni:list-slivers --passwd-file=/opt/foam/etc/foampasswd | egrep sliver_urn.+$slice_urn_egrep | sed -e 's/ *"sliver_urn": "\(.*\)".*/\1/')
     228fv_slicename=$(echo $sliver_urn | awk -F : '{print $NF}')
     229}}}
     230
     231First, test that FOAM notifies you about a sliver that will expire within a day, but only once.
    356232
    357233Change the expiration date of your slice to be nine days away, since otherwise you can't extend the sliver expiration date very far:
    … …  
    364240
    365241{{{
    366 omni -a $am renewsliver $slicename "$(date -d 'now + 12 hours')"
    367 omni -a $am sliverstatus $slicename
     242omni -a $am renewsliver $slicename "$(date -d 'now + 12 hours')" > /dev/null 2>&1
     243omni -a $am sliverstatus $slicename 2>&1 | grep foam_expires
    368244}}}
    369245
    370246(That should generate a "sliver renewed" e-mail message, one to the experimenter and one to the FOAM admin.)
    371247
    372 Add a temporary cron job to tell FOAM to send mail about slivers that are expiring soon and remove any expired slivers:
     248Add a temporary cron job to tell FOAM to send mail about slivers that are expiring in the near future, and remove any expired slivers:
    373249
    374250{{{
    … …  
    376252}}}
    377253
    378 (Within a minute (when the cron job fires), that should generate a "sliver expiring shortly" e-mail message, one to the experimenter and one to the FOAM admin; no messages about expired slivers being deleted; and one message to www-data saying that the cron job ran.)
     254(Within a minute (when the cron job fires), expected e-mail: 1 "sliver expiring shortly", 0 "sliver expired", and confirmation that the cron job ran.)
    379255
    380256Add the cron job again, to confirm that it doesn't send a second notification:
    … …  
    384260}}}
    385261
    386 (Within a minute (when the cron job fires), that should generate no messages about slivers expiring in the future; no messages about expired slivers being deleted; and one message to www-data saying that the cron job ran.)
     262(Within a minute (when the cron job fires), expected e-mail: 0 "sliver expiring", 0 "sliver expired", only confirmation that the cron job ran.)
    387263
    388264Next, test to make sure FOAM notifies you again if you change the expiration date.
    … …  
    391267
    392268{{{
    393 omni -a $am renewsliver $slicename "$(date -d 'now + 1 day')"
    394 omni -a $am sliverstatus $slicename
     269omni -a $am renewsliver $slicename "$(date -d 'now + 1 day')" > /dev/null 2>&1
     270omni -a $am sliverstatus $slicename 2>&1 | grep foam_expires
    395271}}}
    396272
    … …  
    403279}}}
    404280
    405 (Within a minute (when the cron job fires), that should generate a "sliver expiring shortly" e-mail message, one to the experimenter and one to the FOAM admin; no messages about expired slivers being deleted; and one message to www-data saying that the cron job ran.)
     281(Within a minute (when the cron job fires), expected e-mail: 1 "sliver expiring shortly", 0 "sliver expired", and confirmation that the cron job ran.)
    406282
    407283Next, do some tests to make sure that FOAM notifies you about a sliver that will expire within a week, but only once.
    … …  
    410286
    411287{{{
    412 omni -a $am renewsliver $slicename "$(date -d 'now + 2 days')"
    413 omni -a $am sliverstatus $slicename
     288omni -a $am renewsliver $slicename "$(date -d 'now + 2 days')" > /dev/null 2>&1
     289omni -a $am sliverstatus $slicename 2>&1 | grep foam_expires
    414290}}}
    415291
    … …  
    422298}}}
    423299
    424 (Within a minute (when the cron job fires), that should generate a "sliver expiring in about a week" e-mail message, one to the experimenter and one to the FOAM admin; no messages about expired slivers being deleted; and one message to www-data saying that the cron job ran.)
     300(Within a minute (when the cron job fires), expected e-mail: 1 "sliver expiring within a week", 0 "sliver expired", and confirmation that the cron job ran.)
    425301
    426302Add the cron job again, to confirm that it doesn't send a second notification:
    … …  
    430306}}}
    431307
    432 (Within a minute (when the cron job fires), that should generate no messages about slivers expiring in the future; no messages about expired slivers being deleted; and one message to www-data saying that the cron job ran.)
     308(Within a minute (when the cron job fires), expected e-mail: 0 "sliver expiring", 0 "sliver expired", only confirmation that the cron job ran.)
    433309
    434310Next, test to make sure FOAM doesn't notify you at all if your sliver is expiring more than a week from now.
    … …  
    437313
    438314{{{
    439 omni -a $am renewsliver $slicename "$(date -d 'now + 8 days')"
    440 omni -a $am sliverstatus $slicename
     315omni -a $am renewsliver $slicename "$(date -d 'now + 8 days')" > /dev/null 2>&1
     316omni -a $am sliverstatus $slicename 2>&1 | grep foam_expires
    441317}}}
    442318
    … …  
    449325}}}
    450326
    451 (Within a minute (when the cron job fires), that should generate no messages about slivers expiring in the future; no messages about expired slivers being deleted; and one message to www-data saying that the cron job ran.)
     327(Within a minute (when the cron job fires), expected e-mail: 0 "sliver expiring", 0 "sliver expired", only confirmation that the cron job ran.)
    452328
    453329Finally, test to make sure FOAM deletes a sliver if its expiration date is in the past.
    … …  
    456332
    457333{{{
    458 omni -a $am renewsliver $slicename "$(date -d 'now')"
    459 omni -a $am sliverstatus $slicename
     334omni -a $am renewsliver $slicename "$(date -d 'now')" > /dev/null 2>&1
     335omni -a $am sliverstatus $slicename 2>&1 | grep foam_expires
    460336}}}
    461337
    … …  
    468344}}}
    469345
    470 (Within a minute (when the cron job fires), that should generate no messages about slivers expiring in the future; a "sliver expired" message, one to the experimenter and one to the FOAM admin; and one message to www-data saying that the cron job ran.)
    471 
    472 Confirm that the sliver was deleted, does not show up in FlowVisor, and has no flowspace rules::
     346(Within a minute (when the cron job fires), expected e-mail: 0 "sliver expiring", 1 "sliver expired", and confirmation that the cron job ran.)
     347
     348Confirm that the sliver was deleted, does not show up in FlowVisor, and has zero flowspace rules::
    473349
    474350{{{
    … …  
    501377 * On the system where you're running commands, you have the foam-create-and-verify-sliver script.
    502378
    503 Note that the scripts mentioned in the last two items are currently GPO-only. (GPO folks, they're in the syseng repo, under gposw.)
     379Note that the scripts mentioned in the last two items are currently GPO-only. (GPO folks, they're in the syseng repo, under gposw.) We're working on making them public.
    504380
    505381You can run these tests wherever you usually use Omni; as written, they use SSH to run remote commands on the FOAM/FV server.
    … …  
    878754NOTE that this assumes that you have datapaths and port groups defined in your FOAM server, which correspond to the datapaths and port groups in the rspec. FIXME: Need to document how to do this with fake datapaths.
    879755
     756This set of tests doesn't need any additional setup.
     757
    880758One port in a port group, and one that isn't - should be approved:
    881759
    … …  
    916794
    917795This set of tests uses rspecs containing things that should never be auto-approved. (Currently, only one of them, an rspec containing an entire VLAN.)
     796
     797This set of tests doesn't need any additional setup.
    918798
    919799Create and verify the sliver: