[[PageOutline]]

= Detailed test plan for IG-MON-5: GMOC Data Collection Test =

''This page is GPO's working page for performing IG-MON-5.  It is public for informational purposes, but it is not an official status report.  See [wiki:GENIRacksHome/InstageniRacks/AcceptanceTestStatus] for the current status of InstaGENI acceptance tests.''

''Last substantive edit of this page: 2013-02-28''

== Status of test ==

|| '''Step''' || '''State'''                                || '''Date completed''' || '''Open Tickets''' || '''Closed Tickets/Comments''' ||
|| 1          || [[Color(lightgreen,Pass: most criteria)]] || 2013-03-15           ||                    || relational data was successfully examined and looks good, with the exception of a few outstanding issues ||
|| 2A         || No longer planned                          ||                      ||                    ||                               ||
|| 2B         || No longer planned                          ||                      ||                    ||                               ||
|| 3          || No longer planned                          ||                      ||                    ||                               ||
|| 4          || No longer planned                          ||                      ||                    ||                               ||
|| 5          || No longer planned                          ||                      ||                    ||                               ||
|| 6          || No longer planned                          ||                      ||                    ||                               ||
|| 7          || No longer planned                          ||                      ||                    ||                               ||

== High-level description from test plan ==

This test verifies the rack's submission of monitoring data to GMOC.

=== Procedure ===

View the dataset collected at GMOC for the BBN and Utah InstaGENI racks. For each piece of required data, attempt to verify that:

 * The data is being collected and accepted by GMOC and can be viewed at gmoc-db.grnoc.iu.edu
 * The data's "site" tag indicates that it is being reported for the rack located at the gpolab or Utah site (as appropriate for that rack).
 * The data has been reported within the past 10 minutes.
 * For each piece of data, either verify that it is being collected at least once a minute, or verify that it requires more complicated processing than a simple file read to collect, and thus can be collected less often. 

Verify that the following pieces of data are being reported:

 * Is each of the rack InstaGENI and FOAM AMs reachable via the GENI AM API right now?
 * Is each compute or unbound VLAN resource at each rack AM online? Is it available or in use?
 * Sliver count and percentage of rack compute and unbound VLAN resources in use.
 * Identities of current slivers on each rack AM, including creation time for each.
 * Per-sliver interface counters for compute and VLAN resources (where these values can be easily collected).
 * Is the rack data plane switch online?
 * Interface counters and VLAN memberships for each rack data plane switch interface
 * MAC address table contents for shared VLANs which appear on rack data plane switches
 * Is each rack experimental node online?
 * For each rack experimental node configured as an OpenVZ VM server, overall CPU, disk, and memory utilization for the host, current VM count and total VM capacity of the host.
 * For each rack experimental node configured as an OpenVZ VM server, interface counters for each data plane interface.
 * Results of at least one end-to-end health check which simulates an experimenter reserving and using at least one resource in the rack. 

Verify that per-rack or per-aggregate summaries are collected of the count of distinct users who have been active on the rack, either by providing raw sliver data containing sliver users to GMOC, or by collecting data locally and producing trending summaries on demand. 

== Step 1: test all relational data available for examination as of GEC16 ==

This was an ad-hoc test used to check the results of relational data downloaded via gmoc.py from gmoc-db.  Because measurement data is not being submitted, and is not easy to map to relational data using the current gmoc.py, i checked relational data only.

=== Results of testing: 2013-03-15 ===

This test was performed on tikal (monitor.gpolab.bbn.com), which has an automated process which downloads all relational data from known GENI aggregates.
 * The procedure used to get the data was:
{{{
import pickle
import sys
sys.path.append('/usr/local/lib')
f = open('/var/lib/ganglia/gmoc-report/gmoc-relations.pck')
r = pickle.load(f)
f.close()
}}}
 * The data was downloaded finishing at 12:57 on 2013-03-15:
{{{
tikal,[~],13:01(0)$ ls -l /var/lib/ganglia/gmoc-report/gmoc-relations.pck 
-rw-r--r-- 2 ganglia ganglia 658573 Mar 15 12:57 /var/lib/ganglia/gmoc-report/gmoc-relations.pck
tikal,[~],13:01(0)$ date
Fri Mar 15 13:01:40 EDT 2013
}}}
 * Aggregates, resources, and slivers of interest for this test were pulled into variables:
{{{
bbnig = r['aggregate']['instageni.gpolab.bbn.com:12369']
uthig = r['aggregate']['utah.geniracks.net:12369']
nrtig = r['aggregate']['instageni.northwestern.edu:12369']
ukyig = r['aggregate']['lan.sdn.uky.edu:12369']
bbnfoam = r['aggregate']['foam.instageni.gpolab.bbn.com:3626']
uthfoam = r['aggregate']['foam.utah.geniracks.net:3626']
ig_ams = [bbnig, uthig, nrtig, ukyig]
slivers = []
resources = []
for am in ig_ams:
  resources.extend(am.resources)
  slivers.extend(am.slivers)
}}}

Having gotten the data, i ran some tests on it.

==== Check parameters of InstaGENI aggregate objects ====

Do the reported parameters (id, type, version, operator, pop) of each aggregate look like we expect?  Yes:
{{{
>>> print "\n".join([a.id for a in ig_ams])
instageni.gpolab.bbn.com:12369
utah.geniracks.net:12369
instageni.northwestern.edu:12369
lan.sdn.uky.edu:12369

>>> print "\n".join([a.type for a in ig_ams])
protogeni
protogeni
protogeni
protogeni

>>> print "\n".join([a.version for a in ig_ams])
7e584af01b968c18b7d87fe7e78a5dde3549c47f
7e584af01b968c18b7d87fe7e78a5dde3549c47f
7e584af01b968c18b7d87fe7e78a5dde3549c47f
7e584af01b968c18b7d87fe7e78a5dde3549c47f

>>> print "\n".join([a.operator.id for a in ig_ams])
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI

>>> print "\n".join([a.pop.id for a in ig_ams])
urn:publicid:IDN+gmoc.geni.net+pop+gpolab
urn:publicid:IDN+gmoc.geni.net+pop+Utah
urn:publicid:IDN+gmoc.geni.net+pop+Northwestern
urn:publicid:IDN+gmoc.geni.net+pop+Kentucky
}}}

==== Check parameters of InstaGENI resource objects ====

Do the reported parameters (id, type, aggregate, operator, pop, state) of each aggregate look like we expect?:
{{{
>>> print "\n".join([r.id for r in resources])
urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc4
urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc5
urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc1
urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc2
urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc3
urn:publicid:IDN+utah.geniracks.net+node+pc2
urn:publicid:IDN+utah.geniracks.net+node+pc3
urn:publicid:IDN+utah.geniracks.net+node+pc1
urn:publicid:IDN+utah.geniracks.net+node+pc4
urn:publicid:IDN+utah.geniracks.net+node+pc5
urn:publicid:IDN+instageni.northwestern.edu+node+pc3
urn:publicid:IDN+instageni.northwestern.edu+node+pc2
urn:publicid:IDN+instageni.northwestern.edu+node+pc1
urn:publicid:IDN+instageni.northwestern.edu+node+pc5
urn:publicid:IDN+instageni.northwestern.edu+node+pc4
urn:publicid:IDN+lan.sdn.uky.edu+node+pc4
urn:publicid:IDN+lan.sdn.uky.edu+node+pc5
urn:publicid:IDN+lan.sdn.uky.edu+node+pc1
urn:publicid:IDN+lan.sdn.uky.edu+node+pc2
urn:publicid:IDN+lan.sdn.uky.edu+node+pc3

>>> print "\n".join([r.type for r in resources])
host
host
vmserver
vmserver
host
host
vmserver
vmserver
host
vmserver
host
host
vmserver
host
host
host
host
vmserver
vmserver
host

>>> print "\n".join([r.operator.id for r in resources])
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI
urn:publicid:IDN+gmoc.geni.net+organization+InstaGENI

>>> print "\n".join([r.pop.id for r in resources])
urn:publicid:IDN+gmoc.geni.net+pop+gpolab
urn:publicid:IDN+gmoc.geni.net+pop+gpolab
urn:publicid:IDN+gmoc.geni.net+pop+gpolab
urn:publicid:IDN+gmoc.geni.net+pop+gpolab
urn:publicid:IDN+gmoc.geni.net+pop+gpolab
urn:publicid:IDN+gmoc.geni.net+pop+Utah
urn:publicid:IDN+gmoc.geni.net+pop+Utah
urn:publicid:IDN+gmoc.geni.net+pop+Utah
urn:publicid:IDN+gmoc.geni.net+pop+Utah
urn:publicid:IDN+gmoc.geni.net+pop+Utah
urn:publicid:IDN+gmoc.geni.net+pop+Northwestern
urn:publicid:IDN+gmoc.geni.net+pop+Northwestern
urn:publicid:IDN+gmoc.geni.net+pop+Northwestern
urn:publicid:IDN+gmoc.geni.net+pop+Northwestern
urn:publicid:IDN+gmoc.geni.net+pop+Northwestern
urn:publicid:IDN+gmoc.geni.net+pop+Kentucky
urn:publicid:IDN+gmoc.geni.net+pop+Kentucky
urn:publicid:IDN+gmoc.geni.net+pop+Kentucky
urn:publicid:IDN+gmoc.geni.net+pop+Kentucky
urn:publicid:IDN+gmoc.geni.net+pop+Kentucky

>>> print "\n".join([r.state for r in resources])
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
}}}

==== Check parameters of InstaGENI resource interface objects ====

Collect interface objects:
{{{
interfaces = []
for resource in resources:
  interfaces.extend(resource.interfaces)
}}}

Do the reported parameters (id, IPv4 addresses, MAC addresses, VLANs) of each aggregate look like we expect?:
{{{
>>> print "\n".join([i.id for i in interfaces])
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc4:eth99
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc4:eth1
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc4:eth3
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc4:eth0
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc4:eth2
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc5:eth99
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc5:eth1
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc5:eth3
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc5:eth0
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc5:eth2
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc1:eth99
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc1:eth1
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc1:eth3
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc1:eth0
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc1:eth2
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc2:eth99
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc2:eth1
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc2:eth3
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc2:eth0
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc2:eth2
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc3:eth99
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc3:eth1
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc3:eth3
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc3:eth0
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc3:eth2
urn:publicid:IDN+utah.geniracks.net+interface+pc2:eth1
urn:publicid:IDN+utah.geniracks.net+interface+pc2:eth3
urn:publicid:IDN+utah.geniracks.net+interface+pc2:eth0
urn:publicid:IDN+utah.geniracks.net+interface+pc2:eth2
urn:publicid:IDN+utah.geniracks.net+interface+pc2:ilo3
urn:publicid:IDN+utah.geniracks.net+interface+pc3:eth1
urn:publicid:IDN+utah.geniracks.net+interface+pc3:eth3
urn:publicid:IDN+utah.geniracks.net+interface+pc3:eth0
urn:publicid:IDN+utah.geniracks.net+interface+pc3:eth2
urn:publicid:IDN+utah.geniracks.net+interface+pc3:ilo3
urn:publicid:IDN+utah.geniracks.net+interface+pc1:eth0
urn:publicid:IDN+utah.geniracks.net+interface+pc1:eth1
urn:publicid:IDN+utah.geniracks.net+interface+pc1:eth3
urn:publicid:IDN+utah.geniracks.net+interface+pc1:eth2
urn:publicid:IDN+utah.geniracks.net+interface+pc1:ilo3
urn:publicid:IDN+utah.geniracks.net+interface+pc4:eth1
urn:publicid:IDN+utah.geniracks.net+interface+pc4:eth3
urn:publicid:IDN+utah.geniracks.net+interface+pc4:eth0
urn:publicid:IDN+utah.geniracks.net+interface+pc4:eth2
urn:publicid:IDN+utah.geniracks.net+interface+pc4:ilo3
urn:publicid:IDN+utah.geniracks.net+interface+pc5:eth1
urn:publicid:IDN+utah.geniracks.net+interface+pc5:eth3
urn:publicid:IDN+utah.geniracks.net+interface+pc5:eth0
urn:publicid:IDN+utah.geniracks.net+interface+pc5:eth2
urn:publicid:IDN+utah.geniracks.net+interface+pc5:ilo3
urn:publicid:IDN+instageni.northwestern.edu+interface+pc3:eth99
urn:publicid:IDN+instageni.northwestern.edu+interface+pc3:eth1
urn:publicid:IDN+instageni.northwestern.edu+interface+pc3:eth3
urn:publicid:IDN+instageni.northwestern.edu+interface+pc3:eth0
urn:publicid:IDN+instageni.northwestern.edu+interface+pc3:eth2
urn:publicid:IDN+instageni.northwestern.edu+interface+pc2:eth99
urn:publicid:IDN+instageni.northwestern.edu+interface+pc2:eth1
urn:publicid:IDN+instageni.northwestern.edu+interface+pc2:eth3
urn:publicid:IDN+instageni.northwestern.edu+interface+pc2:eth0
urn:publicid:IDN+instageni.northwestern.edu+interface+pc2:eth2
urn:publicid:IDN+instageni.northwestern.edu+interface+pc1:eth99
urn:publicid:IDN+instageni.northwestern.edu+interface+pc1:eth1
urn:publicid:IDN+instageni.northwestern.edu+interface+pc1:eth3
urn:publicid:IDN+instageni.northwestern.edu+interface+pc1:eth0
urn:publicid:IDN+instageni.northwestern.edu+interface+pc1:eth2
urn:publicid:IDN+instageni.northwestern.edu+interface+pc5:eth99
urn:publicid:IDN+instageni.northwestern.edu+interface+pc5:eth1
urn:publicid:IDN+instageni.northwestern.edu+interface+pc5:eth3
urn:publicid:IDN+instageni.northwestern.edu+interface+pc5:eth0
urn:publicid:IDN+instageni.northwestern.edu+interface+pc5:eth2
urn:publicid:IDN+instageni.northwestern.edu+interface+pc4:eth99
urn:publicid:IDN+instageni.northwestern.edu+interface+pc4:eth1
urn:publicid:IDN+instageni.northwestern.edu+interface+pc4:eth3
urn:publicid:IDN+instageni.northwestern.edu+interface+pc4:eth0
urn:publicid:IDN+instageni.northwestern.edu+interface+pc4:eth2
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc4:eth99
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc4:eth1
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc4:eth3
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc4:eth0
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc4:eth2
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc5:eth99
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc5:eth1
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc5:eth3
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc5:eth0
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc5:eth2
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc1:eth99
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc1:eth1
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc1:eth3
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc1:eth0
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc1:eth2
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc2:eth99
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc2:eth1
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc2:eth3
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc2:eth0
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc2:eth2
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc3:eth99
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc3:eth1
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc3:eth3
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc3:eth0
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc3:eth2

>>> for i in interfaces:
...   if len(i.vlans) > 0:
...     print "%s: %s" % (i.id, " ".join(i.vlans))
... 

>>> for i in interfaces:
...   if len(i.addresses) > 0:
...     print "%s: %s" % (i.id, " ".join([ad.addr for ad in i.addresses]))
... 
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc4:eth0: 192.1.242.143
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc5:eth0: 192.1.242.144
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc1:eth0: 192.1.242.140
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc2:eth0: 192.1.242.141
urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc3:eth0: 192.1.242.142
urn:publicid:IDN+utah.geniracks.net+interface+pc2:eth0: 155.98.34.12
urn:publicid:IDN+utah.geniracks.net+interface+pc3:eth0: 155.98.34.13
urn:publicid:IDN+utah.geniracks.net+interface+pc1:eth0: 155.98.34.11
urn:publicid:IDN+utah.geniracks.net+interface+pc4:eth0: 155.98.34.14
urn:publicid:IDN+utah.geniracks.net+interface+pc5:eth0: 155.98.34.15
urn:publicid:IDN+instageni.northwestern.edu+interface+pc3:eth0: 165.124.51.142
urn:publicid:IDN+instageni.northwestern.edu+interface+pc2:eth0: 165.124.51.141
urn:publicid:IDN+instageni.northwestern.edu+interface+pc1:eth0: 165.124.51.140
urn:publicid:IDN+instageni.northwestern.edu+interface+pc5:eth0: 165.124.51.144
urn:publicid:IDN+instageni.northwestern.edu+interface+pc4:eth0: 165.124.51.143
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc4:eth0: 128.163.232.19
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc5:eth0: 128.163.232.20
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc1:eth0: 128.163.232.16
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc2:eth0: 128.163.232.17
urn:publicid:IDN+lan.sdn.uky.edu+interface+pc3:eth0: 128.163.232.18
}}}

==== Check parameters of InstaGENI sliver objects ====

Do the reported parameters (id, slice URN, sliver UUID, creator, created, expires, state) of each sliver look like we expect?:
{{{
>>> print "\n".join([s.id for s in slivers])
urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+211
urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2003
urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2204
urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2250
urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2336
urn:publicid:IDN+utah.geniracks.net+sliver+11075
urn:publicid:IDN+utah.geniracks.net+sliver+13957
urn:publicid:IDN+utah.geniracks.net+sliver+17299
urn:publicid:IDN+utah.geniracks.net+sliver+17303
urn:publicid:IDN+utah.geniracks.net+sliver+17319
urn:publicid:IDN+utah.geniracks.net+sliver+17471
urn:publicid:IDN+instageni.northwestern.edu+sliver+10
urn:publicid:IDN+instageni.northwestern.edu+sliver+109
urn:publicid:IDN+instageni.northwestern.edu+sliver+116
urn:publicid:IDN+instageni.northwestern.edu+sliver+183
urn:publicid:IDN+instageni.northwestern.edu+sliver+187
urn:publicid:IDN+lan.sdn.uky.edu+sliver+172
urn:publicid:IDN+lan.sdn.uky.edu+sliver+208
urn:publicid:IDN+lan.sdn.uky.edu+sliver+332
urn:publicid:IDN+lan.sdn.uky.edu+sliver+336
urn:publicid:IDN+lan.sdn.uky.edu+sliver+599
urn:publicid:IDN+lan.sdn.uky.edu+sliver+796
urn:publicid:IDN+lan.sdn.uky.edu+sliver+798

>>> print "\n".join([s.slice.id for s in slivers])
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tuptymon
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs16
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs15
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tuptyl2pingdebug
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+3sites-OF
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tupty1755
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tuptymon
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs16
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs15
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+transgeo
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tuptyl2pingdebug
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tupty1755
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tuptymon
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+transgeo
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs15
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs16
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tupty1755
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tuptymon
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs15
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs16
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+magicwnd
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+acclne-1439311
urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+acclne-1439312

>>> print "\n".join([str(s.uuid) for s in slivers])
73a1c54c-56ae-11e2-bfa6-029e26f15299
c130641b-85ca-11e2-bfa6-029e26f15299
e701342e-876a-11e2-bfa6-029e26f15299
0e8ae133-8824-11e2-bfa6-029e26f15299
c9bd91c5-8ca8-11e2-921f-029e26f15299
3487385f-0981-11e2-8336-00009b6224df
f02cb7a8-1de1-11e2-8336-00009b6224df
7fe8c7f7-85c9-11e2-b374-00009b6224df
10dfc3e9-85ca-11e2-b374-00009b6224df
5a65fb87-85e9-11e2-b374-00009b6224df
39ce1f4c-8824-11e2-b374-00009b6224df
16b4a9ea-771a-11e2-bc42-02d6cd042885
f97c6dc4-7d12-11e2-bc43-02d6cd042885
74533ef6-7d19-11e2-bc43-02d6cd042885
cde625c7-8548-11e2-bc43-02d6cd042885
d0cbbbeb-854c-11e2-bc43-02d6cd042885
7aac9536-7d0a-11e2-9d0e-025e430ef5d2
a57d7b47-7d35-11e2-9d0e-025e430ef5d2
857bdc49-85c8-11e2-b51f-025e430ef5d2
18e38445-85c9-11e2-b51f-025e430ef5d2
7cb79944-8b5f-11e2-b51f-025e430ef5d2
3de2b862-8cb5-11e2-b51f-025e430ef5d2
560ae43f-8cb5-11e2-b51f-025e430ef5d2

>>> print "\n".join([s.creator.id for s in slivers])
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+tupty
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+jbs
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+jbs
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+tupty
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+tupty
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+tupty
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+jbs
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+jbs
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+inki
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+tupty
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+tupty
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+tupty
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+inki
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+jbs
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+jbs
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+tupty
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+tupty
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+jbs
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+jbs
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+inki
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers
urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers

>>> print "\n".join(["created: %s, expires: %s" % (s.created, s.expires) for s in slivers])
created: 2013-01-04 13:36:49, expires: 2013-03-22 12:50:41
created: 2013-03-05 12:27:49, expires: 2013-03-22 12:50:41
created: 2013-03-07 14:06:44, expires: 2013-03-22 12:50:41
created: 2013-03-08 12:12:07, expires: 2013-03-21 20:00:31
created: 2013-03-14 07:12:19, expires: 2013-03-22 12:50:41
created: 2012-09-28 09:28:58, expires: 2013-03-22 12:50:41
created: 2012-10-24 07:51:47, expires: 2013-03-22 12:50:41
created: 2013-03-05 12:18:51, expires: 2013-03-22 12:50:41
created: 2013-03-05 12:22:55, expires: 2013-03-22 12:50:41
created: 2013-03-05 16:06:54, expires: 2013-03-22 12:50:41
created: 2013-03-08 12:13:20, expires: 2013-03-21 20:00:31
created: 2013-02-14 19:47:56, expires: 2013-03-22 12:50:41
created: 2013-02-22 10:12:07, expires: 2013-03-22 12:50:41
created: 2013-02-22 10:58:32, expires: 2013-03-22 12:50:41
created: 2013-03-04 20:57:37, expires: 2013-03-22 12:50:41
created: 2013-03-04 21:26:19, expires: 2013-03-22 12:50:41
created: 2013-02-22 09:11:19, expires: 2013-03-22 12:50:41
created: 2013-02-22 14:20:19, expires: 2013-03-22 12:50:41
created: 2013-03-05 12:11:50, expires: 2013-03-22 12:50:41
created: 2013-03-05 12:15:58, expires: 2013-03-22 12:50:41
created: 2013-03-12 15:55:06, expires: 2013-03-22 12:50:41
created: 2013-03-14 08:41:28, expires: 2013-03-22 12:50:41
created: 2013-03-14 08:42:08, expires: 2013-03-22 12:50:41

>>> print "\n".join([s.state for s in slivers])
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
}}}
So, those expiration times look suspicious, and i checked with JBS about the eight slivers which are his, and he believes those reported times are in fact wrong.

==== Check parameters of InstaGENI sliver/resource mapping objects ====

Check whether each sliver has some resource_mappings (which we'd expect):
{{{
>>> print "\n".join([str(len(s.resource_mappings)) for s in slivers])
1
1
1
1
1
1
1
1
1
1
1
1
1
4
1
1
1
1
1
1
1
1
1
}}}

Collect the resource_mapping objects so we can look at them:
{{{
mappings = []
for s in slivers:
  mappings.extend(s.resource_mappings)
}}}

Do the reported parameters (type, sliver, resource) of each mapping look like we expect?:
{{{
>>> print "\n".join([str(m.type) for m in mappings])
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None
None

>>> print "\n".join([m.sliver.id for m in mappings])
urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+211
urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2003
urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2204
urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2250
urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2336
urn:publicid:IDN+utah.geniracks.net+sliver+11075
urn:publicid:IDN+utah.geniracks.net+sliver+13957
urn:publicid:IDN+utah.geniracks.net+sliver+17299
urn:publicid:IDN+utah.geniracks.net+sliver+17303
urn:publicid:IDN+utah.geniracks.net+sliver+17319
urn:publicid:IDN+utah.geniracks.net+sliver+17471
urn:publicid:IDN+instageni.northwestern.edu+sliver+10
urn:publicid:IDN+instageni.northwestern.edu+sliver+109
urn:publicid:IDN+instageni.northwestern.edu+sliver+116
urn:publicid:IDN+instageni.northwestern.edu+sliver+116
urn:publicid:IDN+instageni.northwestern.edu+sliver+116
urn:publicid:IDN+instageni.northwestern.edu+sliver+116
urn:publicid:IDN+instageni.northwestern.edu+sliver+183
urn:publicid:IDN+instageni.northwestern.edu+sliver+187
urn:publicid:IDN+lan.sdn.uky.edu+sliver+172
urn:publicid:IDN+lan.sdn.uky.edu+sliver+208
urn:publicid:IDN+lan.sdn.uky.edu+sliver+332
urn:publicid:IDN+lan.sdn.uky.edu+sliver+336
urn:publicid:IDN+lan.sdn.uky.edu+sliver+599
urn:publicid:IDN+lan.sdn.uky.edu+sliver+796
urn:publicid:IDN+lan.sdn.uky.edu+sliver+798

>>> print "\n".join([m.resource.id for m in mappings])                          
urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc1
urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc1
urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc2
urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc2
urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc1
urn:publicid:IDN+utah.geniracks.net+node+pc1
urn:publicid:IDN+utah.geniracks.net+node+pc1
urn:publicid:IDN+utah.geniracks.net+node+pc1
urn:publicid:IDN+utah.geniracks.net+node+pc5
urn:publicid:IDN+utah.geniracks.net+node+pc4
urn:publicid:IDN+utah.geniracks.net+node+pc5
urn:publicid:IDN+instageni.northwestern.edu+node+pc1
urn:publicid:IDN+instageni.northwestern.edu+node+pc1
urn:publicid:IDN+instageni.northwestern.edu+node+pc3
urn:publicid:IDN+instageni.northwestern.edu+node+pc2
urn:publicid:IDN+instageni.northwestern.edu+node+pc5
urn:publicid:IDN+instageni.northwestern.edu+node+pc4
urn:publicid:IDN+instageni.northwestern.edu+node+pc1
urn:publicid:IDN+instageni.northwestern.edu+node+pc1
urn:publicid:IDN+lan.sdn.uky.edu+node+pc1
urn:publicid:IDN+lan.sdn.uky.edu+node+pc2
urn:publicid:IDN+lan.sdn.uky.edu+node+pc1
urn:publicid:IDN+lan.sdn.uky.edu+node+pc2
urn:publicid:IDN+lan.sdn.uky.edu+node+pc5
urn:publicid:IDN+lan.sdn.uky.edu+node+pc1
urn:publicid:IDN+lan.sdn.uky.edu+node+pc1
}}}

As a sanity-check, i'm interested in checking the type of each node which has more than one sliver mapped to it --- i expect those would all be vmservers:
 * First make sure sliver->resources match up well with resource->slivers:
{{{
resource_slivers = 0
sliver_resources = 0
for r in resources:
  resource_slivers += len(r.slivers)

for s in slivers:
  sliver_resources += len(s.resources)

>>> print resource_slivers
26
>>> print sliver_resources
26
}}}
 * Now do our sanity-check:
{{{
>>> for r in resources:
...   if len(r.slivers) > 1:
...     print r.type
... 
vmserver
vmserver
vmserver
vmserver
vmserver
vmserver
vmserver

>>> for r in resources:
...   if len(r.slivers) == 1:
...     print r.type
... 
host
host
host
host
host
host
}}}

==== Summary of issues detected on 2013-03-15 ====

Things to discuss:
 * Need resource->state be "Unknown"?
 * Need sliver->state be "Unknown"?
 * Can we usefully report any VLAN information about interfaces?
 * Can we usefully report any non-control IPv4 address information about interfaces?
 * Can we report MAC address information about interfaces?
 * Should we have the emulab.net SA report slices, so that we can see IG slivers from emulab slices?
 * The reported sliver expiration times look suspicious and are probably wrong.
 * Resource mapping type should be something other than "None"

== Step 2: verify AM reachability data ==

=== Step 2A: verify GPO remote testing of AMs ===

==== Overview of step 2A ====

 * Look in recent-gmoc.xml for data reported by ashur.gpolab.bbn.com containing these tests:
   * geni_am_getversion
   * geni_am_listresources
   for these hosts:
   * instageni.gpolab.bbn.com ProtoGENI
   * foam.instageni.gpolab.bbn.com FOAM
   * utah.geniracks.net ProtoGENI
   * foam.utah.geniracks.net FOAM
 * Make sure each test has recent data (some non-null values)

=== Step 2B: verify aggregate self-reporting of any data ===

==== Overview of step 2B ====

In the python shell:
{{{
print uthig.last_updated
print uthfoam.last_updated
print bbnig.last_updated
print bbnfoam.last_updated
}}}
Each of those dates should be a timestamp which is recent relative to when the data was collected.

== Step 3: verify rack compute resource reporting ==

Fill in details of how to determine:
 * Is each compute or unbound VLAN resource at each rack AM online? Is it available or in use?
 * Is each rack experimental node online?
 * For each rack experimental node configured as an OpenVZ VM server, overall CPU, disk, and memory utilization for the host, current VM count and total VM capacity of the host.
 * Is the rack data plane switch online?

== Step 4: verify ProtoGENI aggregate sliver reporting ==

Fill in details of how to determine:
 * Sliver count and percentage of compute and unbound VLAN resources in use for the rack SM.
 * Identities of current slivers on each rack AM, including creation time for each.

== Step 5: verify interface counter reporting ==

Fill in details of how to determine:
 * Per-sliver interface counters for compute and VLAN resources (where these values can be easily collected).
 * Interface counters and VLAN memberships for each rack data plane switch interface
 * MAC address table contents for shared VLANs which appear on rack data plane switches
 * For each rack experimental node configured as an OpenVZ VM server, interface counters for each data plane interface.

== Step 6: verify end-to-end health check ==

 * Results of at least one end-to-end health check which simulates an experimenter reserving and using at least one resource in the rack. 

== Step 7: verify distinct user summarization ==

Verify that per-rack or per-aggregate summaries are collected of the count of distinct users who have been active on the rack, either by providing raw sliver data containing sliver users to GMOC, or by collecting data locally and producing trending summaries on demand.