ION looses circuit status information over time

[lnevers@bbn.com]

A stitching topology (lncube) was set up with 4 stitched links between two IG sites. A sliver status ran with 20 seconds sleep between calls from 11:06:37 to 12:02:52:

Circuits allocated:

"urn:publicid:IDN+ch.geni.net:ln-test+sliver+lncube_vlan_ion.internet2.edu-123861",
"urn:publicid:IDN+ch.geni.net:ln-test+sliver+lncube_vlan_ion.internet2.edu-123871",
"urn:publicid:IDN+ch.geni.net:ln-test+sliver+lncube_vlan_ion.internet2.edu-123881",
"urn:publicid:IDN+ch.geni.net:ln-test+sliver+lncube_vlan_ion.internet2.edu-123891",

Sliver status output shows:

Circuit 123861: Shows in all sliverstatus up to 12:01:00 but is missing
from the sliverstatus at:
    - 12:01:28
    - 12:01:56
    - 12:02:24
    - 12:02:52

Circuit 123871: Shows up in all sliverstatus except on at 11:59:34.

Circuit 123881: Missing from 11:15:51 to 11:27:22 and again missing for
one call at 11:27:22

Circuit 123891: Missing for one sliverstatus at 12:00:03.

Attaching two files:

• sliverstatus-log.txt - full output from sliverstatus commands
• summary-output.txt - only timestamp for sliverstatus and the circuit geni_urn returned

[lnevers@bbn.com]

Also attaching the request RSpec used to create the 4 stitched links

[lnevers@bbn.com]

Have not seen this type of problem in a while, although running sliverstatus for extended periods of time as not happened. However, this is not likely to get fix.