OESS aggregate does not expire slivers

[lnevers@bbn.com]

On October 10th I created 4 slices:

    urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-losa-nobkup:bc6e8aa5-2a2d-460f-a5ae-f69ff3c7d8b8
    urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-ny-nobkup:0c1130d8-629e-4a79-9af9-cc7f7a544e7f
    urn:publicid:IDN+ch.geni.net:ln-prj+slice+ny-2-losa-nobkup:7b973966-01e8-441b-b050-2472c462327d
    urn:publicid:IDN+ch.geni.net:ln-prj+slice+wash-2-ny-nobkup:b09e19ab-4b6b-4021-b817-39d8bd95c965

I left these 4 slices running so they would expire on their own, their slice expiration was "2013-10-16 00:00:00" Today (10/17@ 4pm) I check sliver status with omni, which reported that each slice had expired. I then logged in to the OESS web interface and found that all 4 slices were still active.

Note: All 4 slivers were renewed to the date "2013-10-16 00:00:00" before they were left to expire, this is not the original expiration time.

[lnevers@bbn.com]

This problem is till unresolved.

Had a slice and sliver at OESS aggregate named "atla-2-gpo". The slice "atla-2-gpo" expired in December. I just re-created the slice and the sliver still has the resources that were allocated before the slice expired.

1. Sliver status fails for expired sliver

   $ omni.py sliverstatus -a oess atla-2-gpo 
   10:51:31 INFO     omni: Downloaded latest `agg_nick_cache` from 'http://trac.gpolab.bbn.com/gcf/raw-attachment/wiki/Omni/agg_nick_cache' and copied to '/home/lnevers/.gcf/agg_nick_cache'.
   10:51:31 INFO     omni: Loading agg_nick_cache file '/home/lnevers/.gcf/agg_nick_cache'
   10:51:31 INFO     omni: Loading config file /home/lnevers/.gcf/omni_config
   10:51:31 INFO     omni: Using control framework portal
   10:51:31 INFO     omni: Substituting AM nickname oess with URL https://foam-oess-stage.grnoc.iu.edu:3626/foam/gapi/2, URN unspecified_AM_URN
   10:51:32 ERROR    omni.pgch: Call for Get Slice Cred for slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo failed.: Exception: GENI Clearinghouse Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo does not exist.
   10:51:32 ERROR    omni.pgch:     ..... Run with --debug for more information
   10:51:32 ERROR    omni: Cannot do SliverStatus for urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo: Could not get slice credential: Exception: GENI Clearinghouse Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo does not exist.
   Cannot do SliverStatus for urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo: Could not get slice credential: Exception: GENI Clearinghouse Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo does not exist.
   

2. Re-created the slice:

   $  omni.py createslice atla-2-gpo
   10:51:40 INFO     omni: Loading agg_nick_cache file '/home/lnevers/.gcf/agg_nick_cache'
   10:51:40 INFO     omni: Loading config file /home/lnevers/.gcf/omni_config
   10:51:40 INFO     omni: Using control framework portal
   10:51:40 INFO     omni: Created slice with Name atla-2-gpo, URN urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo, Expiration 2014-01-13 15:51:40
   10:51:40 INFO     omni:  ------------------------------------------------------------
   10:51:40 INFO     omni:  Completed createslice:
   Args: createslice atla-2-gpo
   
     Result Summary: Created slice with Name atla-2-gpo, URN urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo, Expiration 2014-01-13 15:51:40
    
   10:51:40 INFO     omni:  ============================================================
   

3. Re-ran the sliver status to verify that resources from previous run had been released upon expiration, but found otherwise:

   $ omni.py sliverstatus -a oess atla-2-gpo 
   10:51:53 INFO     omni: Loading agg_nick_cache file '/home/lnevers/.gcf/agg_nick_cache'
   10:51:53 INFO     omni: Loading config file /home/lnevers/.gcf/omni_config
   10:51:53 INFO     omni: Using control framework portal
   10:51:53 INFO     omni: Substituting AM nickname oess with URL https://foam-oess-stage.grnoc.iu.edu:3626/foam/gapi/2, URN unspecified_AM_URN
   10:51:54 INFO     omni: Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo expires on 2014-01-13 15:51:40 UTC
   10:51:54 INFO     omni: Status of Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo:
   10:51:54 INFO     omni: Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo at AM https://foam-oess-stage.grnoc.iu.edu:3626/foam/gapi/2 has overall SliverStatus: configuring
   10:51:54 INFO     omni: Sliver status for Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo at AM URL https://foam-oess-stage.grnoc.iu.edu:3626/foam/gapi/2
   10:51:54 INFO     omni: {
     "geni_status": "configuring", 
     "geni_urn": "urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo:79019e64-bc76-4dc9-a31e-6469aa64dca0", 
     "foam_pend_reason": [], 
     "foam_expires": "2013-12-23 14:13:48+00:00", 
     "geni_resources": [
       {
         "geni_urn": "urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo:79019e64-bc76-4dc9-a31e-6469aa64dca0", 
         "geni_error": "", 
         "geni_status": "configuring"
       }
     ], 
     "foam_status": "approved"
   }
   10:51:54 INFO     omni:  ------------------------------------------------------------
   10:51:54 INFO     omni:  Completed sliverstatus:
   Args: sliverstatus atla-2-gpo
   
     Result Summary: Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo expires on 2014-01-13 15:51:40 UTC
   Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo at AM https://foam-oess-stage.grnoc.iu.edu:3626/foam/gapi/2 has overall SliverStatus: configuring.
    Returned status of slivers on 1 of 1 possible aggregates. 
   10:51:54 INFO     omni:  ============================================================
   

[lnevers@bbn.com]

The OESS still does not expire slivers. The following sliver was set to expire at 4 AM UTC:

23:27:44 INFO    : {
  "geni_status": "configuring", 
  "geni_urn": "urn:publicid:IDN+ch.geni.net:Luisa_Nevers-Test-Proj_01+slice+atla-2-gpo:aba13c32-bd5a-4774-b26f-189c77696b2d", 
  "foam_pend_reason": [], 
  "foam_expires": "2014-06-27 04:00:00+00:00", 

But on 2014-06-27 the still was still present at 8 AM EDT:

08:45:31 INFO    : {
  "geni_status": "configuring", 
  "geni_urn": "urn:publicid:IDN+ch.geni.net:Luisa_Nevers-Test-Proj_01+slice+atla-2-gpo:aba13c32-bd5a-4774-b26f-189c77696b2d", 
  "foam_pend_reason": [], 
  "foam_expires": "2014-06-27 04:00:00+00:00", 

[lnevers@bbn.com]

To verify and close this ticket setting up slice "lnoess":

• Slice expires "2014-10-10 00:00:00"
• OESS sliver expires "2014-10-03 20:03:21 UTC"

Today is 2014-10-02, will check back tomorrow after "2014-10-03 20:03:21 UTC" or "16:03:21 EDT" to verify that sliver is properly expired by OESS aggregate.

[lnevers@bbn.com]

It is now 2014-10-03 21:09 UTC (17:09 EDT) and checking sliverstatus for sliver that should have expired shows that sliver was not expired:

17:09:59 INFO    : Sliver status for Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+lnoess at AM oess
17:09:59 INFO    : {
  "geni_status": "ready", 
  "geni_urn": "urn:publicid:IDN+ch.geni.net:ln-test+slice+lnoess:a07fb9cd-90a6-4a5c-86be-713672f8f7c4", 
  "foam_pend_reason": [], 
  "foam_expires": "2014-10-03 20:03:21+00:00", 
  "geni_resources": [
    {
      "geni_urn": "urn:publicid:IDN+ch.geni.net:ln-test+slice+lnoess:a07fb9cd-90a6-4a5c-86be-713672f8f7c4", 
      "geni_error": "", 
      "geni_status": "ready"
    }
  ], 
  "foam_status": "approved"
}
Slice lnoess at AM oess has overall SliverStatus: ready.

[luke@grnoc.iu.edu]

I made some chagnes to the cron config on the host which caused the expire cron job to actually execute properly. The following slices just expired automatically without my intervention:

urn:publicid:IDN+ch.geni.net:ln-test+slice+lnoess:a07fb9cd-90a6-4a5c-86be-713672f8f7c4 urn:publicid:IDN+ch.geni.net:ln-test+slice+IG-ST-6:b689e7ea-dda0-4736-9bed-647730f926b3

[lnevers@bbn.com]

Verified that "zombie" slivers are now expired. Closing ticket.