Opened 6 years ago

Closed 5 years ago

#1142 closed (fixed)

OESS aggregate does not expire slivers

Reported by: lnevers@bbn.com Owned by: luke@grnoc.iu.edu
Priority: major Milestone:
Component: STITCHING Version: SPIRAL6
Keywords: oess Cc: matt@internet2.edu, eboyd@internet2.edu
Dependencies:

Description

On October 10th I created 4 slices:

    urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-losa-nobkup:bc6e8aa5-2a2d-460f-a5ae-f69ff3c7d8b8
    urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-ny-nobkup:0c1130d8-629e-4a79-9af9-cc7f7a544e7f
    urn:publicid:IDN+ch.geni.net:ln-prj+slice+ny-2-losa-nobkup:7b973966-01e8-441b-b050-2472c462327d
    urn:publicid:IDN+ch.geni.net:ln-prj+slice+wash-2-ny-nobkup:b09e19ab-4b6b-4021-b817-39d8bd95c965

I left these 4 slices running so they would expire on their own, their slice expiration was "2013-10-16 00:00:00" Today (10/17@ 4pm) I check sliver status with omni, which reported that each slice had expired. I then logged in to the OESS web interface and found that all 4 slices were still active.

Note: All 4 slivers were renewed to the date "2013-10-16 00:00:00" before they were left to expire, this is not the original expiration time.

Change History (6)

comment:1 Changed 6 years ago by lnevers@bbn.com

This problem is till unresolved.

Had a slice and sliver at OESS aggregate named "atla-2-gpo". The slice "atla-2-gpo" expired in December. I just re-created the slice and the sliver still has the resources that were allocated before the slice expired.

  1. Sliver status fails for expired sliver
    $ omni.py sliverstatus -a oess atla-2-gpo 
    10:51:31 INFO     omni: Downloaded latest `agg_nick_cache` from 'http://trac.gpolab.bbn.com/gcf/raw-attachment/wiki/Omni/agg_nick_cache' and copied to '/home/lnevers/.gcf/agg_nick_cache'.
    10:51:31 INFO     omni: Loading agg_nick_cache file '/home/lnevers/.gcf/agg_nick_cache'
    10:51:31 INFO     omni: Loading config file /home/lnevers/.gcf/omni_config
    10:51:31 INFO     omni: Using control framework portal
    10:51:31 INFO     omni: Substituting AM nickname oess with URL https://foam-oess-stage.grnoc.iu.edu:3626/foam/gapi/2, URN unspecified_AM_URN
    10:51:32 ERROR    omni.pgch: Call for Get Slice Cred for slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo failed.: Exception: GENI Clearinghouse Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo does not exist.
    10:51:32 ERROR    omni.pgch:     ..... Run with --debug for more information
    10:51:32 ERROR    omni: Cannot do SliverStatus for urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo: Could not get slice credential: Exception: GENI Clearinghouse Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo does not exist.
    Cannot do SliverStatus for urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo: Could not get slice credential: Exception: GENI Clearinghouse Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo does not exist.
    
  2. Re-created the slice:
    $  omni.py createslice atla-2-gpo
    10:51:40 INFO     omni: Loading agg_nick_cache file '/home/lnevers/.gcf/agg_nick_cache'
    10:51:40 INFO     omni: Loading config file /home/lnevers/.gcf/omni_config
    10:51:40 INFO     omni: Using control framework portal
    10:51:40 INFO     omni: Created slice with Name atla-2-gpo, URN urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo, Expiration 2014-01-13 15:51:40
    10:51:40 INFO     omni:  ------------------------------------------------------------
    10:51:40 INFO     omni:  Completed createslice:
    Args: createslice atla-2-gpo
    
      Result Summary: Created slice with Name atla-2-gpo, URN urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo, Expiration 2014-01-13 15:51:40
     
    10:51:40 INFO     omni:  ============================================================
    
  3. Re-ran the sliver status to verify that resources from previous run had been released upon expiration, but found otherwise:
    $ omni.py sliverstatus -a oess atla-2-gpo 
    10:51:53 INFO     omni: Loading agg_nick_cache file '/home/lnevers/.gcf/agg_nick_cache'
    10:51:53 INFO     omni: Loading config file /home/lnevers/.gcf/omni_config
    10:51:53 INFO     omni: Using control framework portal
    10:51:53 INFO     omni: Substituting AM nickname oess with URL https://foam-oess-stage.grnoc.iu.edu:3626/foam/gapi/2, URN unspecified_AM_URN
    10:51:54 INFO     omni: Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo expires on 2014-01-13 15:51:40 UTC
    10:51:54 INFO     omni: Status of Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo:
    10:51:54 INFO     omni: Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo at AM https://foam-oess-stage.grnoc.iu.edu:3626/foam/gapi/2 has overall SliverStatus: configuring
    10:51:54 INFO     omni: Sliver status for Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo at AM URL https://foam-oess-stage.grnoc.iu.edu:3626/foam/gapi/2
    10:51:54 INFO     omni: {
      "geni_status": "configuring", 
      "geni_urn": "urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo:79019e64-bc76-4dc9-a31e-6469aa64dca0", 
      "foam_pend_reason": [], 
      "foam_expires": "2013-12-23 14:13:48+00:00", 
      "geni_resources": [
        {
          "geni_urn": "urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo:79019e64-bc76-4dc9-a31e-6469aa64dca0", 
          "geni_error": "", 
          "geni_status": "configuring"
        }
      ], 
      "foam_status": "approved"
    }
    10:51:54 INFO     omni:  ------------------------------------------------------------
    10:51:54 INFO     omni:  Completed sliverstatus:
    Args: sliverstatus atla-2-gpo
    
      Result Summary: Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo expires on 2014-01-13 15:51:40 UTC
    Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+atla-2-gpo at AM https://foam-oess-stage.grnoc.iu.edu:3626/foam/gapi/2 has overall SliverStatus: configuring.
     Returned status of slivers on 1 of 1 possible aggregates. 
    10:51:54 INFO     omni:  ============================================================
    

comment:2 Changed 5 years ago by lnevers@bbn.com

The OESS still does not expire slivers. The following sliver was set to expire at 4 AM UTC:

23:27:44 INFO    : {
  "geni_status": "configuring", 
  "geni_urn": "urn:publicid:IDN+ch.geni.net:Luisa_Nevers-Test-Proj_01+slice+atla-2-gpo:aba13c32-bd5a-4774-b26f-189c77696b2d", 
  "foam_pend_reason": [], 
  "foam_expires": "2014-06-27 04:00:00+00:00", 

But on 2014-06-27 the still was still present at 8 AM EDT:

08:45:31 INFO    : {
  "geni_status": "configuring", 
  "geni_urn": "urn:publicid:IDN+ch.geni.net:Luisa_Nevers-Test-Proj_01+slice+atla-2-gpo:aba13c32-bd5a-4774-b26f-189c77696b2d", 
  "foam_pend_reason": [], 
  "foam_expires": "2014-06-27 04:00:00+00:00", 

comment:3 Changed 5 years ago by lnevers@bbn.com

To verify and close this ticket setting up slice "lnoess":

  • Slice expires "2014-10-10 00:00:00"
  • OESS sliver expires "2014-10-03 20:03:21 UTC"

Today is 2014-10-02, will check back tomorrow after "2014-10-03 20:03:21 UTC" or "16:03:21 EDT" to verify that sliver is properly expired by OESS aggregate.

comment:4 Changed 5 years ago by lnevers@bbn.com

It is now 2014-10-03 21:09 UTC (17:09 EDT) and checking sliverstatus for sliver that should have expired shows that sliver was not expired:

17:09:59 INFO    : Sliver status for Slice urn:publicid:IDN+ch.geni.net:ln-test+slice+lnoess at AM oess
17:09:59 INFO    : {
  "geni_status": "ready", 
  "geni_urn": "urn:publicid:IDN+ch.geni.net:ln-test+slice+lnoess:a07fb9cd-90a6-4a5c-86be-713672f8f7c4", 
  "foam_pend_reason": [], 
  "foam_expires": "2014-10-03 20:03:21+00:00", 
  "geni_resources": [
    {
      "geni_urn": "urn:publicid:IDN+ch.geni.net:ln-test+slice+lnoess:a07fb9cd-90a6-4a5c-86be-713672f8f7c4", 
      "geni_error": "", 
      "geni_status": "ready"
    }
  ], 
  "foam_status": "approved"
}
Slice lnoess at AM oess has overall SliverStatus: ready.

comment:5 Changed 5 years ago by luke@grnoc.iu.edu

I made some chagnes to the cron config on the host which caused the expire cron job to actually execute properly. The following slices just expired automatically without my intervention:

urn:publicid:IDN+ch.geni.net:ln-test+slice+lnoess:a07fb9cd-90a6-4a5c-86be-713672f8f7c4 urn:publicid:IDN+ch.geni.net:ln-test+slice+IG-ST-6:b689e7ea-dda0-4736-9bed-647730f926b3

comment:6 Changed 5 years ago by lnevers@bbn.com

Resolution: fixed
Status: newclosed

Verified that "zombie" slivers are now expired. Closing ticket.

Note: See TracTickets for help on using tickets.