Changes between Version 16 and Version 17 of URNConsolidation


Ignore:
Timestamp:
07/13/16 11:03:44 (3 years ago)
Author:
hdempsey@bbn.com
Comment:

clarify notificaitons

Legend:

Unmodified
Added
Removed
Modified
  • URNConsolidation

    v16 v17  
    99== 1. Generate Tickets and check for conflicts with upcoming GENI events ==
    1010
    11 Make sure tickets are opened at GMOC for the events listing all affected GENI resources.  Also make sure that GMOC generates corresponding requests to Internet2 Engineering (GRNOC).  Tickets should notify operators and experimenters.  Adam Williams will coordinate efforts for GMOC, but initial requests should go to the usual GMOC email for ticket requests. 
     11=== Create GMOC tickets ===
     12Open tickets with GMOC for the scheduled maintenance events listing all affected GENI resources as soon as we receive notice of the scheduled days (this comes in an email from Eric Boyd NLT 1 week before the outage.  Confirm that GMOC generates corresponding requests to Internet2 Engineering (GRNOC).  GMOC tickets should notify operators and experimenters lists.  Although the outage is only scheduled for 1 day on the Internet2 schedule, the GENI ticket may require a longer outage.  The ticket should include a warning that the outage may be extended at the end of the day if there are issues with any updates (include this in your initial email to GMOC).  Adam Williams will coordinate efforts for GMOC, but initial requests should go to the usual gmoc@grnoc.iu.edu address.
    1213
    1314Note that Internet2 schedules both an IP and an AL2S outage (usually on different days) for each PoP consolidation.  The IP event has no related GENI URN work needed, and will simply result in the GENI resources being unreachable (because the entire device is disconnected).  The GMOC should create tickets for both events, since they both have GENI impact, and the rack admins should see the tickets if they read their GENI operators email.
    1415
    15 Internet2 won't change their schedule, but you should notify any conflicting events about the maintenance and work with them to avoid any impact as much as possible.
    16 
    17 If the consolidation event goes longer than the scheduled tickets, be sure to email updates to the GMOC when you know that will happen, and every 2 hours thereafter.  If the event will continues to the next day, say so in your last ticket update, and tell them when you'll check in again the next day.  (You don't have to update in between).
    18 
    19 If there are any significant problems during the event, be sure to escalate to Heidi Dempsey (hdempsey@bbn.com) while you work on them (in addition to noting them in the ticket).
     16Note that GMOC should check the GENI calendar for any conflicting events that are scheduled to overlap, and send followup email if they find any.  Rather than wait for this to happen,check the existing GENI tickets or calendar yourself for conflicts and notify any affected event coordinator via email directly.
     17
     18=== Check the test SCS for affected sites and generate warning emails ===
     19
     20The GMOC may not have records of GENI connections for some nodes that are only supported on the test SCS (e.g. CloudLab).  If the scheduled maintenance will affect a test SCS node (rack, switch etc.), email the contact for that node directly and cc: gpo-infra, informing them of the scheduled outage, and asking them to be available to test connectivity after the update.  Add the test SCS node owner contact to any status update emails you send during the outage.  (Sometimes test SCS nodes are no longer in use, so the owner may indicate they can be retired instead of revised.  Include retiring resources as part of the outage work.)
     21
     22=== Changing the Schedule and Escalation ===
     23
     24Internet2 won't change their schedule, but we can work with affected sites and event contacts to try to priortize the work to minimize the outage impact for priority sites if needed. 
     25
     26If the consolidation event goes longer than the scheduled outage ticket lasts, be sure to email updates to the GMOC and to anyone who was contacted via email (from the test SCS or event lists) as soon as you know an extension is needed.  Update the same list if you need to extend more than once.  If the event will continue to the next day, indicate when work will start again on the next day in your update.  You should send updates to the GMOC ticket for any significant events or changes that happen during the maintenance (version updates, bad hardware etc.).  Do not send these type of updates only to an ops email list, because that info won't get out to resource owners.
     27
     28If there are any significant problems during the event, escalate to Heidi Dempsey (hdempsey@bbn.com) while you work on them (in addition to noting them in the ticket).
    2029
    2130== 2. Identify Affected Stitching Endpoints ==