[[PageOutline(1-2)]]

= OPS-005-A Scheduled Site Maintenance =

This procedures describes how to handle a GENI Scheduled Site Maintenance. A Scheduled Maintenance may be requested by the Rack team or by operations (i.e. service upgrade). This type of request may also be submitted by a site contact for local maintenance activities. 

Regardless of the source for the reported event, a ticket must be written to track the scheduled maintenance completion. Ticket must copy the issue reporter and the GENI Experimenters at  geni-users@googlegroups.com.   But before writing the ticket, GMOC should check the GMOC operations calendars (http://gmoc.grnoc.iu.edu/gmoc/index/support/gmoc-operations-calendars.html) and:
 - Look for upcoming GENI events (e.g. conferences and tutorials) and note the times.
 - Check for other upcoming GENI maintenance and note the times.

GMOC may have to contact requester to make them aware of other upcoming GENI Events or Maintenance that may interfere with their maintenance request.

= 1. Issue Reported =

GMOC gathers technical details about the Scheduled Maintenance including:
 - Requester Organization 
 - Requester Name 
 - Requester email
 - GENI aggregates impacted and services within that aggregate will be directly affected.
 - Estimated duration. 
 - Individuals/organizations needed to complete the maintenance.

GMOC classifies the problem priority based on its urgency `Normal`: Usually a routine installation/provisioning or reporter initiated maintenance.

== 1.1 GENI Event Type Prioritization ==

GMOC normally classifies a scheduled maintenance as `Normal` priority that is Usually a routine installation/provisioning or reporter initiated maintenance. There may be priority exceptions when a scheduled maintenance impacts high priority resources like AL2S or services like the SCS. In those cases the priority should selection should consider the event type and the impact on the GENI Environment. Following is the Event Type prioritization that can be referenced to increase priority for a maintenance:

||'''Priority'''||'''#'''||'''Event type'''                ||'''Dispatch'''||'''Procedure'''||
|| Critical     || 1     ||Emergency Stop and LLR          || GMOC         || GMOC Emergency Stop and LLR procedures exist ||
|| Critical     || 2     ||Security Event ''[*]''          || GMOC         || <<Insert OPS procedure link>> ||
|| Critical     || 3     ||GENI !Clearinghouse/Portal Event        || GMOC         || <<Insert OPS procedure link>> || 
|| Critical     || 4     ||Stitching Computation Service   || GMOC         || GMOC I2 SCS Procedure  exists        ||
|| Critical     || 5     ||AL2S Aggregate Event            || GMOC         || GMOC OESS Procedures exist           ||
|| Critical     || 6     ||GENI Stitching Event            || GMOC         || [wiki:LuisaSandbox/GENIOperationsTrial/GENINetworkStitching OPS-003-B: Network Stitching Experiment Debugging Procedure] ||
|| High         || 7     ||WiMAX Multicast VLANs Events    || GMOC         || <<Insert OPS procedure link>> ||
|| High         || 8     ||Regional (AM+Switches+links)    || Rack or Site Group || <<Insert OPS procedures links>> ||
|| High         || 9     ||Site Events reported by site contact|| GMOC         ||   <<Insert OPS procedures links>> ||
|| High         || 10     ||Site Events(AM+Switches+links reported by experimenters or tools) ''[**]''|| Rack or Site Group || <<Insert OPS procedures links>> ||
|| High         || 11     ||Experimenter Tools Events(Portal, jacks, omni..)|| Tools Contacts ||  ||
|| High         || 12    ||Monitoring Infrastructure Events|| UKY Monitoring team ||<<Insert GENI Monitoring procedures  ||

 ''[*] Security Events start as Critical and may be re-prioritized upon investigation. [[BR]]''
 ''[**] Some `Site Events` may affect multiple sites (ExoSM) or non-GENI functions (!CloudLab, Emulab, Apt). These events require no special GMOC action and should be assigned to the team that owns the resources.''

== 1.2 Create Ticket ==

The GMOC ticketing system is used to capture maintenance information.  The ticket creation operation results in an email notification to the original requester and to the GENI experimenter list geni-users@googlegroups.com. GMOC will follow up to verify completion of the maintenance task.  Subsequent updates and interactions between GMOC and reporter will also generate notifications to the issue reporter. GMOC may postpone a ticket if the scheduled maintenance does not take place or if it cannot be completed in the time allocated.

= 2. Investigate and Identify Response =

GENI Operation only tracks GENI scheduled maintenance and verifies that maintenace completion has taken place.

== 2.1 Investigate the Problem ==

None

== 2.2 Identify Potential Response == 

None

= 3. GMOC Response =

Scheduled Maintenance ticket is usually dispatched to the organization responsible for the resource and usually requesting the maintenance. The GMOC verifies that the maintenance is completed and closes the ticket when the scheduled maintenance is done.  If a scheduled maintenance cannot be completed in the scheduled time, the GMOC updates the ticket to capture the new scheduled time.

== 3.1 Implement Response ==

The GMOC executes the steps outlined and verifies the completion of the scheduled event. The maintenance may take few iteration and GMOC may may have to post-pone ticket. 

== 3.2 Procedure Updates ==

When instructions in a procedure are found to miss symptoms, required actions, or potential impact, then action must be taken by the GMOC to provide feedback to enhance the procedure for future use.

= 4. Resolution =

For scheduled event, the GMOC coordinate with the person that originally scheduled the event to make sure that it was completed successfully.  There is also a potential for scheduled event tickets being postponed, and remaining open until the next scheduled time.

== 4.1 Document Resolution and Close Ticket ==

GMOC captures maintenance completion in the ticket and closes the ticket. If the maintenance does not fully address its goals or introduces a new problem, a new ticket may be created to track the remaining issue.