Changes between Initial Version and Version 1 of GENIOperationsTrial/DataPlaneDebugging


Ignore:
Timestamp:
06/18/15 08:08:41 (9 years ago)
Author:
lnevers@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • GENIOperationsTrial/DataPlaneDebugging

    v1 v1  
     1[[PageOutline(1-2)]]
     2
     3= OPS-003-A: GENI Dataplane Debugging =
     4
     5This procedures introduces a methodology for detecting and debugging dataplane issues in GENI.  A GENI dataplane failure may be reported by the [http://genimon.uky.edu/ GENI Monitoring System],  by [http://tamassos.gpolab.bbn.com/nagios3/ GPO Nagios],  or by a GENI experimenter.
     6
     7Regardless of the source for the reported event, a ticket must be written to handle the investigation and resolution of the problem. Ticket must copy the issue reporter and the GENI Experimenters at geni-users@googlegroups.com.
     8
     9
     10= 1. Issue Reported =
     11
     12GMOC gathers technical details for failures including:
     13 - Requester Organization
     14 - Requester Name
     15 - Requester email
     16 - Requester GENI site-name
     17 - Slice Name, any site sliver details available
     18 - Problem symptoms and impact 
     19
     20GMOC should classifies a dataplane failure as `High` priority. This type of issue may be deemed `Critical`  if the person reporting the issue identifies it as `Critical`. For example if the issue impacts a demo, a training session, or a conference.
     21
     22== 1.1 GENI Event Type Prioritization ==
     23
     24GMOC should classifies a dataplane failure as `High` priority. This type of issue may be deemed `Critical`  if the person reporting the issue identifies it as `Critical`. For example if the issue impacts a demo, a training session, or a conference.
     25
     26== 1.2 Create Ticket ==
     27
     28The GMOC ticketing system is used to capture issue information. GMOC may follow up to request additional information as the problem is investigated. The ticket creation operation results in an email notification to the reporter.  Subsequent updates and interactions between GMOC and reporter will also generate notifications to the issue reporter.
     29
     30= 2. Investigate and Identify Response =
     31
     32== 2.1 Investigate the Problem ==
     33
     34=== 2.1.1 Nagios ===
     35 * Nagios is currently used as an alerting system for monitoring data collected by the [http://genimon.uky.edu/ GENI Monitoring System]. To access the necessary alerts, click on "Services" from the [http://tamassos.gpolab.bbn.com/nagios3/ nagios interface]. In particular, each service name corresponds to an active stream of pings between two GENI endpoints.
     36 * Grab the service name and use instructions from the GENI Monitoring System below to obtain detailed statistics.
     37
     38=== 2.1.2 GENI Monitoring System ===
     39 * From the GENI Monitoring System dashboard, select "Checks" (which, for our data plane detection and debugging purposes, is equivalent to Nagios Services).
     40 * From the list of "Checks" shown, select/search for the check of interest which you obtained from Nagios.
     41 * Scroll to the "Information" section to obtain detailed statistics for your particular check. This information will serve as a guide in the "Debugging Methodology" section below.
     42
     43== 2.2 Identify Potential Response ==
     44
     45This section outlines debugging methodology to be used to determine the source and potential response for a dataplane problem:
     46
     47=== 2.2.1 Start ===
     48
     49 * Verify that both endpoints in the test are up and working as expected.
     50 * Replicate the test setup as much as possible in another slice.  This can help identify if the problem is slice-specific, or if there is a general substrate problem.  You can skip this step if you know for sure that the problem is not slice specific.
     51 * If you believe the problem is substrate specific:
     52  * You will need to identify where the problem is, and which aggregates are affected.  Once you know this information, you will want contact any aggregate or network operators (this might include a rack team and a site operational contact) as well as the GMOC.  You will want the operators to help fix the problem, and you will want the GMOC to track the problem in a ticket and announce it to the operator and user community.  Continue reading for information on how to track down where the problem is.
     53  * If the failing connectivity is an !OpenFlow connection, go [#OpenFlow here].
     54  * If the failing connectivity is a non-!OpenFlow (stitched or static) connection, go [#Non-OpenFlow here]
     55
     56=== 2.2.2 !OpenFlow issue ===
     57
     58==== Control plane ====
     59 * Use netstat at the controller to see if the number of connecting switches is as expected.
     60 * If the connection is in place, take a packet capture on the controller's !OpenFlow control plane interface and see if !OpenFlow traffic from the switch is getting through and it matches your expectations.
     61 * If the connection is not in place:
     62  * Verify that your reservation still exists at each aggregate manager in the path.
     63  * Ensure that all switches in the path are up.  Ops-monitoring and the aggregate manager advertisement may be able to help.  If you cannot find the info there, then you may need to ask site admins to verify this.
     64  * Ensure that no switches are seeing consistently high CPU usage.  If they are:
     65   * Sending a lot of traffic through a software table on the switch.
     66   * Sending a lot of traffic through the control plane.
     67  * Ensure all proxies in the path (e.g. slicers, etc) are up and connected to both the switch and your controller.  Ops-monitoring and the aggregate manager advertisement may be able to help.  If you cannot find the info there, then you may need to ask site admins to verify this.
     68
     69==== Data plane ====
     70 * At this point, you can use a combination of the techniques below in conjunction with those mentioned in the non-!OpenFlow debugging section.
     71 * The goal at this point is to isolate where the problem is occurring in the data plane, so you will want to use a combination searching from the middle (like a binary search) and educated guesses based on the information you have.
     72  * You can query for flowstats as you generate data plane probes.  Trace through the flow stats and see if you can figure out where a switch is not being hit.
     73  * If your controller is reactive, follow packet-ins and packet-outs as well using packet captures.
     74  * Remember that in some rare cases, you will encounter error cases due to software or hardware issues in an !OpenFlow switch.  In those cases, you will likely need to use the above techniques.
     75
     76=== 2.2.3 Non-!OpenFlow issue ===
     77
     78 * Start by trying to spot the traffic from the middle and working your way towards the problem.  Get as much information as you can on your own using tools, or devices which you have access to.  Tools include:
     79  * [http://routerproxy.grnoc.iu.edu/al2s/ AL2S Router Proxy] (more on using this too [#AL2SRouterProxy below])
     80  * [http://genimon.uky.edu/login GENI operational monitoring]
     81 * Isolate the breakage(s) to segments that you don't have visibility into.  From there, you will need to work with operators of aggregates and intermediary networks.  You will want to have them:
     82  * Try to verify that all switches in the path of your slice are up.
     83  * Trace MAC addresses through the path by checking the MAC learning tables in the switches.  Note that in some VLANs, MAC address learning has been disabled throughout the path, and they will need to temporarily enable MAC address learning on those VLANs.
     84
     85
     86=== 2.2.4 Debugging Tools ===
     87
     88==== AL2S Router Proxy ====
     89
     90AL2S router proxy can show you information that a given switch knows.  The types of queries that you use will be vendor specific.  Below are some useful queries at the time of this writing, organized by vendor.
     91
     92==== Brocade ====
     93Show flows on a device which match an ingress port of <portnum>
     94{{{
     95show openflow flows ethernet <portnum>
     96}}}
     97
     98After running this command, you will want to search the output for the flows corresponding to the VLAN ID that is associated with the slice at that switch.  You should be able to look up flow statistics using this method.
     99
     100==== Juniper ====
     101Find flow ID associated with a slice:
     102{{{
     103show openflow flows detail
     104}}}
     105
     106After running this command, you will want to search the output for the flows corresponding to the VLAN ID that is associated with the slice at that switch.  When you find the flows you care about, save the corresponding Flow ID.
     107
     108Once you have the flow IDs you care about, you will want to run the following to look up statistics for those flows:
     109{{{
     110show openflow statistics flows <flow ID>
     111}}}
     112
     113= 3. GMOC Response =
     114
     115The GMOC implements the actions outlined to identify the source of the issue and updates the ticket to capture actions taken.  In some scenarios the GMOC may dispatch a problem to other organizations, following is a table of organizations that will provide support listed by area of responsibility:
     116 
     117|| ''' Team '''        || ''' Area of !Responsibility/Tools''' ||
     118|| GPO Dev Team        || GENI Tools (gcf, omni, stitcher), GENI Portal, GENI Clearinghouse ||
     119|| RENCI Dev Team      || ExoGENI Racks, ExoGENI Stitching ||
     120|| GENI Operations     || InstaGENI Racks ||
     121|| UKY Operations Team || GENI Monitoring System, Stitching Computation System ||
     122|| Utah Dev Team       || Jack Tool, !CloudLab, Emulab, Apt||
     123
     124== 3.1 Implement Response ==
     125
     126The GMOC executes the steps outlined. The response implementation may take few iteration, as some attempt may not yield the expected results. GMOC may may have to go back and try further actions in case where new symptoms may occur, or where the procedure is found to be lacking.  For both cases, an update to the procedures may be required. Actions should be taken to get the procedures updated.
     127
     128== 3.2 Procedure Updates ==
     129
     130When instructions in a procedure are found to miss symptoms, required actions, or potential impact, then action must be taken by the GMOC to provide feedback to enhance the procedure for future use.
     131
     132= 4. Resolution =
     133
     134GMOC verifies the the problem is no longer happening by coordinating with the problem reporter or by checking the tool/log that originally signaled the problem.
     135
     136== 4.1 Document Resolution and Close Ticket ==
     137
     138GMOC captures how the problem is resolved in the ticket and closes the ticket. If the problem solution does not fully resolve the problem, a new ticket may be created to generate a new ticket to track the remaining issue.
     139
     140Whether the problem is fully resolved (ticket closed) or partially resolved (new ticket open), both should result in notification back to the problem reporter.
     141