wiki:GENIMonitoring/GENIMonTips

Version 16 (modified by lnevers@bbn.com, 5 years ago) (diff)

--

GENI Monitoring Tips

This page highlights how to use GENI Monitoring features to investigate GENI Operations issues. All procedures assume that GENI Monitoring user is logged into the GENI Monitoring System.

Slice Topology and Bandwidth Usage

This scenario shows how to map a slice name to its topology (nodes and links) and to the bandwidth usage for each site in the slice. This sections shows how to determine the topology for a slice named "triangle" and then shows usage for one of the sites in this experiment.

  1. In the left-hand navigation bar, type "triangle" in Search box. This will show matching any item in monitoring containing the string "triangle" in the main panel.
  1. In the Slices listing select the slice ID "ch.geni.net_ln-test_slice_triangle". This will show all slice details in the main panel including:
    • Map tab - A resource allocation map, not the actual topology of the slice.
    • Active Members tab - The project owner of the the slice
    • Slivers tab - The list of compute resource slivers and network slivers associated with the slice.

  1. For our triangle slice, the Slivers tab will include a lists of 4 slivers (1 from AL2S and 3 from Rack Aggregates). From this list you can see that te slice has 3 nodes (1 at each site) and 3 links at AL2S, as shown below:

  1. You can select the each of the site slivers to determine which VM server or raw device the compute resource is allocated on. For example, selecting:
    • "geni.uchicago.edu_sliver_5003" shows that compute resource is on "geni.uchicago.edu_node_pcvm3-1"
    • "instageni.cenic.net_sliver_1266" shows that compute resource is on "instageni.cenic.net_node_pcvm2-2"
    • "instageni.nysernet.org_sliver_24968" shows that compute resource is on "instageni.nysernet.org_node_pcvm3-4"
  1. You can determine the slice topology and the site usage statistics by using the VLANs IDs allocated to the slice. VLANs can be determined from the AL2S sliver details:
    • Selected the AL2S sliver "urn:publicid:IDN+ch.geni.net:ln-test+slice+triangle:1bc9e85e-146c-4991-a4e9-96c5f3a5dee9" for the list of slivers for this slice.
    • Select each of the links that are part of the sliver to find VLANs allocated:
      • I2-LOSA-NEWY32AOA-VLAN-51145 includes sdn-sw.losa.net.internet2.edu:eth5/1:1550 and sdn-sw.newy32aoa.net.internet2.edu:eth7/2:1714
      • I2-LOSA-STAR-VLAN-51146 includes sdn-sw.losa.net.internet2.edu:eth5/1:1565 and sdn-sw.star.net.internet2.edu:eth7/1:3092
      • I2-NEWY32AOA-STAR-VLAN-51144 includes sdn-sw.newy32aoa.net.internet2.edu:eth7/2:1715 and sdn-sw.star.net.internet2.edu:eth7/1:3076

NOTE: Steps 3, 4 and 5 provide information to determine the topology for the slice. Following is a diagram of the details reported from step 3, 4 and 5:

  1. Assuming we are interested in the bandwidth usage for the NYSERNet connections (sdn-sw.newy32aoa.net.internet2.edu), you can see from the previous step that there are two VLANs associated with the site (1714 and 1715).
  1. Select Dashborad->VLANs, this will show the full list of VLAN in use in GENI. In the resulting VLAN listing type the VLAN ID 1715 in the panel search to narrow the results to you sliver:

  1. Select the ID instageni.nysernet.org_interface_pc3:eth1:1715 to see usage statistics:

  1. For the second NYSERNet connection, select Dashborad->VLANs, this will show the full list of VLAN in use in GENI. In the resulting VLAN listing type the VLAN ID 1714 in the panel search to narrow the results to you sliver:

  1. Select the ID instageni.nysernet.org_interface_pc3:eth1:1714 to see usage statistics:

Largest Traffic Source at an Aggregate

This scenario shows how to determine the slice that is generating the largest traffic amount at a site. This example assumes the site that is being investigated is the Chicago InstaGENI site.

  1. In Dashboard click on the Aggregates button, this will display all known aggregates.
  1. Narrow down the list by typing "chicago" in the upper-right Search field, which will show only Chicago aggregates. As you can see from the Nodes and Slivers counts, the InstaGENI rack has the most activity and most likely the source of the issue. So will we focus on uchicago-ig:

  1. We will use GENI reporting to find the VLAN that is the source of most traffic. Select GENI Reporting-> Reports.
  2. On the resulting page select the "Report Type" `VLAN Network Throughput [By Aggregate] from the pull-down and in the "Report Target" type start typing "chicago", the field will automatically list all aggregate names that match, select "uchicago-ig".
  1. In the Reports List results section, choose the Report Type named "VLAN Network Throughput [By Aggregate]", select the Report Target uchicago-ig, and the Report Range "Today". The resulting report "VLAN Network Throughput for Aggregate [uchicago-ig]" will show VLAN usage for the aggregate form highest to lowest for each VM Server Host interface in use. Following are the VLAN statistics report for one of the VM Servers:

  1. We are going to pick the highest usage VLAN ID on the VM Server PC2 using VLAN 3076):

  1. Go to Dashboard and click on the VLANs button, this shows list of all known VLANs. Narrow down the list by typing "3076" in the upper-right Search field. Results will now include AL2S and Chicago IG monitored VLANs.

  1. Select "sdn-sw.star.net.internet2.edu:eth7/1:3076" and you will get AL2S link details:

  1. In the "Endpoints" panel, select the link "I2-NEWY32AOA-STAR-VLAN-51144" to find the endpoints of the link:

  1. Select the "Slivers" panel to find the name of the sliver (and slice) that owns the VLAN:

Note: The Slice name is part of the AL2S Sliver Name.

Resources allocated by an Experimenter

This scenario shows how to map a GENI user ID to the resource allocation for that user's slices. In this example the user id lnevers is used.

  1. In Dashboard click on the Users button, this will display all active GENI Users. Note this is not the list of all users, it is the list of users that have resources in GENI, including slices without any resources.
  1. Narrow down the list by typing "lnevers" in the upper-right Search field, which will show the account `lnevers. From the slice results, we can see that "lnevers" has 12 Slivers and 19 Slices:

  1. The user "lnevers" has 12 Slivers which map to allocated resources. Note that slices are just containers and may or may not include resources. We will find resources allocated by clicking on the user ID "lnevers". The resulting page shows a Slivers tab and Slices tab for the user, select the "Slices"tab.
  1. In the "Slices" tab, the default list shows 10 slices, to view all 12 use the "records per page" pull-down". Once all slices are shown, sort by the "Slivers" count field to group slices that have resources. Any slice with a slivers count of zero can be ignored. In our example, there are 6 slices that have resources allocated in slivers:

  1. Click on the Slice ID ch.geni.net_ln-test_slice_triangle and then select "Slivers" panel. For our triangle slice, the Slivers tab will include a lists of 4 slivers (1 from AL2S and 3 from Rack Aggregates). From this list you can see that te slice has 3 nodes (1 at each site) and 3 links at AL2S, as shown below:

  1. Repeat step five for each of the other 5 slices with resources from step 4.

Slice Network Throughput

In this section we look again at the triangle slice and show how to see how much traffic this slice is generating.

  1. We will use GENI reporting to find slice statistics. Select GENI Reporting-> Reports.
  2. Select the Report Type named "Network Throughput [By Slice] and type "triangle" in the Report Target field and chose the matching slice ID.
  3. Select the default Report Range, which is Today and hist submit.
  4. When the report is completed and see the results:

Note: You may notice that these are AL2S interfaces towards the sites and that they match the topology we showed earlier.

AL2S VLAN Allocation

The GENI monitoring system reports VLANs used by GENI experiments as well as AL2S VLAN in use by non-GENI users. To get a list of VLANs used in GENI experiments do the following:

  1. In Dashboard click on the VLANS button, this will display all active GENI Users. Note the list also includes site VLANs.
  1. To get the list of GENI Experiment VLANs connecting to AL2S, type "al2s" in the upper-right Search field, which will show the GENI VLANs in use.
  1. To get the list of all Al2S VLANs, type either "internet2" or "sdn-sw" in the upper-right Search field, which will show all AL2S VLANs in use.
  1. Additionally it possible to view more details for each VLAN at the Internet2 Router proxy at "http://routerproxy.grnoc.iu.edu/internet2/" where you can choose an individual switch and issue the command "show openflow flows" for the selected switch. Note that the results will show 2 entries for each VLAN (input and output interfaces).

Attachments (18)