wiki:GEMINI/Tutorial/GEC20/GENI_Desktop_and_GEMINI_data/RetrieveData

Version 11 (modified by mkeele@indiana.edu, 5 years ago) (diff)

intro to rest api

Navigation: Up

Retrieving Data from the Periscope Supported Backend

Overview

In the previous exercise, you may have already configured measurement probes via the GENI Desktop GUI. Here we are going to show you what happens in UNIS, which is the topology service that actually supports and directs the measurements.


How the Components are Connected


The above diagram shows the components of the GEMINI framework. When a user reserves a slice, along with the slivers on that slice, hosts are created and connections between these hosts are built as well. Next step is to Initialize and Instrumentize these raw resources, which will install the tools that the GEMINI framework uses. BLiPP is the probe agent that is installed at each measurement point. The major task that BLiPP carries out is running various types of probes including ping, iperf, traceroute, etc. A user can either let these agents run a default probe or manually start the probes at selected times. Here's the link https://github.com/periscope-ps/blipp to the BLiPP code repository and here's the wiki page https://github.com/GENI-GEMINI/GEMINI/wiki/BLiPP-Documentation where you can investigate more about using BLiPP for your custom probes.


BLiPP Generates Metadata and Data from Measurements

BLiPP runs user measurements via probes, parses the command line output, and once completed BLiPP uploads the result data to the Measurement Store (MS). The MS is running on the Global Node which will act as a collection point for the data generated on the user slice. Due to the fact that there could be many different kinds of measurement data collected at different periods in the experiment, we separate the data into Metadata and Data. The Metadata will describe which particular Measurement Event Type a real data set belongs to and this is stored in UNIS for bookkeeping and for easy retrieval of the desired information. The actual data, will be stored on the Global Node via the Measurement Store on a per user slice basis, this allows us to tie data to a slice instance. The three UNIS objects you need to be familiar with for getting a full picture of your experiment: the measurement, the metadata of the measurement and the data of the metadata.


  • The Measurement object: A measurement object depicts what probe the user want to use and how that probe will run. For example, probe command line arguments for user preference (e.g. how many times to repeat? and at what interval?).
  • The Metadata object: Is a handle stored at UNIS to retrieve the actual data on a users Global Node, per slice. Also, this is the key information a user should specify should they need access to the raw data.
  • The Data object: Is a series of key value pairs in JSON format.


How to Access Data from UNIS and the MS

step-by-step process of scanning the topology, looking for metadata, finding the eventTypes you are interested in (using unis_client.py or a browser), and then ms_plot.py as a simple graphing tool tht can be customized for whatever graphing the user needs

Before we provide a step-by-step guide to retrieving data, lets go more in depth with these components. UNIS is a topology service that is used to describe a network but underlying this concept is a RESTful API. As you might expect UNIS supports the standard HTTP Verbs: GET, POST, PUT, and DELETE.

  • For example, if we wanted to query all the nodes on our slice we could: GET unis_host:port/nodes
  • For example, if we wanted to query one particular node on our slice we could: GET unis_host:port/nodes/:id

These HTTP requests to UNIS can be executed through the browser, a script, or via the command line. For a more detailed explaination of the UNIS RESTful API see https://github.com/GENI-GEMINI/GEMINI/wiki/UNIS-REST-API. A user can supply the same HTTP verbs to interact with the MS but remember that the MS is located on the Global Node for a given slice. For a more detailed explaination of the MS RESTful API see https://github.com/GENI-GEMINI/GEMINI/wiki/MS-REST-API. If you would like more information about how UNIS represents the individual components of a network see this detailed resource http://monitor.incntre.iu.edu/docs/.


Attachments (5)

Download all attachments as: .zip