Changes between Initial Version and Version 1 of GEMINI/Tutorial/GEC21/GENI_Desktop_and_GEMINI_data/RetrieveData


Ignore:
Timestamp:
10/09/14 15:32:50 (9 years ago)
Author:
mkeele@indiana.edu
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • GEMINI/Tutorial/GEC21/GENI_Desktop_and_GEMINI_data/RetrieveData

    v1 v1  
     1Navigation: [wiki:GEMINI/Tutorial/GEC20/GENI_Desktop_and_GEMINI_data Up]
     2
     3[[PageOutline]]
     4{{{
     5#!html
     6<h1> Retrieving Data from the Periscope Supported Backend </h1>
     7}}}
     8
     9== Overview ==
     10
     11In the previous exercise, you may have already configured measurement probes via the GENI Desktop GUI. Here we are going to show you what happens in UNIS, which is the topology service that actually supports and directs the measurements.
     12
     13[[BR]]
     14
     15== How the Components are Connected ==
     16
     17[[Image(wiki:GEMINI/Tutorial/Images:IandM_diagram.png)]]
     18[[BR]]
     19The above diagram shows the components of the GEMINI framework. When a user reserves a slice, along with the slivers on that slice, hosts are created and connections between these hosts are built as well. Next step is to Initialize and Instrumentize these raw resources, which will install the tools that the GEMINI framework uses. BLiPP is the probe agent that is installed at each measurement point. The major task that BLiPP carries out is running various types of probes including ping, iperf, traceroute, etc. A user can either let these agents run a default probe or manually start the probes at selected times.
     20Here's the link [https://github.com/periscope-ps/blipp] to the BLiPP code repository and here's the wiki page [https://github.com/GENI-GEMINI/GEMINI/wiki/BLiPP-Documentation] where you can investigate more about using BLiPP for your custom probes.
     21
     22[[BR]]
     23
     24== BLiPP Generates Metadata and Data from Measurements ==
     25
     26BLiPP runs user measurements via probes, parses the command line output, and once completed BLiPP uploads the result data to the Measurement Store (MS). The MS is running on the Global Node which will act as a collection point for the data generated on the user slice. Due to the fact that there could be many different kinds of measurement data collected at different periods in the experiment, we separate the data into Metadata and Data. The Metadata will describe which particular Measurement Event Type a real data set belongs to and this is stored in UNIS for bookkeeping and for easy retrieval of the desired information. The actual data, will be stored on the Global Node via the Measurement Store on a per user slice basis, this allows us to tie data to a slice instance. The three UNIS objects you need to be familiar with for getting a full picture of your experiment: the measurement, the metadata of the measurement and the data of the metadata.
     27
     28[[BR]]
     29* The Measurement object: A measurement object depicts what probe the user want to use and how that probe will run. For example, probe command line arguments for user preference (e.g. how many times to repeat? and at what interval?).[[BR]]
     30* The Metadata object: Is a handle stored at UNIS to retrieve the actual data on a users Global Node, per slice. Also, this is the key information a user should specify should they need access to the raw data.[[BR]]
     31* The Data object: Is a series of key value pairs in JSON format.
     32
     33[[BR]]
     34
     35== How to Access Data from UNIS and the MS ==
     36
     37Before we provide a step-by-step guide to retrieving data, lets go more in depth with these components. UNIS is a topology service that is used to describe a network but underlying this concept is a RESTful API. As you might expect UNIS supports the standard HTTP Verbs: GET, POST, PUT, and DELETE.
     38
     391. if we wanted to query all the nodes on our slice we could: GET unis_host:port/nodes [[BR]]
     402. if we wanted to query one particular node on our slice we could: GET unis_host:port/nodes/:id
     41
     42These HTTP requests to UNIS can be executed through the browser, a script, or via the command line. For a more detailed explaination of the UNIS RESTful API see [https://github.com/GENI-GEMINI/GEMINI/wiki/UNIS-REST-API]. A user can supply the same HTTP verbs to interact with the MS but remember that the MS is located on the Global Node for a given slice. For a more detailed explaination of the MS RESTful API see [https://github.com/GENI-GEMINI/GEMINI/wiki/MS-REST-API]. If you would like more information about how UNIS represents the individual components of a network see this detailed resource [http://monitor.incntre.iu.edu/docs/].
     43
     44If you completed the previous exercise, [http://groups.geni.net/geni/wiki/GEMINI/Tutorial/GEC20/GENI_Desktop_and_GEMINI_blipp/ExecuteExperiment], then you should have an Instrumentized slice with a couple of Virtual Machines and a Global Node.
     45
     46[[Image(slice.png)]]
     47
     48Above is a sample server topology that will be used in this example. The GN0 node represents the Global Node and is where the MS runs and where data is collectioned. VM and VM-0 are the Measurement Points where BLiPP executes the probes we supply.
     49
     50=== Login to the Global Node ===
     51
     52{{{
     53$ ssh username@pcvm2-4.instageni.illinois.edu
     54 > Global Node
     55$ ssh username@pc2.instageni.illinois.edu -p 33339
     56 > Measurement Point
     57$ ssh username@pc2.instageni.illinois.edu -p 33340
     58 > Measurement Point
     59}}}
     60
     61  * If one of the machines asks for a password your ssh keys may not be in place. Visit [https://portal.geni.net/secure/profile.php] and click SSH Keys to download your keys.
     62 
     63Your command prompt should be similar to "[username@GN0]" for confirmation. There are a few important files to note on this machine.
     64
     65 
     661. The file {{{ /usr/local/etc/node.info }}} contains information about your slice and what services it's using. This file can be found on any Global Node as it is created during the Instrumentation process. [[BR]]
     67
     682. The file {{{ /usr/local/etc/certs/unis-proxy.pem }}} is the key, cert used to query the UNIS API, which is also created during the Instrumentation process.
     69
     70When making a request to the UNIS API you will need to supply your key, cert and also the UUID of your slice so that you only see your subsection of UNIS. The slice UUID can be found in the "node.info" file with the following format, "auth_uuid=fc028e53-cfbf-463d-9bf3-d26c35333b10".
     71
     721. query all the nodes on your slice: GET /nodes?properties.geni.slice_uuid=fc028e53-cfbf-463d-9bf3-d26c35333b10
     73
     74If you're going to be using a command line tool, such as curl, you will need to follow this structure to retrieve information from UNIS. Here, we provide a python script [https://github.com/periscope-ps/unis/blob/master/example/unis_client.py] to help facilitate this process.
     75
     76If you would like to try a periscope tool that handles your key, cert and slice uuid for you, then you will need to install a dependency on the Global Node. For this example my Global Node is Fedora 15 but if you're using another image you can see more installation instructions here [https://github.com/periscope-ps/periscope/tree/master/peri-js].
     77
     78=== Install Dependencies ===
     79{{{
     80$ cd /usr/src
     81$ sudo wget http://nodejs.org/dist/v0.10.28/node-v0.10.28.tar.gz
     82$ sudo tar -xvzf node-v0.10.28.tar.gz
     83$ cd node-v0.10.28
     84$ sudo ./configure
     85$ sudo make
     86$ sudo make install
     87$ sudo yum install git
     88}}}
     89
     90=== Test Dependencies ===
     91{{{
     92$ node -v
     93 > v0.10.28
     94$ npm -v
     95 > 1.4.9
     96$ git --version
     97 > git version 1.7.6.5
     98}}}
     99
     100=== Clone Periscope Tool ===
     101{{{
     102$ cd ~
     103$ git clone https://github.com/periscope-ps/periscope.git
     104$ cd periscope/peri-js/
     105}}}
     106
     107=== Install Peri-JS Dependencies ===
     108{{{
     109$ sudo npm install
     110$ sudo npm install -g bower
     111$ bower install
     112}}}
     113
     114=== Start Peri-JS ===
     115{{{
     116$ sudo npm start
     117}}}
     118
     119Upon starting this tool you will see the UNIS instance you are connecting to, the MS store instance, and various information about the Global Node. The Peri-JS tool provides a RESTful API that represents your subsection of UNIS. You can query this tool using the browser or command line.
     120
     1211. http://pcvm2-4.instageni.illinois.edu:42424/api [[BR]]
     1222. $ curl pcvm2-4.instageni.illinois.edu:42424/api
     123
     124Either way, when you request this URL you will see a list of routes you can query. These routes accept the same HTTP verbs as the UNIS and MS API's but Peri-JS only displays the data for your slice.
     125
     126=== Routes ===
     127{{{
     128GET /api/slice
     129GET /api/nodes
     130GET /api/nodes/:id
     131GET /api/services
     132GET /api/services/:id
     133GET /api/measurements
     134GET /api/measurements/:id
     135POST /api/measurements/:id
     136PUT /api/measurements/:id
     137DELETE /api/measurements/:id
     138GET /api/data/:id
     139GET /api/links
     140GET /api/links/:id
     141GET /api/ports
     142GET /api/ports/:id
     143GET /api/domains
     144GET /api/domains/:id
     145}}}
     146
     1471. http://pcvm2-4.instageni.illinois.edu:42424/api/measurements
     148
     149[[Image(cpu_measurement.png)]]
     150
     151Once some measurements have been posted and executed by BLiPP, metadata will be generated. Every measurement object has a list of event types associated with it and you will use this event type of find the data you desire. For example, if you were to schedule a throughput test you could expect an event type for bandwidth. BLiPP executes a few measurement probes directly after Instrumentation, such as cpu, mem, and net.
     152
     153Now, lets step through the process of gathering cpu usage data.
     154
     1552. http://pcvm2-4.instageni.illinois.edu:42424/api/metadata
     156  * list of all metadata
     157
     158Search for "cpu" and you will see every event types such as: [[BR]]
     159{{{
     160"ps:tools:blipp:linux:cpu:utilization:iowait"
     161"ps:tools:blipp:linux:cpu:utilization:hwirq"
     162"ps:tools:blipp:linux:cpu:load:fivemin"
     163}}}
     164
     165For the cpu used event type copy the value for the id key and add that value to your route
     166
     1673. http://pcvm2-4.instageni.illinois.edu:42424/api/metadata/53a486a7377f972d2e1b8e73
     168  * single metadata object for load min load
     169
     170[[Image(cpu_metadata.png)]]
     171
     172Replace metadata with data in the URL to get the data associated with the metadata event type
     173
     1744. http://pcvm2-4.instageni.illinois.edu:42424/api/data/53a486a7377f972d2e1b8e73
     175  * actual cpu load data
     176
     177[[Image(cpu_data.png)]]
     178
     179Now you will see all your data in JSON format. To reiterate, each metadata object represents an event type for a particular measurement. Navigating to a particular event type/metadata object by the metadata ID will get the single metadata object. Using that metadata ID, you can query the data route to get the data associated with that event type.
     180
     181Once you have your data you might like to visualize it to help make discoveries from the data. At this repository, [https://github.com/GENI-GEMINI/UW/blob/master/msplot_python/ms_plot.py] we provide a script to help you with data visualization.
     182
     1835. http://pcvm2-4.instageni.illinois.edu:42424/metadata/53a486a7377f972d2e1b8e73
     184  * simple graph of data
     185
     186[[Image(graph.png)]]
     187
     188[[BR]]