Changes between Version 1 and Version 2 of OperationalMonitoring/DatastorePolling


Ignore:
Timestamp:
02/17/14 14:28:22 (6 years ago)
Author:
rirwin@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • OperationalMonitoring/DatastorePolling

    v1 v2  
    11Datastore polling approach
     2
     3= Proposed API to poll local datastores for monitoring data =
     4
     5''This is a working page for the operational monitoring project.  It is a draft representing work in progress.''
     6
     7== API Basics ==
     8
     9This page describes the API to be used to poll Local datastores for monitoring data.  This API is component (c) of the [http://www.gpolab.bbn.com/monitoring/components/ component diagram].  The API will be developed with a polling mechanism first.  Overview:
     10
     11 * Polls are done via a REST API which returns JSON text of the queried data. See [http://rest.elkstein.org/ REST Overview] for a nice introduction to REST, and [http://www.w3schools.com/json/json_intro.asp JSON Overview] for nice introduction to JSON.
     12
     13 * The Aggregator polls a Local datastore at '/info/' of the local store url to get information about what the datastore has.  This is done through multiple polls. 
     14
     15 * The Aggregator polls for time-series data at '/data/' to get event-based or measurement data.  The query contains a set of event types, a set of object IDs, and timestamp filters.
     16
     17== Datastore/JSON Schema ==
     18
     19The REST noun and JSON format include the following:
     20
     21 * Noun describing what type of data it is
     22
     23 * Metadata about the response in the response like which local datastore and type of response.
     24
     25 * Result groupings that group together all time-value pairs which are keyed by all combinations of other attributes (e.g. aggregate-id and resource-id). See the example below for clarification.
     26
     27The reference implementations for A - E from the [http://www.gpolab.bbn.com/monitoring/components/ component diagram] leverages the schema for both the REST API and table structure in the databases.  Reusing a common schema for the datastores and REST API has eased development thus far.
     28
     29The list of nouns and attributes associated with each noun are being formulated [wiki:OperationalMonitoring/DataSchema here].
     30
     31Example:
     32The current prototype has components get the following from the config store:
     33{{{
     34schema["mem_used"] = [("id","varchar"), ("ts", "int8"), ("v", "float4")]
     35}}}
     36
     37{{{
     38http://127.0.0.1:5000/data/q?={"filters":{"eventType": ["mem_used","cpu_util"],"ts":{"gte":1391192225475202,"lt":1391192225480000},"obj":{"type":"node","id":["404-ig-pc1","404-ig-pc2"]}}}
     39}}}
     40{{{
     41{
     42"response_type": "data_poll",
     43"data_type": "memory_util",
     44"results":
     45  [{
     46   "aggregate_id": "404-ig",
     47   "resource_id": "compute_node_1",
     48   "measurements": {"ts": 1391192225381372, "v": 27.3}, {"ts": 1391192225589189, "v": 27.3}, {"ts": 1391192225792371, "v": 27.3}
     49   },
     50   {
     51   "aggregate_id": "404-ig",
     52   "resource_id": "compute_node_2",
     53   "measurements":{"ts": 1391192225381987, "v": 29.5}, {"ts": 1391192225589468, "v": 29.5}
     54  }]
     55}
     56}}}
     57== Simple API in Noun Hierarchy and Set of Filters ==
     58
     59The goal of the aggregator is to gather data about a collection of resources, so having a lot of /noun1/noun2/noun3/noun4/etc. may not be necessary since /noun1 will gather all of items whenever possible.  For example "/memory_util/<compute_node_id>" is not in the initial implementation.
     60
     61The same thing applies to filters provided with '?' after the noun.  The "?since=<senconds since epoch>" filter is the only required filter now.  Other filters in consideration are data transformation filters like sampling_rate or average_since and simpler filters like greater_than or less_than.
     62
     63== Security ==
     64
     65Each Local store authenticates and encrypts data to Aggregators using certificates.
     66
     67== Configuration (OUT OF DATE, UPDATED SOON) ==
     68
     69The Aggregator queries the Config Local datastore for a list of all Local stores for connection information (URL/port) and the nouns each local store possesses.  This will also use a simple REST API called "/local_info" and potentially "/local_info/local_store_i".
     70
     71The Aggregator can query the Local store using the "/info/<noun>" to get information about the Local stores time-value pair collections.  To follow the example above, the REST call
     72{{{
     73http://127.0.0.1:5000/info/memory_util
     74}}}
     75would yield
     76{{{
     77{
     78"response_type": "info_poll",
     79"data_type": "memory_util",
     80"measured_components":
     81  [ {"aggregate_id":"aggregate_1", "resource_id":"compute_node_1"} ,
     82    {"aggregate_id":"aggregate_1", "resource_id":"compute_node_2"} ]
     83}
     84}}}
     85The Aggregator is configured by its operator to query a set of local stores for a set of their data.
     86
     87== Reference Prototypes ==
     88
     89The reference prototype includes components A - E to test end-to-end functionality of the entire system, which exceeds the scope of the topic of this page.  The requirements include topics for the REST calls and JSON responses as well as security.
     90
     91The reference prototype will be made available soon (said 1/30/2014).