Changes between Version 10 and Version 11 of OperationalMonitoring/DatastorePolling


Ignore:
Timestamp:
02/17/14 17:51:19 (6 years ago)
Author:
rirwin@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • OperationalMonitoring/DatastorePolling

    v10 v11  
    5353}}}
    5454
    55 The response to this REST data query is a list of data items.  Here are [wiki:OperationalMonitoring/DataSchema#Measurementsusedfortheusecases a few example responses of a single data item].
     55The response to this REST data query is a list of data items.  Here are [wiki:OperationalMonitoring/DataSchema#Measurementsusedfortheusecases a few example responses of a single data item]. Here is an example of [wiki:OperationalMonitoring/DataSchema#BulkDataQueries a bulk query response]
    5656
    57 The REST noun and JSON format include the following:
    58 
    59  * Noun describing what type of data it is
    60 
    61  * Metadata about the response in the response like which local datastore and type of response.
    62 
    63  * Result groupings that group together all time-value pairs which are keyed by all combinations of other attributes (e.g. aggregate-id and resource-id). See the example below for clarification.
    64 
    65 The reference implementations for A - E from the [http://www.gpolab.bbn.com/monitoring/components/ component diagram] leverages the schema for both the REST API and table structure in the databases.  Reusing a common schema for the datastores and REST API has eased development thus far.
    66 
    67 The list of nouns and attributes associated with each noun are being formulated [wiki:OperationalMonitoring/DataSchema here].
    68 
    69 Example:
    70 The current prototype has components get the following from the config store:
    71 {{{
    72 schema["mem_used"] = [("id","varchar"), ("ts", "int8"), ("v", "float4")]
    73 }}}
    74 
    75 {{{
    76 http://127.0.0.1:5000/data/q?={"filters":{"eventType": ["mem_used","cpu_util"],"ts":{"gte":1391192225475202,"lt":1391192225480000},"obj":{"type":"node","id":["404-ig-pc1","404-ig-pc2"]}}}
    77 }}}
    78 {{{
    79 {
    80 "response_type": "data_poll",
    81 "data_type": "memory_util",
    82 "results":
    83   [{
    84    "aggregate_id": "404-ig",
    85    "resource_id": "compute_node_1",
    86    "measurements": {"ts": 1391192225381372, "v": 27.3}, {"ts": 1391192225589189, "v": 27.3}, {"ts": 1391192225792371, "v": 27.3}
    87    },
    88    {
    89    "aggregate_id": "404-ig",
    90    "resource_id": "compute_node_2",
    91    "measurements":{"ts": 1391192225381987, "v": 29.5}, {"ts": 1391192225589468, "v": 29.5}
    92   }]
    93 }
    94 }}}
    95 == Simple API in Noun Hierarchy and Set of Filters ==
    96 
    97 The goal of the aggregator is to gather data about a collection of resources, so having a lot of /noun1/noun2/noun3/noun4/etc. may not be necessary since /noun1 will gather all of items whenever possible.  For example "/memory_util/<compute_node_id>" is not in the initial implementation.
    98 
    99 The same thing applies to filters provided with '?' after the noun.  The "?since=<senconds since epoch>" filter is the only required filter now.  Other filters in consideration are data transformation filters like sampling_rate or average_since and simpler filters like greater_than or less_than.
    10057
    10158== Security ==
     
    10360Each Local store authenticates and encrypts data to Aggregators using certificates.
    10461
    105 == Configuration (OUT OF DATE, UPDATED SOON) ==
    106 
    107 The Aggregator queries the Config Local datastore for a list of all Local stores for connection information (URL/port) and the nouns each local store possesses.  This will also use a simple REST API called "/local_info" and potentially "/local_info/local_store_i".
    108 
    109 The Aggregator can query the Local store using the "/info/<noun>" to get information about the Local stores time-value pair collections.  To follow the example above, the REST call
    110 {{{
    111 http://127.0.0.1:5000/info/memory_util
    112 }}}
    113 would yield
    114 {{{
    115 {
    116 "response_type": "info_poll",
    117 "data_type": "memory_util",
    118 "measured_components":
    119   [ {"aggregate_id":"aggregate_1", "resource_id":"compute_node_1"} ,
    120     {"aggregate_id":"aggregate_1", "resource_id":"compute_node_2"} ]
    121 }
    122 }}}
    123 The Aggregator is configured by its operator to query a set of local stores for a set of their data.
    12462
    12563== Reference Prototypes ==