OpenGENI monitoring exception and naming issues

[lnevers@bbn.com]

NOTE: writing as "GPO" ticket, but will move to "OPENGENI" components as soon as it is created.

This ticket tracks multiple issues:

1. The monitoring URL for Clemson OG is:

• ​https://clemson-clemson-control-1.clemson.edu/info/aggregate/gram-clemson

while for GPO OG we use:

• ​https://bbn-cam-ctrl-1.gpolab.bbn.com/info/aggregate/gpo-og

The Clemson URL be updated to use "clemson-og", so it can also match its site short name.

2. While querying both Clemson and GPO OG site there is an error due to a link only having one end-point.

   Visiting https://192.1.243.2/info/link/urn_publicid_IDN_bbn-cam-ctrl-1.gpolab.bbn.com_sliver_link7db04f88-0a40-4a7b-9984-0f6836995c69
   returns: Received exception Value [{u'urn': 
   u'urn:publicid:IDN+bbn-cam-ctrl-1.gpolab.bbn.com+sliver+interfacea2f6093e-bdc7-4ffa-a30c-493f85c8e7d2',
   u'href': u'https://192.1.243.2/info/interfacevlan/urn_publicid_IDN_bbn-cam-ctrl-1.gpolab.bbn.com_sliver_interfacea2f6093e-bdc7-4ffa-a30c-493f85c8e7d2'}]
   for field '<obj>.endpoints' must have length greater than or equal to 2
   while trying to validate: {u'selfRef':
   u'https://192.1.243.2/info/link/urn_publicid_IDN_bbn-cam-ctrl-1.gpolab.bbn.com_sliver_link7db04f88-0a40-4a7b-9984-0f6836995c69',
   u'urn': u'urn:publicid:IDN+bbn-cam-ctrl-1.gpolab.bbn.com+sliver+link7db04f88-0a40-4a7b-9984-0f6836995c69',
   u'ts': 1409764638564179L, u'$schema':
   u'http://www.gpolab.bbn.com/monitoring/schema/20140501/link#',
   u'endpoints': [{u'urn': 
   u'urn:publicid:IDN+bbn-cam-ctrl-1.gpolab.bbn.com+sliver+interfacea2f6093e-bdc7-4ffa-a30c-493f85c8e7d2',
   u'href': u'https://192.1.243.2/info/interfacevlan/urn_publicid_IDN_bbn-cam-ctrl-1.gpolab.bbn.com_sliver_interfacea2f6093e-bdc7-4ffa-a30c-493f85c8e7d2'}],
   u'id': u'urn_publicid_IDN_bbn-cam-ctrl-1.gpolab.bbn.com_sliver_link7db04f88-0a40-4a7b-9984-0f6836995c69'}
   

3. Once Marshall changes the Clemson OG URL, the monitoring team needs to update ops-monitoring/config/opsconfig.json file

[mbrinn@bbn.com]

#1 is fixed. It now responds to clemson-og and not gram-clemson.

#2 is odd. I can't reproduce this now: when I run the same URL it comes back happy. I'll try some different configurations.

• Marwshall

[lnevers@bbn.com]

Verified change to the Clemson OG URL:

$ ./rest_call.py -c ~/collector.pem https://clemson-clemson-control-1.clemson.edu/info/aggregate/clemson-og
/usr/local/lib/python2.7/dist-packages/requests/packages/urllib3/connectionpool.py:730: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html (This warning will only appear once by default.)
  InsecureRequestWarning)
------------------------------------------------------------------------------- 
Visiting https://clemson-clemson-control-1.clemson.edu/info/aggregate/clemson-og returns:
{
    "$schema": "http://www.gpolab.bbn.com/monitoring/schema/20140501/aggregate#", 
    "id": "clemson-og", 
    "measRef": "https://clemson-clemson-control-1.clemson.edu/data/", 
    "resources": [
        {
            "href": "https://clemson-clemson-control-1.clemson.edu/info/link/urn_publicid_IDN_clemson-clemson-control-1.clemson.edu_sliver_link8076f991-dc24-4414-9072-4a95ecc0cdb1", 
            "urn": "urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+sliver+link8076f991-dc24-4414-9072-4a95ecc0cdb1"
        }, 
        {
            "href": "https://clemson-clemson-control-1.clemson.edu/info/node/clemson-og.clemson-clemson-compute-2", 
            "urn": "urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+node+clemson-clemson-compute-2"
        }, 
        {
            "href": "https://clemson-clemson-control-1.clemson.edu/info/node/clemson-og.clemson-clemson-compute-1", 
            "urn": "urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+node+clemson-clemson-compute-1"
        }
    ], 
    "selfRef": "https://clemson-clemson-control-1.clemson.edu/info/aggregate/clemson-og", 
    "slivers": [
        {
            "href": "https://clemson-clemson-control-1.clemson.edu/info/sliver/urn_publicid_IDN_ch.geni.net_ln-test_slice_OG-CT-3_CLEM-OG", 
            "urn": "urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+sliver+vmcb2b5c54-7f47-4164-9d6e-8374d627b5c8"
        }, 
        {
            "href": "https://clemson-clemson-control-1.clemson.edu/info/sliver/urn_publicid_IDN_ch.geni.net_ln-test_slice_OG-CT-3_CLEM-OG:if0", 
            "urn": "urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+sliver+interface797cc881-e395-4b49-9c2d-531efd8b3a50"
        }
    ], 
    "ts": 1409773769387851, 
    "urn": "urn:publicid:IDN+clemson-clemson-control-1.clemson.edu+authority+am"
}
Response from https://clemson-clemson-control-1.clemson.edu/info/aggregate/clemson-og is valid
1 URLs visited , 1 valid

[lnevers@bbn.com]

Replying to mbrinn@…:

#2 is odd. I can't reproduce this now: when I run the same URL it comes back happy. I'll try some different configurations.

Marshall I can see the error when I use the rest_call.py "-f" option (--follow-urls) and the return is validated by validictory:

 ./rest_call.py -f -c ~/collector.pem https://clemson-clemson-control-1.clemson.edu/info/aggregate/clemson-og 

or

./rest_call.py -f -c ~/collector.pem https://bbn-cam-ctrl-1.gpolab.bbn.com/info/aggregate/gpo-og 

[dwiggins@bbn.com]

Replying to lnevers@…:

Replying to mbrinn@…:

#2 is odd. I can't reproduce this now: when I run the same URL it comes back happy. I'll try some different configurations.

Marshall I can see the error when I use the rest_call.py "-f" option (--follow-urls) and the return is validated by validictory:

 ./rest_call.py -f -c ~/collector.pem https://clemson-clemson-control-1.clemson.edu/info/aggregate/clemson-og 

or

./rest_call.py -f -c ~/collector.pem https://bbn-cam-ctrl-1.gpolab.bbn.com/info/aggregate/gpo-og 

You can see the problem without -f; I just tried. Putting -f won't hurt, but it doesn't affect validation.

[lnevers@bbn.com]

Replying to dwiggins@…:

You can see the problem without -f; I just tried. Putting -f won't hurt, but it doesn't affect validation.

When I tried the link directly I did not get failure, but I was using:

./rest_call.py -f -c ~/collector.pem https://192.1.243.2/info/link/urn_publicid_IDN_bbn-cam-ctrl-1.gpolab.bbn.com_sliver_link7db04f88-0a40-4a7b-9984-0f6836995c69

I can see the failure when I use:

./rest_call.py -c ~/collector.pem https://192.1.243.2/info/link/urn_publicid_IDN_bbn-cam-ctrl-1.gpolab.bbn.com_sliver_link7db04f88-0a40-4a7b-9984-0f6836995c69

[mbrinn@bbn.com]

I think this is all fixed. I got the rest_call.py to work for all HREF's on the GPO-OG rack. I put in an additional ops_vlaninterface and ops_link_vlaninterface for all stitching connections. Please give it a try and then let's close this out. - Marshall

[lnevers@bbn.com]

I now get 2 links showing an exception. Attaching output to show errors.

[mbrinn@bbn.com]

Another fix. Sorry. Seems to be working now. Please try it out when you get a chance. - Marshall

[lnevers@bbn.com]

Tried again, little better there is only one exception now:

$ ./rest_call.py -f -c ~/collector.pem https://bbn-cam-ctrl-1.gpolab.bbn.com/info/aggregate/gpo-og 

....<many lines deleted>....

Visiting https://bbn-cam-ctrl-1.gpolab.bbn.com/info/link/urn_publicid_IDN_bbn-cam-ctrl-1.gpolab.bbn.com_sliver_link7db04f88-0a40-4a7b-9984-0f6836995c69 returns:
{
    "$schema": "http://www.gpolab.bbn.com/monitoring/schema/20140501/link#", 
    "endpoints": [
        {
            "href": "https://bbn-cam-ctrl-1.gpolab.bbn.com/info/interfacevlan/urn_publicid_IDN_bbn-cam-ctrl-1.gpolab.bbn.com_sliver_interfacea2f6093e-bdc7-4ffa-a30c-493f85c8e7d2", 
            "urn": "urn:publicid:IDN+bbn-cam-ctrl-1.gpolab.bbn.com+sliver+interfacea2f6093e-bdc7-4ffa-a30c-493f85c8e7d2"
        }
    ], 
    "id": "urn_publicid_IDN_bbn-cam-ctrl-1.gpolab.bbn.com_sliver_link7db04f88-0a40-4a7b-9984-0f6836995c69", 
    "selfRef": "https://bbn-cam-ctrl-1.gpolab.bbn.com/info/link/urn_publicid_IDN_bbn-cam-ctrl-1.gpolab.bbn.com_sliver_link7db04f88-0a40-4a7b-9984-0f6836995c69", 
    "ts": 1409852416016642, 
    "urn": "urn:publicid:IDN+bbn-cam-ctrl-1.gpolab.bbn.com+sliver+link7db04f88-0a40-4a7b-9984-0f6836995c69"
}
Received exception Value [{u'urn': u'urn:publicid:IDN+bbn-cam-ctrl-1.gpolab.bbn.com+sliver+interfacea2f6093e-bdc7-4ffa-a30c-493f85c8e7d2', u'href': u'https://bbn-cam-ctrl-1.gpolab.bbn.com/info/interfacevlan/urn_publicid_IDN_bbn-cam-ctrl-1.gpolab.bbn.com_sliver_interfacea2f6093e-bdc7-4ffa-a30c-493f85c8e7d2'}] for field '<obj>.endpoints' must have length greater than or equal to 2 while trying to validate: {u'selfRef': u'https://bbn-cam-ctrl-1.gpolab.bbn.com/info/link/urn_publicid_IDN_bbn-cam-ctrl-1.gpolab.bbn.com_sliver_link7db04f88-0a40-4a7b-9984-0f6836995c69', u'urn': u'urn:publicid:IDN+bbn-cam-ctrl-1.gpolab.bbn.com+sliver+link7db04f88-0a40-4a7b-9984-0f6836995c69', u'ts': 1409852416016642L, u'$schema': u'http://www.gpolab.bbn.com/monitoring/schema/20140501/link#', u'endpoints': [{u'urn': u'urn:publicid:IDN+bbn-cam-ctrl-1.gpolab.bbn.com+sliver+interfacea2f6093e-bdc7-4ffa-a30c-493f85c8e7d2', u'href': u'https://bbn-cam-ctrl-1.gpolab.bbn.com/info/interfacevlan/urn_publicid_IDN_bbn-cam-ctrl-1.gpolab.bbn.com_sliver_interfacea2f6093e-bdc7-4ffa-a30c-493f85c8e7d2'}], u'id': u'urn_publicid_IDN_bbn-cam-ctrl-1.gpolab.bbn.com_sliver_link7db04f88-0a40-4a7b-9984-0f6836995c69'}
  Traceback (most recent call last):
  File "/usr/local/ops-monitoring/collector/response_validator.py", line 109, in validate
    validictory.validate(json_resp, schema, required_by_default=False)
  File "/usr/local/lib/python2.7/dist-packages/validictory/__init__.py", line 36, in validate
    return v.validate(data, schema)
  File "/usr/local/lib/python2.7/dist-packages/validictory/validator.py", line 543, in validate
    self.__validate("data", {"data": data}, schema, '<obj>')
  File "/usr/local/lib/python2.7/dist-packages/validictory/validator.py", line 566, in __validate
    validator(data, fieldname, schema, path, newschema.get(schemaprop))
  File "/usr/local/lib/python2.7/dist-packages/validictory/validator.py", line 244, in validate_properties
    path + '.' + eachProp)
  File "/usr/local/lib/python2.7/dist-packages/validictory/validator.py", line 566, in __validate
    validator(data, fieldname, schema, path, newschema.get(schemaprop))
  File "/usr/local/lib/python2.7/dist-packages/validictory/validator.py", line 429, in validate_minLength
    length=length, path=path)
  File "/usr/local/lib/python2.7/dist-packages/validictory/validator.py", line 176, in _error
    raise err
FieldValidationError: Value [{u'urn': u'urn:publicid:IDN+bbn-cam-ctrl-1.gpolab.bbn.com+sliver+interfacea2f6093e-bdc7-4ffa-a30c-493f85c8e7d2', u'href': u'https://bbn-cam-ctrl-1.gpolab.bbn.com/info/interfacevlan/urn_publicid_IDN_bbn-cam-ctrl-1.gpolab.bbn.com_sliver_interfacea2f6093e-bdc7-4ffa-a30c-493f85c8e7d2'}] for field '<obj>.endpoints' must have length greater than or equal to 2

Response from https://bbn-cam-ctrl-1.gpolab.bbn.com/info/link/urn_publicid_IDN_bbn-cam-ctrl-1.gpolab.bbn.com_sliver_link7db04f88-0a40-4a7b-9984-0f6836995c69 is NOT valid

....<many lines deleted>....

Response from https://bbn-cam-ctrl-1.gpolab.bbn.com/info/interface/urn_publicid_IDN_bbn-cam-ctrl-1.gpolab.bbn.com_node_bbn-cam-cmpe-2_eth2 is valid
24 URLs visited , 23 valid