gmoc.py relational download hangs sometimes

[chaos@bbn.com]

Occasionally, gmoc.py relational download hangs while trying to obtain object data from gmoc-db. Since there is no timeout, it hangs forever and does not retry.

• Each connection to gmoc-db should time out after a certain amount of time
• Ideally, connections should retry once or twice before giving up on the entire download

[chaos@bbn.com]

I had to kill another hung retrieval process which had gotten wedged over the weekend, just now.

[chaos@bbn.com]

Listing this ticket as a GPO priority because it is causing occasional monitoring failures which we have to resolve by hand to unwedge our monitoring process.

[chaos@bbn.com]

gmoc.py-1.2.17 should address both of these:

• GMOCClient() now has optional timeout and retries options
• The timeout option is passed along to HTTP connections, and defaults to 30 seconds
• The retries option is used to retry downloads a few times when they fail, and defaults to 3

I've installed 1.2.17 on our monitoring server, where we have been seeing noisy failures (maybe mitigated by retries) on the order of once a day, and have been seeing hangs (maybe mitigated by timeout) on the order of once every few weeks. I'll follow up on whether the changes seem to help.

[chaos@bbn.com]

Since a change was made and this is an intermittent problem, i'll go ahead and close it. I might need to reopen later if it turns out we're still seeing hangs. But the "hang forever" behavior only arises every few weeks, so it'll take awhile to get good data.