every time data is reported, new entries are added to *_instantiation tables

[chaos@bbn.com]

The aggregate_instantiation table contains one entry for every aggregate report which has ever been received, meaning that it gains one line every five minutes for every reporting aggregate. I believe this is unnecessary: the purpose of the table is just to store aggregate metadata, and this will bog down the database eventually.

This issue affects other *_instantiation tables, including:

interface_instantiation
resource_instantiation
slice_authority_instantiation
slice_instantiation

[chaos@bbn.com]

It's not just the aggregate_instantiation table which has this behavior --- i'll add other tables as i come across them. We identified this problem prior to GEC14, and i thought it actually did cause some problems then, though i don't remember details and may be wrong.

[chaos@bbn.com]

I think there are also downstream effects on interface_netaddr_map and interface_vlap_map, which maybe map based on interface_instantiation rather than on interface, though i'm not as sure what's going on there.

[chaos@bbn.com]

We think this should be fixed, and will look at it more tomorrow.

[chaos@bbn.com]

The partition on which this database sits is only 37% full, so this bug is not going to break a GEC15 demo (so i'm not going to list it as a GEC15 blocker).

[chaos@bbn.com]

Last Wednesday, we hit a tipping point of some sort in terms of the number of entries in gmoc-db, and submissions became slow enough that many/most new submissions started failing due to timeouts. As a result, we effectively do not have usable GMOC data for GENI right now.

On Thursday, Mitch offered to remove some entries by hand as a stopgap measure.

I'm raising the priority of this ticket to blocker, because monitoring data is currently not usable.

[chaos@bbn.com]

Mitch pushed the fix out onto gmoc-db in a maintenance window today, so that should prevent new duplicate entries from being created. He's going to keep an eye on the situation for a few days, and then schedule another maintenance to run a script to remove existing duplicate entries.

[chaos@bbn.com]

Josh noticed that a side effect of Mitch's change is that the "Last Updated" column in the web UI is no longer accurate --- it's turned into "Last Modified". See e.g. ​https://gmoc-db.grnoc.iu.edu/protected-openid/index.pl?method=aggregates&search=3626. We think we prefer "Last Updated", so people can use that field to infer whether data is being submitted, whether a sliver has gone away, etc.

Mitch thinks he can change the code so that the "Last Updated" column always gets updated even if there are no other changes, while still using the logic of the new less-database-inserting code. We're testing that out on gmoc-db2 now.

[chaos@bbn.com]

Mitch pushed out the code with that fix onto gmoc-db2 and (since we're still in the outage window) gmoc-db. I see updated "Last Updated" fields for foam5.gpolab on gmoc-db2 and foam1.gpolab on gmoc-db. I'll keep an eye on this for the next while, but it looks like this was fixed.

[chaos@bbn.com]

Updating milestone to indicate that this is an active priority. (Mitch is actively working on finishing it).

[chaos@bbn.com]

We think this is basically done.