organization metadata submission needs an overhaul

[chaos@bbn.com]

When new GENI sites start submitting monitoring data to GMOC, they follow the procedure at ​http://groups.geni.net/geni/wiki/GENIMetaOps/SiteCredentials to report their credentials and contact info to the GMOC service desk. The service desk then follows an internal procedure to add submission credentials to gmoc-db, and also to add organizational metadata to gmoc-db tables.

Currently, the data which is added to gmoc-db via this procedure:

• Is not sufficient for site data submission via gmoc.py (so sites need to submit some separate metadata using gmoc.py itself before they can submit data)
• Is not sufficient for Organization object download via gmoc.py (so download of metadata for organizations and aggregates which have been setup this way, fail)
• Does not contain all supported metadata which the service desk would like (e.g. escalation contact, organization type, which are in the database, but don't get populated via this procedure)

Make any needed changes so that the service desk's procedure leads to gmoc-db data which contains the fields the Service Desk needs, and which is compatible with gmoc.py upload and download.

[chaos@bbn.com]

Eldar and i discussed this for awhile with input from Dan and Mitch, and we think the following things needs to be done:

1. A minor database change --- the Service Desk has been using the organization.name field to contain a display name (with whitespace), while gmoc.py users have been using it to contain a URN. So we should change to having separate displayname and urn fields. (I'd prefer to continue with the field which contains a URN as the unique key for the table.)
2. Determine the full set of new database entries which are needed so that, when a new site comes on board:
   1. gmoc.py upload of aggregate/resource data using that stub organization/POP succeeds
   2. gmoc.py download of that Organization object (which includes the POP) succeeds
3. Modify the web UI the service desk uses to submit new organizations so that it populates all the fields in the manner needed for (2), as well as all the fields which the service desk needs.
4. Test everything, and update public documentation for GENI sites
5. Fix existing organizations in the database so that their metadata works with gmoc.py and contains the information the service desk needs

[chaos@bbn.com]

Just to clarify one thing: during discussion last week, we decided that the fix we want is:

• The GMOC NOC submits all Organization/POP metadata based on information they get from new submission clients
• That initial submission sets all table entries which are needed for relational data submission to work. Therefore, there is no need for report_site_metadata, and we get rid of it
• The Organization/POP table entries are immutable once they've been set

[chaos@bbn.com]

Mitch rolled out new code onto gmoc-db2, and a staging copy of the web UI which the service desk uses to populate new organizations and contacts. Eldar and i staged a test of adding a new organization, and found the following issues:

1. There's no service desk UI to instantiate a new POP. If we're going to have the service desk do that rather than use report_site_metadata (and we agreed we would--- report_site_metadata is not up for the job, and if POP table entries are immutable, some person should be responsible for maintaining them)
2. The new organization script needs the service desk user to submit the organization name as a URN, but doesn't communicate this. It would be better to provide documentation, provide a default, or (best IMO) simply generate a URN-style value
3. The organization script doesn't fill in an organization type. Two issues here:
   • Eldar would prefer that the service desk user be able to select the type to be accurate (not technically part of this ticket, but a longstanding service desk request)
   • gmoc.py download fails if the type isn't set
4. The new contact script doesn't allow the service desk to submit a URN for the contact, nor (again, better) generate a valid one behind the scenes for the benefit of the service desk user.

Mitch is going to work on fixing 3 and 4, after which Eldar and i will test again.

[chaos@bbn.com]

• Yesterday, we tested improvements for items 2-4, and those seem to all be resolved on gmoc-db2. Eldar and i were able to test a full insertion of a new Org/new contacts, with an existing POP, and i was then able to submit and download data for that Organization
• Dan is going to work on the new POP UI (item 1), with an ETA of early-to-mid next week. We don't want to block cleaning up the data in gmoc-db on that, so i spun that task off into another ticket (GMT 159). Meanwhile, the workaround is for new POP entries to be inserted by hand.
• Mitch is going to make the database column change and Organization/Contact? submission UI changes after lunch today.

[chaos@bbn.com]

I'm going to go ahead and close this ticket, because i think we did this work. We may want to do further cleanup later, but let's open new tickets for those new tasks.