Opened 7 years ago

Closed 6 years ago

#74 closed (fixed)

InstaGENI Compute resource aggregate in GMOC no longer shows Slivers and Resources data

Reported by: lnevers@bbn.com Owned by: sedwards@bbn.com
Priority: major Milestone:
Component: AM Version: SPIRAL5
Keywords: Cc:
Dependencies:

Description

The GPO InstaGENI rack (instageni.gpolab.bbn.com:12369) is now part of the the list of aggregates at:

https://gmoc-db.grnoc.iu.edu/protected-openid/index.pl?method=aggregates&search=instageni.gpolab.bbn.com+12369

But there are no details for the for "Slivers" and "Resources", see :

https://gmoc-db.grnoc.iu.edu/protected-openid/index.pl?method=aggregate_details;aggregate=instageni.gpolab.bbn.com%3A12369

Also checked the Utah aggregate (utah.geniracks.net:12369) and it also has no "Slivers" and no "Resources".

This problem may be related to the addition of port 12369 URL change. Sarah is investigating.

Change History (9)

comment:1 Changed 7 years ago by sedwards@bbn.com

Status: newassigned

comment:2 Changed 7 years ago by sedwards@bbn.com

Ok. So I originally thought this was going to be a problem with my aggregate/resource code which Leigh is running. However, closer inspection leads me to believe this may be a subtle gmoc client data submission bug.

In particular, the UI doesn't show there being any resources for these two aggregates:

instageni.gpolab.bbn.com:12369
utah.geniracks.net:12369

These formerly were reported as:

instageni.gpolab.bbn.com:80
utah.geniracks.net:80 

In particular I now notice that the old aggregate utah.geniracks.net:80 hasn't updated for awhile: https://gmoc-db.grnoc.iu.edu/protected-openid/index.pl?method=aggregate_details;aggregate=utah.geniracks.net%3A80

But the resources for it are listed as having reported today: https://gmoc-db.grnoc.iu.edu/protected-openid/index.pl?method=aggregate_details;aggregate=utah.geniracks.net%3A80

Likewise the old aggregate instageni.gpolab.bbn.com:80 hasn't updated in awhile: https://gmoc-db.grnoc.iu.edu/protected-openid/index.pl?method=aggregate_details;aggregate=instageni.gpolab.bbn.com%3A80

HOWEVER there are no resources for it listed at all: https://gmoc-db.grnoc.iu.edu/protected-openid/index.pl?method=aggregate_details;aggregate=instageni.gpolab.bbn.com%3A80

The first case (the utah.geniracks.net case) makes me think the link between the aggregate and the resources wasn't updated when the aggregate name changed. However, the second case doesn't clearly support that thinking.

I remember that we had some discussions about things like what can be updated once it's reported when Kevin built the client so this may be somewhat expected behavior. That said, I'm not sure how to proceed and get the resources linked to the new aggregate. I've emailed Kevin and Mitch to get them to poke at the DB and see if they can provide additional guidance here.

comment:3 Changed 7 years ago by sedwards@bbn.com

One more data point...

I found a resource that looks like it is an IG node on the GPO rack: https://gmoc-db.grnoc.iu.edu/protected-openid/index.pl?method=node_details;node=urn%3Apublicid%3AIDN%2Binstageni.gpolab.bbn.com%2Bnode%2Bpc1

But it doesn't like it has an aggregate at all.

Also, we switched the reported name on the instageni.gpolab.com rack pretty quickly after reporting it at all. Whereas utah.geniracks.net had been reporting for awhile before we made the switch.

comment:4 Changed 7 years ago by sedwards@bbn.com

I spoke to the GMOC folks and they are going to look at this soon and at least nail down the issue before the next release (which is supposed to be this week).

I created a GMOC ticket here: http://groups.geni.net/gmoc/ticket/146

comment:5 Changed 7 years ago by sedwards@bbn.com

Mitch looked into what was going on and reminded me of some salient facts which gave me an idea. I'll try that out and report back.

comment:6 Changed 6 years ago by sedwards@bbn.com

Ok. Fixed some stuff with my code but when I submit I get an error. Chaos is actively debugging with the GMOC folks. So I'll work on debugging this a bit later. I'll also may need to update my version of gmoc.py.

<?xml version="1.0" ?>
<gmoc_topology time="1358904776" version="4">
	<pop name="urn:publicid:IDN+gmoc.geni.net+pop+gpolab"/>
	<aggregate name="instageni.gpolab.bbn.com:12369">
		<sliver approved="false" created="1356059087" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+jbs" expires="1359491450" local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+144" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs" slice_uuid="ee66ba9f-00fe-11e1-bf86-000c29f89f7b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+144" resource="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc3" type="host"/>
		</sliver>
		<sliver approved="false" created="1357668991" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+tupty" expires="1359491485" local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+252" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tipdemoprimary" slice_uuid="aa964ac2-59cb-11e2-bf18-000c29cca98b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+252" resource="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc2" type="vm"/>
		</sliver>
		<sliver approved="false" created="1357669288" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+tupty" expires="1359491485" local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+256" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tipdemobackup" slice_uuid="94207f53-59cb-11e2-bf18-000c29cca98b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+256" resource="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc1" type="vm"/>
		</sliver>
		<sliver approved="false" created="1358860100" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expires="1359491497" local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+766" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-EXP-7" slice_uuid="1f103145-64a5-11e2-bf18-000c29cca98b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+766" resource="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc2" type="vm"/>
			<resource_mapping local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+766" resource="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc2" type="vm"/>
			<resource_mapping local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+766" resource="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc2" type="vm"/>
		</sliver>
		<sliver approved="false" created="1355445356" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+jbs" expires="1359491530" local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+10" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs15" slice_uuid="ff9a2477-a41c-11e0-acc6-000c29f89f7b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+10" resource="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc2" type="vm"/>
		</sliver>
		<sliver approved="false" created="1355451614" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+jbs" expires="1359491530" local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+14" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs16" slice_uuid="30ec474a-a41d-11e0-acc6-000c29f89f7b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+14" resource="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc2" type="vm"/>
		</sliver>
		<sliver approved="false" created="1357324609" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+tupty" expires="1359491546" local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+211" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tuptymon" slice_uuid="88a518f0-af53-11e1-8a62-000c29cca98b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+211" resource="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc1" type="vm"/>
		</sliver>
		<sliver approved="false" created="1357668991" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+tupty" expires="1359491546" local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+252" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tipdemoprimary" slice_uuid="aa964ac2-59cb-11e2-bf18-000c29cca98b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+252" resource="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc2" type="vm"/>
		</sliver>
		<sliver approved="false" created="1357669288" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+tupty" expires="1359491546" local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+256" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tipdemobackup" slice_uuid="94207f53-59cb-11e2-bf18-000c29cca98b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+256" resource="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc1" type="vm"/>
		</sliver>
		<sliver approved="false" created="1358719104" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expires="1359491558" local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+713" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-EXP-5-exp1" slice_uuid="1db446f5-635d-11e2-bf18-000c29cca98b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+713" resource="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc1" type="vm"/>
		</sliver>
		<sliver approved="false" created="1358860100" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expires="1359491558" local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+766" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-EXP-7" slice_uuid="1f103145-64a5-11e2-bf18-000c29cca98b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+766" resource="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc2" type="vm"/>
			<resource_mapping local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+766" resource="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc2" type="vm"/>
			<resource_mapping local_name="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+766" resource="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc2" type="vm"/>
		</sliver>
	</aggregate>
</gmoc_topology>

DBD::mysql::st execute failed: Duplicate entry '-1-1578-301' for key 1 at xchange2db3 line 1046.
Could not execute statement INSERT INTO sliver_resource_map (end_version_id, sliver_id, resource_id, name, start_version_id, type) VALUES (-1, ?, ?, ?, ?, ?).


<?xml version="1.0" ?>
<gmoc_topology time="1358904777" version="4">
	<pop name="urn:publicid:IDN+gmoc.geni.net+pop+Utah"/>
	<aggregate name="utah.geniracks.net:12369">
		<sliver approved="false" created="1355931143" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+jbs" expires="1359491450" local_name="urn:publicid:IDN+utah.geniracks.net+sliver+15762" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs16" slice_uuid="30ec474a-a41d-11e0-acc6-000c29f89f7b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+utah.geniracks.net+sliver+15762" resource="urn:publicid:IDN+utah.geniracks.net+node+pc3" type="vm"/>
		</sliver>
		<sliver approved="false" created="1355932347" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+jbs" expires="1359491450" local_name="urn:publicid:IDN+utah.geniracks.net+sliver+15770" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+jbs15" slice_uuid="ff9a2477-a41c-11e0-acc6-000c29f89f7b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+utah.geniracks.net+sliver+15770" resource="urn:publicid:IDN+utah.geniracks.net+node+pc3" type="vm"/>
		</sliver>
		<sliver approved="false" created="1348838938" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+tupty" expires="1359491466" local_name="urn:publicid:IDN+utah.geniracks.net+sliver+11075" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tupty1755" slice_uuid="483eb6c2-7ffd-11e1-8136-000c29cca98b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+utah.geniracks.net+sliver+11075" resource="urn:publicid:IDN+utah.geniracks.net+node+pc1" type="vm"/>
		</sliver>
		<sliver approved="false" created="1357669388" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+tupty" expires="1359491485" local_name="urn:publicid:IDN+utah.geniracks.net+sliver+15882" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tipdemobackup" slice_uuid="94207f53-59cb-11e2-bf18-000c29cca98b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+utah.geniracks.net+sliver+15882" resource="urn:publicid:IDN+utah.geniracks.net+node+pc1" type="vm"/>
		</sliver>
		<sliver approved="false" created="1358860132" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expires="1359491497" local_name="urn:publicid:IDN+utah.geniracks.net+sliver+16304" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-EXP-7" slice_uuid="1f103145-64a5-11e2-bf18-000c29cca98b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+utah.geniracks.net+sliver+16304" resource="urn:publicid:IDN+utah.geniracks.net+node+pc5" type="vm"/>
			<resource_mapping local_name="urn:publicid:IDN+utah.geniracks.net+sliver+16304" resource="urn:publicid:IDN+utah.geniracks.net+node+pc5" type="vm"/>
			<resource_mapping local_name="urn:publicid:IDN+utah.geniracks.net+sliver+16304" resource="urn:publicid:IDN+utah.geniracks.net+node+pc5" type="vm"/>
		</sliver>
		<sliver approved="false" created="1358879176" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expires="1359491497" local_name="urn:publicid:IDN+utah.geniracks.net+sliver+16317" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+acclne-202527" slice_uuid="dcdfe7b7-64d1-11e2-bf18-000c29cca98b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+utah.geniracks.net+sliver+16317" resource="urn:publicid:IDN+utah.geniracks.net+node+pc1" type="vm"/>
			<resource_mapping local_name="urn:publicid:IDN+utah.geniracks.net+sliver+16317" resource="urn:publicid:IDN+utah.geniracks.net+node+pc1" type="vm"/>
		</sliver>
		<sliver approved="false" created="1358879277" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expires="1359491497" local_name="urn:publicid:IDN+utah.geniracks.net+sliver+16326" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+acclne-2026360" slice_uuid="05de9c5f-64d2-11e2-bf18-000c29cca98b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+utah.geniracks.net+sliver+16326" resource="urn:publicid:IDN+utah.geniracks.net+node+pc3" type="vm"/>
			<resource_mapping local_name="urn:publicid:IDN+utah.geniracks.net+sliver+16326" resource="urn:publicid:IDN+utah.geniracks.net+node+pc3" type="vm"/>
		</sliver>
		<sliver approved="false" created="1357850511" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+tupty" expires="1359491515" local_name="urn:publicid:IDN+utah.geniracks.net+sliver+15997" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tipdemoprimary" slice_uuid="aa964ac2-59cb-11e2-bf18-000c29cca98b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+utah.geniracks.net+sliver+15997" resource="urn:publicid:IDN+utah.geniracks.net+node+pc5" type="vm"/>
		</sliver>
		<sliver approved="false" created="1357669388" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+tupty" expires="1359491546" local_name="urn:publicid:IDN+utah.geniracks.net+sliver+15882" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tipdemobackup" slice_uuid="94207f53-59cb-11e2-bf18-000c29cca98b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+utah.geniracks.net+sliver+15882" resource="urn:publicid:IDN+utah.geniracks.net+node+pc1" type="vm"/>
		</sliver>
		<sliver approved="false" created="1358860132" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expires="1359491558" local_name="urn:publicid:IDN+utah.geniracks.net+sliver+16304" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-EXP-7" slice_uuid="1f103145-64a5-11e2-bf18-000c29cca98b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+utah.geniracks.net+sliver+16304" resource="urn:publicid:IDN+utah.geniracks.net+node+pc5" type="vm"/>
			<resource_mapping local_name="urn:publicid:IDN+utah.geniracks.net+sliver+16304" resource="urn:publicid:IDN+utah.geniracks.net+node+pc5" type="vm"/>
			<resource_mapping local_name="urn:publicid:IDN+utah.geniracks.net+sliver+16304" resource="urn:publicid:IDN+utah.geniracks.net+node+pc5" type="vm"/>
		</sliver>
		<sliver approved="false" created="1358879176" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expires="1359491558" local_name="urn:publicid:IDN+utah.geniracks.net+sliver+16317" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+acclne-202527" slice_uuid="dcdfe7b7-64d1-11e2-bf18-000c29cca98b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+utah.geniracks.net+sliver+16317" resource="urn:publicid:IDN+utah.geniracks.net+node+pc1" type="vm"/>
			<resource_mapping local_name="urn:publicid:IDN+utah.geniracks.net+sliver+16317" resource="urn:publicid:IDN+utah.geniracks.net+node+pc1" type="vm"/>
		</sliver>
		<sliver approved="false" created="1358879277" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers" expires="1359491558" local_name="urn:publicid:IDN+utah.geniracks.net+sliver+16326" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+acclne-2026360" slice_uuid="05de9c5f-64d2-11e2-bf18-000c29cca98b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+utah.geniracks.net+sliver+16326" resource="urn:publicid:IDN+utah.geniracks.net+node+pc3" type="vm"/>
			<resource_mapping local_name="urn:publicid:IDN+utah.geniracks.net+sliver+16326" resource="urn:publicid:IDN+utah.geniracks.net+node+pc3" type="vm"/>
		</sliver>
		<sliver approved="false" created="1351079507" creator="urn:publicid:IDN+pgeni.gpolab.bbn.com+user+tupty" expires="1359491575" local_name="urn:publicid:IDN+utah.geniracks.net+sliver+13957" slice_urn="urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+tuptymon" slice_uuid="88a518f0-af53-11e1-8a62-000c29cca98b" state="Unknown" uuid="">
			<resource_mapping local_name="urn:publicid:IDN+utah.geniracks.net+sliver+13957" resource="urn:publicid:IDN+utah.geniracks.net+node+pc1" type="vm"/>
		</sliver>
	</aggregate>
</gmoc_topology>

DBD::mysql::st execute failed: Duplicate entry '-1-1582-182' for key 1 at xchange2db3 line 1046.
Could not execute statement INSERT INTO sliver_resource_map (end_version_id, sliver_id, resource_id, name, start_version_id, type) VALUES (-1, ?, ?, ?, ?, ?).

comment:7 Changed 6 years ago by sedwards@bbn.com

Chaos looked over this and we agree there seems to be a problem with the DB at the GMOC end.

We'll propose debugging this after the DB cutover early next week.

comment:8 Changed 6 years ago by sedwards@bbn.com

Did a few more rounds of bug fixes. I'm able to submit code again, but it's not showing up in the UI interface and Chaos can't download it.

I sent mail to the GMOC guys and will follow up tomorrow.

comment:9 Changed 6 years ago by lnevers@bbn.com

Resolution: fixed
Status: assignedclosed

This problem is resolved. Resolution verified by IG-CT-5 Monitoring tests for GPO, Northwestern and Kentucky. Closing ticket.

Note: See TracTickets for help on using tickets.