Issuing reboot on GPO and Utah IG nodes results in node not available

[lnevers@bbn.com]

In several scenario, both at Utah and GPO InstaGENI, issuing a restart causes for the node to not come back up:

[lnevers@top ~]$ sudo shutdown -r now
Failed to talk to init daemon.
[lnevers@top ~]$ Connection to pc2.instageni.gpolab.bbn.com closed by remote host.
Connection to pc2.instageni.gpolab.bbn.com closed.
lnevers@arendia:~/gcf-2.2-rc4$ sleep 360
lnevers@arendia:~/gcf-2.2-rc4$ ssh -p 30780  lnevers@pc2.instageni.gpolab.bbn.com
ssh: connect to host pc2.instageni.gpolab.bbn.com port 30780: Connection refused

There are currently 4 nodes that are part of the slice IG-EXP-7 and were restarted and 30 minutes later are still not accepting ssh connections:

• pc2.instageni.gpolab.bbn.com port 30780
• pc2.instageni.gpolab.bbn.com port 30779
• pc3.utah.geniracks.net port 32060
• pc3.utah.geniracks.net port 32058

After the restart, the nodes am_status changes from "ready" to "notready".

[lnevers@bbn.com]

Jon, thanks for the detailed explanation. Since restarting the nodes can be done with the AM API V3 geni_restart or (geni_stop + geni_start). I believe this is low priority. Capturing email exchange to track exchange.

On 1/15/13 12:05 PM, Jonathon Duerig wrote:

This is a known issue. The client-side on the shared nodes does not automatically 
restart stopped vms. When rebooting nodes, you will need to use the AM or CM 
interfaces. 

On Tue, 15 Jan 2013, Luisa Nevers wrote:

Could you please elaborate on what you mean by "use the AM or CM interfaces"?

On 1/15/13 1:28 PM, Jonathon Duerig wrote:

When you shut down a vm from within the node, there is nothing in the root 
context that will restart it. There doesn't seem to be an easy way to distinguish 
between when a 'shutdown -r' and a 'shutdown' command was issued inside the vm.

Solving this will require writing a new daemon that monitors vm state, knows which
nodes should be booted, and restarting them if they shut down. Thus far, this hasn't 
been a high priority.

However, you can still start/restart nodes by talking to the aggregate manager 
directly. The CMv2 interface has commands for booting/rebooting nodes.

So does the AMv3 interface. And although, there are some calls in AMv3 that I know 
don't work properly at the moment, changing the operational state does AFAIK. To
 restart (or start a node after having shut it down from inside), you need to contact
 the aggregate manager and send it a geni_restart (or geni_start) command. 

[lnevers@bbn.com]

Did not intend to close ticket. Re-opening and lowering priority to minor to track resolution described by Jon.