A 2 VM sliver (1@InstaGENI & 1@PG Utah) shows pg_status 'notready' 40 minutes after slivercreate

[lnevers@bbn.com]

The slice used for this experiment is urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+1nsta-1pg and was created at around 4:21 pm eastern.

Twenty minutes after the two sliver were created at InstaGENI AM and PG Utah AM, the pg_status' was still 'notready'.

[lnevers@bbn.com]

Still in the same state after 40 minutes.

[lnevers@bbn.com]

On 5/9/12 5:14 PM, Leigh Stoller wrote:

Still in the same state after 40 minutes.

Yep, something is whacky. Please leave this slice active.

Thanks! Lbs

Any update on this ticket?

[lnevers@bbn.com]

Re-ran the experiment with 1 VM @instageni and 1 VM @ PG Utah, and again got the 2 nodes that were requested.

Connected to each host and pinged the remote, and found poor connectivity. On InstaGENI Rack host:

[lnevers@VM-0 ~]$ ping 192.168.3.2
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
<<output deleted>>>
--- 192.168.3.2 ping statistics ---
65 packets transmitted, 16 received, 75% packet loss, time 64001ms
rtt min/avg/max/mdev = 0.172/34.236/544.708/131.803 ms

On PG Utah Rack:

[lnevers@VM ~]$ ping 192.168.3.1
PING 192.168.3.1 (192.168.3.1) 56(84) bytes of data.
<<<output deleted>>>>
--- 192.168.3.1 ping statistics ---
138 packets transmitted, 19 received, 86% packet loss, time 137009ms
rtt min/avg/max/mdev = 0.204/123.744/1629.581/381.678 ms, pipe 2
[lnevers@VM ~]$ 

[lnevers@bbn.com]

Synching up ticket with email exchange:

On 5/10/12 12:46 PM, Leigh Stoller wrote:

Could you please elaborate on the fix.

Hi. I forgot to kill dhcpd on the control node, so we had two dhcpds running, which is very bad for Emulab.

Lbs

On 5/10/12 1:02 PM, Leigh Stoller wrote:

Re-ran the experiment with 1 VM @instageni and 1 VM @ PG Utah, and again

got the 2 nodes that were requested.

Connected to each host and pinged the remote, and found poor connectivity.

Yep, looks bad. Please leave this active so I can poke around.

Lbs

[lnevers@bbn.com]

Re-ran the experiment with 1 VM @InstaGENi and 1 VM @ PG Utah.

Both InstaGENI and PG Utah slivers were created at the same time without error. The InstaGENI VM was available and ready for login in a few minutes, but 40 minutes after creating the sliver, the PG Utah node is still note ready.

 'geni_status': 'unknown',
 'geni_urn': 'urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+1nsta-1pg',
 'pg_expires': '2012-05-25T17:31:49',
 'pg_status': 'notready',

Unable to verify issue resolution.

[lnevers@bbn.com]

Re-ran this scenario at approx 10 AM and was able to successfully set up the sliver, slice named "1vminsta-1vmpgutah2".

Once logged in allocated VMs, could not exchange any traffic between the InstaGENI VM endpoint and the PG Utah VM endpoint. Also noticed that the InstaGENI VM did not use the hostname defined in the request RSpec. The command prompt showed:

"[lnevers@unknown ~]$ "

rather than the expected

"[lnevers@VM-0 ~]$ "

Attaching sliver manifests, and leaving the slivers running.

[lnevers@bbn.com]

On 6/6/12 1:38 PM, Leigh Stoller wrote:

Re-ran this scenario at approx 10 AM and was able to successfully set up the sliver, slice named "1vminsta-1vmpgutah2".

Once logged in allocated VMs, could not exchange any traffic between the InstaGENI VM endpoint and the PG Utah VM endpoint.

This should be fixed ... It was a firewall problem.

Also noticed that the InstaGENI VM did not use the hostname defined in the request RSpec. The command prompt showed:

"[lnevers@unknown ~]$ "

I logged in and it looked okay:

[root@VM-0 ~]# hostname VM-0.1vminsta-1vmpgutah2.pgeni-gpolab-bbn-com.utah.geniracks.net

I am now able to exchange traffic between the two endpoints. Problem is solved, Closing ticket.