Slow logins on BBN IG PG VMs

[jbs@bbn.com]

When I create a PG VM on the BBN InstaGENI rack, logging in to it is very slow.

[jbs@bbn.com]

It takes about thirty seconds, and doesn't seem to be a DNS problem -- once I'm logged in, I can look up hostnames just fine.

'ssh -v' says

debug1: Next authentication method: gssapi-with-mic 
debug1: Unspecified GSS failure.  Minor code may provide more information 
Credentials cache file '/tmp/krb5cc_1000' not found 

debug1: Unspecified GSS failure.  Minor code may provide more information 
Credentials cache file '/tmp/krb5cc_1000' not found 

debug1: Unspecified GSS failure.  Minor code may provide more information 


debug1: Next authentication method: publickey 

...hanging before each "unspecified GSS failure" for ten seconds or so, and then zipping right along once it gets around to pubkey. Not sure what's amiss there; this looks a *lot* like the image on the Utah IG rack, where my VM has 'GSSAPIAuthentication yes' in sshd_config (and the same results for 'sudo grep -i gss /etc/ssh/sshd_config' overall), but doesn't have this problem.

Huh, although 'ssh -v' to my Utah VM also contains the GSS failure messages:

debug1: Next authentication method: gssapi-with-mic 
debug1: Unspecified GSS failure.  Minor code may provide more information 
Credentials cache file '/tmp/krb5cc_1000' not found 

debug1: Unspecified GSS failure.  Minor code may provide more information 
Credentials cache file '/tmp/krb5cc_1000' not found 

debug1: Unspecified GSS failure.  Minor code may provide more information 


debug1: Next authentication method: publickey 

It just plows through the errors quickly. (grin) So I'm not sure what it's doing while it's pausing; I can try to find out, if y'all can't reproduce this.

[jbs@bbn.com]

Leigh said:

A few oddities here.

First off you are correct that GSS should be turned off. I will need to fix
the openvz image. Might not happen today.

The next thing I noticed is that you see a delay and I do not. But more on
that in a moment.

Google indicates that lack of reverse DNS causes the delay when GSS is
turned on. But the VMs do reverse resolve within the rack, but I do not
know what actual VM you were trying or where you were coming from.

The next thing I noticed is that the authentication order was odd; it had
GSS first and publickey at the end. Which ultimately is why there is no
delay for me; when I log in publickey is at the beginning of the list, and
so it never has to try GSS. I notice that you can set this order in your
~/.ssh/config file. Do you have GSS configured ahead of publickey in your
config file?

[jbs@bbn.com]

First off you are correct that GSS should be turned off. I will need to fix the openvz image. Might not happen today.

Ok, cool.

The next thing I noticed is that you see a delay and I do not. But more on that in a moment.

Huh, interesting.

Google indicates that lack of reverse DNS causes the delay when GSS is turned on. But the VMs do reverse resolve within the rack, but I do not know what actual VM you were trying or where you were coming from.

Who needs to have the reverse DNS, the client or the server? I would've thought the server, but maybe the client's trying to do something here.

That seems to be borne out empirically: When I log in to my bbn-ig-jbs15 VM (ssh pc2.instageni.gpolab.bbn.com -p 30010), it's fast from ops.instageni.gpolab.bbn.com, but slow from my desktop on BBN's internal network.

Huh, but that's interesting: It's also fast from a machine on BBN's external network. Is the problem that the VM can't look up the IP address of my internal machine? No, that seems to work:

[07:42:15] jbs@bbn-ig-jbs15:/users/jbs
+$ host 128.89.80.108
108.80.89.128.in-addr.arpa domain name pointer anubis.bbn.com.

(Huh, that timestamp is wrong; I'll make another ticket for that.)

So, I'm not sure what's going on here: I can't look up 192.1.242.141 from either a BBN intnet (128.89.80.108) or a BBN extnet (192.1.249.134) host, and I can look up both the BBN intnet and BBN extnet IP addresses from my VM on 192.1.242.141. So why can I log in quickly from BBN extnet and not BBN intnet? Hoom.

The next thing I noticed is that the authentication order was odd; it had GSS first and publickey at the end. Which ultimately is why there is no delay for me; when I log in publickey is at the beginning of the list, and so it never has to try GSS. I notice that you can set this order in your ~/.ssh/config file. Do you have GSS configured ahead of publickey in your config file?

Hmm, I wouldn't have thought so. My ~/.ssh/config says

PreferredAuthentications publickey

at the bottom (outside of any Host context), so I don't think I should be doing any GSS anything at any point in here. Hmm.

What does your SSH config look like to avoid GSS?

[jbs@bbn.com]

Leigh says:

> What does your SSH config look like to avoid GSS?

I don't do anything.

Well, hoom.

My inclination is to fix the DNS stuff on ticket #64, and see if that changes things here. If it doesn't, we should probably dig further; if it does, though, I'm sort of content to conclude that this is somehow BBN-specific (since the problem only seems to manifest when you log in from BBN intnet)

[jbs@bbn.com]

My inclination is to fix the DNS stuff on ticket #64, and see if that changes things here.

Leigh did that, and indeed, my login is now quick from BBN intnet too. Closing this one out.