bbnadmins do not have sudo on BBN rack worker nodes

[chaos@bbn.com]

I noticed that bbnadmins no longer have sudo on rack worker nodes. I see:

bbn-w2,[~],20:44(0)$ groups
nonrenci bbnadmins

bbn-w2,[~],20:44(0)$ sudo -l
[sudo] password for chaos: 
Sorry, user chaos may not run sudo on bbn-w2.

bbn-w2,[~],20:44(0)$ sudo whoami
[sudo] password for chaos: 
chaos is not in the sudoers file.  This incident will be reported.

Josh reported similar output from sudo -l on bbn-w1.

I'm pretty sure this used to work for us. Can someone take a look?

Thanks!

[jonmills@renci.org]

Please do not classify this as a 'major' flaw. No one has lost anything in the way of functionality.

A few months ago, we discovered a problem with having sudo in ldap on the worker nodes. The neuca client on the worker nodes polls sudo *a lot*. More than once a second, at times. This was effectively DDoSing the site LDAP replica servers (I first noticed when I couldn't explain the slow response times in ssh logins.)

Then, in an unrelated set of circumstances, ExoGENI Ops decided to remove the restriction to ban the /bin/su command from sudo. Doing any kind of real work was just getting ridiculous when having type 'sudo' in front of a zillion commands. Most of us were just bypassing it anyway via root passwords or shell escapes.

The result is that you can now 'sudo su - ' from a head node if you are a site admin.

Once you are root on your site's rack head node, you can, obviously, ssh without passwords (using keys) directly to any worker node, where you will be root.

We fixed the worker node sudo/ldap/DDoS problem by just removing the ldap database line from nsswitch.conf on those nodes:

[root@bbn-w4 ~]# grep sudoers /etc/nsswitch.conf sudoers: files [root@bbn-w4 ~]#

There isn't any net loss of functionality that I can see, and the result is that we aren't crushing LDAP replicas and sending more messages than rsyslog can possibly forward. Now, is it the ideal solution that I want long term? I don't know about that. I think we could potentially get somewhere with SSSD and credential caching. But we won't have SSSD until we upgrade all our infrastructure to CentOS 6.4, and even then, credential caching is a double-edged sword.

[chaos@bbn.com]

Downgrading to minor in deference to Jonathan. I'll verify that the other functionality works and (presumably) close on Monday.

[chaos@bbn.com]

Okay, indeed:

sudo ssh -i /opt/orca-12080/xcat/id_rsa root@bbn-w1

works. I think this should be on the ExoGENI wiki somewhere, so that site admins can find out where the SSH key file to use is.

[lnevers@bbn.com]

Adding a note to track where on the FIU rack the ssh key file is located:

 sudo ssh -i /etc/orca/am+broker-12080/xcat/id_rsa  root@fiu-w2 

[lnevers@bbn.com]

On 5/2/13 12:30 PM, Jonathan Mills wrote:

So this is a change that came with ORCA 4.0 and is not specific to FIU.

Adding to ticket, to capture that this is the new "and only" location for the ssh keys.

[lnevers@bbn.com]

This is no longer an issue, able to login to the bare metal nodes without specifying a path to the identity file. Closing ticket.