wiki:GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-ADM-2

Version 15 (modified by chaos@bbn.com, 12 years ago) (diff)

--

  1. Detailed test plan for IG-ADM-2: Rack Administrator Access Test
    1. Page format
    2. Status of test
    3. High-level description from test plan
        1. Procedure
      1. Criteria to verify as part of this test
    4. Step 1: verify access to rack boss node
      1. Step 1A: verify that SSH to boss succeeds and allows public keys only
        1. Results of testing step 1A: 2012-05-15
      2. Step 1B: verify the absence of common unencrypted login protocols
        1. Results of testing step 1B: 2012-05-16
      3. Step 1C: verify sudo and sudo logging
        1. Results of testing step 1C: 2012-05-16
    5. Step 2: verify access to rack ops node
      1. Step 2A: verify that SSH to ops succeeds and allows public keys only
        1. Results of testing step 2A: 2012-05-16
        2. Results of testing step 2A: 2012-05-21
      2. Step 2B: verify the absence of common unencrypted login protocols
        1. Results of testing step 2B: 2012-05-16
      3. Step 2C: verify sudo and sudo logging
        1. Results of testing step 2C: 2012-05-16
    6. Step 3: verify access to rack foam node
      1. Step 3A: verify that SSH to foam succeeds and allows public keys only
        1. Results of testing step 3A: 2012-11-12
      2. Step 3B: verify the absence of common unencrypted login protocols
        1. Results of testing step 3B: 2012-11-12
      3. Step 3C: verify sudo and sudo logging
        1. Results of testing step 3C: 2012-11-12
    7. Step 4: verify access to rack FlowVisor node
      1. Step 4A: verify that SSH to FlowVisor succeeds and allows public keys only
        1. Results of testing step 4A: 2012-11-12
      2. Step 4B: verify the absence of common unencrypted login protocols
        1. Results of testing step 4B: 2012-11-12
      3. Step 4C: verify sudo and sudo logging
        1. Results of testing step 4C: 2012-11-12
    8. Step 5: verify access to rack infrastructure VM server host
      1. Step 5A: verify that SSH to infrastructure host succeeds and allows …
        1. Results of testing step 5A: 2012-05-16
      2. Step 5B: verify the absence of common unencrypted login protocols
        1. Results of testing step 5B: 2012-05-17
      3. Step 5C: verify sudo and sudo logging
        1. Results of testing step 5C: 2012-05-17
    9. Step 6: verify access to experimental OpenVZ node
      1. Step 6A: verify that SSH to experimental OpenVZ node succeeds and …
        1. Results of testing step 6A: 2012-05-17
      2. Step 6B: verify the absence of common unencrypted login protocols
        1. Results of testing step 6B: 2012-05-17
      3. Step 6C: verify sudo and sudo logging
        1. Results of testing step 6C: 2012-05-17
    10. Step 7: verify access to control network switch
      1. Step 7A: verify SSH access
        1. Results of testing step 7A: 2012-05-17
      2. Step 7B: verify privileged access to the control network switch
        1. Results of testing step 7B: 2012-05-17
      3. Step 7C: verify absence of unencrypted login access
        1. Results of testing step 7C: 2012-05-17
      4. Step 7D: verify serial console access to the device
    11. Step 8: verify access to dataplane switch
      1. Step 8A: verify SSH access
        1. Results of testing step 8A: 2012-05-17
      2. Step 8B: verify privileged access to the dataplane switch
        1. Results of testing step 8B: 2012-05-17
      3. Step 8C: verify absence of unencrypted login access
        1. Results of testing step 8C: 2012-05-17
      4. Step 8D: verify serial console access to the device
    12. Step 9: verify that iLO is not accessible via unencrypted protocols
        1. Results of testing step 9: 2012-05-17

Detailed test plan for IG-ADM-2: Rack Administrator Access Test

This page is GPO's working page for performing IG-ADM-2. It is public for informational purposes, but it is not an official status report. See GENIRacksHome/InstageniRacks/AcceptanceTestStatus for the current status of InstaGENI acceptance tests.

Last substantive edit of this page: 2012-11-12

Page format

  • The status chart summarizes the state of this test
  • The high-level description from test plan contains text copied exactly from the public test plan and acceptance criteria pages.
  • The steps contain things i will actually do/verify:
    • Steps may be composed of related substeps where i find this useful for clarity
    • Each step is identified as either "(prep)" or "(verify)":
      • Prep steps are just things we have to do. They're not tests of the rack, but are prerequisites for subsequent verification steps
      • Verify steps are steps in which we will actually look at rack output and make sure it is as expected. They contain a Using: block, which lists the steps to run the verification, and an Expect: block which lists what outcome is expected for the test to pass.

Status of test

Step State Date completed Open Tickets Closed Tickets/Comments
1A Color(green,Pass)? 2012-05-15 (18) question about root SSH access was resolved satisfactorily with no change to rack
1B Color(green,Pass)? 2012-05-16
1C Color(green,Pass)? 2012-05-16
2A Color(green,Pass)? 2012-05-21 (22) password-based login to ops is now disallowed for all users
2B Color(green,Pass)? 2012-05-16
2C Color(green,Pass)? 2012-05-16
3A Color(orange,Blocked)? 59 blocked on resolution of FOAM remote password login issue
3B Color(green,Pass)? 2012-11-12
3C Color(green,Pass)? 2012-11-12
4A Color(green,Pass)? 2012-11-12
4B Color(green,Pass)? 2012-11-12
4C Color(green,Pass)? 2012-11-12
5A Color(green,Pass)? 2012-05-16
5B Color(green,Pass)? 2012-05-17
5C Color(green,Pass)? 2012-05-17
6A Color(green,Pass)? 2012-05-17 password-based login is allowed, but passwords are pseudorandom and considered sufficiently strong to discourage guessing
6B Color(green,Pass)? 2012-05-17
6C Color(green,Pass)? 2012-05-17
7A Color(#98FB98,Pass: most criteria)? 2012-05-17 unencrypted login is assumed to be okay provided only boss has an interface on the switch control network
7B Color(green,Pass)? 2012-05-17
7C N/A per 7A, device allows telnet from private network, so not testing this step
7D Color(red,Fail)? 2012-05-26 InstaGENI does not plan to provide serial console access to control switch
8A Color(#98FB98,Pass: most criteria)? 2012-05-17 unencrypted login is assumed to be okay provided only boss has an interface on the switch control network
8B Color(green,Pass)? 2012-05-17
8C N/A 2012-05-17 per 8A, device allows telnet from private network, so not testing this step
8D Color(orange,Blocked)? 37 blocked on serial access to dataplane switch
9 Color(green,Pass)? 2012-05-17

High-level description from test plan

This test verifies local and remote administrative access to rack devices.

Procedure

  1. For each type of rack infrastructure node, including the server host itself, the boss VM, the ops VM, the FOAM VM and an experimental host configured for OpenVZ, use a site administrator account to test:
    • Login to the node using public-key SSH.
    • Verify that you cannot login to the node using password-based SSH, nor via any unencrypted login protocol.
    • When logged in, run a command via sudo to verify root privileges.
  2. For each rack infrastructure device (switches, remote PDUs if any), use a site administrator account to test:
    • Login via SSH.
    • Login via a serial console (if the device has one).
    • Verify that you cannot login to the device via an unencrypted login protocol.
    • Use the "enable" command or equivalent to verify privileged access.
  3. Test that iLO (the InstaGENI remote console solution for rack hosts) can be used to access the consoles of the server host and an experimental host:
    • Login via SSH or other encrypted protocol.
    • Verify that you cannot login via an unencrypted login protocol.

Criteria to verify as part of this test

  • V.01. For all rack infrastructure Unix hosts, including rack servers (both physical and VM) and experimental VM servers, site administrators should be able to login at a console (physical or virtual). (C.3.a)
  • V.02. Site administrators can login to all rack infrastructure Unix hosts using public-key SSH. (C.3.a, C.3.b)
  • V.03. Site administrators cannot login to any rack infrastructure Unix hosts using password-based SSH, nor via any unencrypted login protocol. (C.3.a)
  • V.04. Site administrators can run any command with root privileges on all rack infrastructure Unix hosts. (C.3.a)
  • V.05. Site administrators can login to all network-accessible rack infrastructure devices (network devices, remote KVMs, remote PDUs, etc) via serial console and via SSH. (C.3.a, C.3.b)
  • V.06. Site administrators cannot login to any network-accessible rack device via an unencrypted login protocol. (C.3.a)

Step 1: verify access to rack boss node

Step 1A: verify that SSH to boss succeeds and allows public keys only

Note: i am testing this on the Utah rack, so the hostname under test is boss.utah.geniracks.net.

Using:

  • SSH to chaos@boss.instageni.gpolab.bbn.com from outside of the rack using public-key SSH
  • SSH to chaos@boss.instageni.gpolab.bbn.com from outside of the rack using password-based SSH

Verify:

  • Public-key SSH succeeds
  • Password-based SSH does not succeed

Results of testing step 1A: 2012-05-15

  • Public key authentication works:
    $ ssh -o PubkeyAuthentication=yes boss.utah.geniracks.net
    Last login: Tue May 15 09:39:15 2012 from capybara.bbn.co
    Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
            The Regents of the University of California.  All rights reserved.
    
    FreeBSD 8.3-RC1 (XEN) #0: Tue Mar 13 16:27:12 MDT 2012
    
    Welcome to FreeBSD!
    
    To erase a line you've written at the command prompt, use "Ctrl-U".
                    -- Dru <genesis@istar.ca>
    > 
    
  • Password authentication with my emulab password does not work:
    capybara,[~],16:55(0)$ ssh -o PubkeyAuthentication=no boss.utah.geniracks.net
    Password:
    Password:
    Password:
    Permission denied (publickey,keyboard-interactive).
    
    It appears that i have a locked password, and that's why this does the right thing:
    $ sudo grep chaos /etc/master.passwd 
    chaos:*:505:500::0:0:Chaos Golubitsky:/users/chaos:/bin/tcsh
    
  • Hmm, however, root does have a password, and that can be used:
    boss,[~],14:59(0)$ grep "^PermitRootLogin" /etc/ssh/sshd_config 
    PermitRootLogin yes
    
  • I opened instaticket:18, but, after doing so, started wondering whether i'm wrong: /etc/ssh/sshd_config also contains:
    # Change to yes to enable built-in password authentication.
    #PasswordAuthentication no
    

I will let them respond with what they think the situation is, and proceed accordingly.

Step 1B: verify the absence of common unencrypted login protocols

Using:

  • Use netstat to enumerate the network-listening processes running on boss
  • Identify each process and determine whether it is a common unencrypted login protocol
  • For any unencrypted login protocols found to be listening, try to access the relevant port remotely and determine whether login is possible

Verify:

  • No unencrypted login protocols are listening on accessible networks
  • Login does not succeed via any unencrypted login protocol

Results of testing step 1B: 2012-05-16

Note: this isn't a penetration test. I'm just looking for known unencrypted login protocols on public networks. On FreeBSD, sockstat -lL46 shows IPv4 and IPv6 listeners on non-loopback networks.

I found the following listeners, none of which are problematic for our purposes here:

httpd
sshd
inetd (serving the flashpolicy service for Flack)
sslxmlrpc_server.py (emulab)
sdcollectd (emulab)
capserver (emulab)
tmcd (emulab)
bootinfo (emulab)
dhcpd
sendmail
pubsubd (emulab)
mfrisbeed (emulab)
ntpd
mountd
rpcbind
named
syslogd

Step 1C: verify sudo and sudo logging

Using:

  • On boss, run: sudo whoami
  • Look for a syslog file containing a record of the sudo command which was run

Verify:

  • The sudo command should succeed
  • The command which was run should be recorded in a log

Results of testing step 1C: 2012-05-16

  • Command succeeds:
    boss,[~],20:49(0)$ sudo whoami
    root
    
  • Record of command shows up in /var/log/messages:
    boss,[~],20:57(0)$ tail -1 /var/log/messages
    May 16 20:56:31 boss sudo:    chaos : TTY=pts/6 ; PWD=/users/chaos ; USER=root ; COMMAND=/usr/bin/whoami
    

Step 2: verify access to rack ops node

Step 2A: verify that SSH to ops succeeds and allows public keys only

Using:

  • SSH to chaos@ops.instageni.gpolab.bbn.com from outside of the rack using public-key SSH
  • SSH to chaos@ops.instageni.gpolab.bbn.com from outside of the rack using password-based SSH

Verify:

  • Public-key SSH succeeds
  • Password-based SSH does not succeed

Results of testing step 2A: 2012-05-16

  • Public key authentication succeeds:
    capybara,[~],22:58(0)$ ssh -o PubkeyAuthentication=yes ops.utah.geniracks.net
    Last login: Wed May 16 09:23:18 2012 from capybara.bbn.co
    Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
            The Regents of the University of California.  All rights reserved.
    
    FreeBSD 8.3-RC1 (XEN) #0: Tue Mar 13 16:27:12 MDT 2012
    
    Welcome to FreeBSD!
    
    ops,[~],20:58(0)$ 
    
  • Password-based authentication also succeeds:
    capybara,[~],22:59(0)$ ssh -o PubkeyAuthentication=no ops.utah.geniracks.net
    Password:
    Last login: Wed May 16 20:58:43 2012 from capybara.bbn.co
    Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
            The Regents of the University of California.  All rights reserved.
    
    FreeBSD 8.3-RC1 (XEN) #0: Tue Mar 13 16:27:12 MDT 2012
    
    Welcome to FreeBSD!
    
    ops,[~],20:59(0)$ 
    
  • Given that password-based authentication works, i am worried about the setting:
    $ grep "^PermitRootLogin" /etc/ssh/sshd_config
    PermitRootLogin yes
    PermitRootLogin yes
    

Results of testing step 2A: 2012-05-21

  • Public key authentication succeeds:
    capybara,[~],10:01(0)$ ssh -o PubkeyAuthentication=yes ops.utah.geniracks.net
    Last login: Mon May 21 07:55:38 2012 from capybara.bbn.co
    Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
            The Regents of the University of California.  All rights reserved.
    
    FreeBSD 8.3-RC1 (XEN) #0: Tue Mar 13 16:27:12 MDT 2012
    
    Welcome to FreeBSD!
    
    ops,[~],08:01(0)$ 
    
  • Password-based authentication now fails:
    capybara,[~],10:01(0)$ ssh -o PubkeyAuthentication=no ops.utah.geniracks.net
    Permission denied (publickey).
    
  • The root password setting is still effectively "yes", i believe:
    ops,[~],08:02(1)$  grep "^PermitRootLogin" /etc/ssh/sshd_config
    PermitRootLogin without-password
    PermitRootLogin yes
    
    but this doesn't matter since passwords are disallowed entirely.

Step 2B: verify the absence of common unencrypted login protocols

Using:

  • Use netstat to enumerate the network-listening processes running on ops
  • Identify each process and determine whether it is a common unencrypted login protocol
  • For any unencrypted login protocols found to be listening, try to access the relevant port remotely and determine whether login is possible

Verify:

  • No unencrypted login protocols are listening on accessible networks
  • Login does not succeed via any unencrypted login protocol

Results of testing step 2B: 2012-05-16

On FreeBSD, sockstat -lL46 shows IPv4 and IPv6 listeners on non-loopback networks.

I found the following listeners, none of which are problematic for our purposes here:

httpd 
sshd
sendmail
pubsubd (emulab)
ntpd
nfsd
mountd
rpcbind
syslogd
mysqld

Step 2C: verify sudo and sudo logging

Using:

  • On ops, run: sudo whoami
  • Look for a syslog file containing a record of the sudo command which was run

Verify:

  • The sudo command should succeed
  • The command which was run should be recorded in a log

Results of testing step 2C: 2012-05-16

  • Command succeeds:
    ops,[~],21:27(0)$ sudo whoami
    root
    
  • Command is logged:
    $ tail -1 /var/log/messages
    May 16 21:37:55 ops sudo:    chaos : TTY=pts/2 ; PWD=/q/users/chaos ; USER=root ; COMMAND=/usr/bin/whoami
    

Step 3: verify access to rack foam node

Step 3A: verify that SSH to foam succeeds and allows public keys only

Using:

  • SSH to chaos@foam.instageni.gpolab.bbn.com from outside of the rack using public-key SSH
  • SSH to chaos@foam.instageni.gpolab.bbn.com from outside of the rack using password-based SSH

Verify:

  • Public-key SSH succeeds
  • Password-based SSH does not succeed

Results of testing step 3A: 2012-11-12

  • Public-key SSH works:
    $ ssh -o PubkeyAuthentication=yes foam.utah.geniracks.net
    Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-24-generic x86_64)
    
     * Documentation:  https://help.ubuntu.com/
    Last login: Mon Nov 12 10:35:49 2012 from capybara.bbn.com
    
  • Password-based SSH also works, hmm:
    $ ssh -o PubkeyAuthentication=no foam.utah.geniracks.net
    chaos@foam.utah.geniracks.net's password: 
    Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-24-generic x86_64)
    
     * Documentation:  https://help.ubuntu.com/
    Last login: Mon Nov 12 10:36:18 2012 from capybara.bbn.com
    foam,[~],10:37(0)$ 
    

I'll open a ticket about that.

Step 3B: verify the absence of common unencrypted login protocols

Using:

  • Use netstat to enumerate the network-listening processes running on foam
  • Identify each process and determine whether it is a common unencrypted login protocol
  • For any unencrypted login protocols found to be listening, try to access the relevant port remotely and determine whether login is possible

Verify:

  • No unencrypted login protocols are listening on accessible networks
  • Login does not succeed via any unencrypted login protocol

Results of testing step 3B: 2012-11-12

On Ubuntu, netstat -anp | grep LISTEN shows IPv4 and IPv6 listeners.

I found the following listeners, none of which are problematic for our purposes here:

nginx
sshd

Step 3C: verify sudo and sudo logging

Using:

  • On foam, run: sudo whoami
  • Look for a syslog file containing a record of the sudo command which was run

Verify:

  • The sudo command should succeed
  • The command which was run should be recorded in a log

Results of testing step 3C: 2012-11-12

  • Sudo command succeeded:
    foam,[~],10:40(0)$ sudo whoami
    root
    
  • The file /var/log/auth.log contains:
    Nov 12 10:42:59 foam sudo:    chaos : TTY=pts/0 ; PWD=/home/chaos ; USER=root ; COMMAND=/usr/bin/whoami
    

Step 4: verify access to rack FlowVisor node

Step 4A: verify that SSH to FlowVisor succeeds and allows public keys only

Using:

  • SSH to chaos@flowvisor.instageni.gpolab.bbn.com from outside of the rack using public-key SSH
  • SSH to chaos@flowvisor.instageni.gpolab.bbn.com from outside of the rack using password-based SSH

Verify:

  • Public-key SSH succeeds
  • Password-based SSH does not succeed

Results of testing step 4A: 2012-11-12

  • Public-key SSH works:
    $ ssh -o PubkeyAuthentication=yes flowvisor.utah.geniracks.net
    Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-24-generic x86_64)
    
     * Documentation:  https://help.ubuntu.com/
    Last login: Mon Nov 12 10:14:09 2012 from capybara.bbn.com
    
  • Password-based SSH does not work on flowvisor:
    $ ssh -o PubkeyAuthentication=no flowvisor.utah.geniracks.net
    Permission denied (publickey).
    
    • From a login, i can corroborate that the config is right here:
      flowvisor,[~],10:42(0)$ grep PasswordAuthentication /etc/ssh/sshd_config | nocomment 
      PasswordAuthentication no
      

Step 4B: verify the absence of common unencrypted login protocols

Using:

  • Use netstat to enumerate the network-listening processes running on flowvisor
  • Identify each process and determine whether it is a common unencrypted login protocol
  • For any unencrypted login protocols found to be listening, try to access the relevant port remotely and determine whether login is possible

Verify:

  • No unencrypted login protocols are listening on accessible networks
  • Login does not succeed via any unencrypted login protocol

Results of testing step 4B: 2012-11-12

On Ubuntu, netstat -anp | grep LISTEN shows IPv4 and IPv6 listeners.

I found the following listeners, none of which are problematic for our purposes here:

sshd
flowvisor

(Flowvisor allows remote access on port 6633 from switches, and may allow access to port 8080 and/or 8081 from FOAM, but does not allow them from offsite, e.g.

$ telnet flowvisor.utah.geniracks.net 8080
Trying 155.98.34.7...

hangs from capybara.bbn.com, likewise for port 8081.)

Step 4C: verify sudo and sudo logging

Using:

  • On flowvisor, run: sudo whoami
  • Look for a syslog file containing a record of the sudo command which was run

Verify:

  • The sudo command should succeed
  • The command which was run should be recorded in a log

Results of testing step 4C: 2012-11-12

  • Sudo command succeeded:
    flowvisor,[~],10:48(0)$ sudo whoami
    root
    
  • The file /var/log/auth.log contains:
    Nov 12 10:48:21 flowvisor sudo:    chaos : TTY=pts/1 ; PWD=/home/chaos ; USER=root ; COMMAND=/usr/bin/whoami
    

Step 5: verify access to rack infrastructure VM server host

Step 5A: verify that SSH to infrastructure host succeeds and allows public keys only

Using:

  • SSH to chaos account on rack infrastructure server from outside of the rack using public-key SSH
  • SSH to chaos account on rack infrastructure server from outside of the rack using password-based SSH

Verify:

  • Public-key SSH succeeds
  • Password-based SSH does not succeed

Results of testing step 5A: 2012-05-16

Testing with Utah rack, whose control node is utah.control.geniracks.net (which believes its hostname is control.utah.geniracks.net, see instaticket:23).

  • Public key login succeeds:
    capybara,[~],23:55(0)$ ssh -o PubkeyAuthentication=yes utah.control.geniracks.net
    Welcome to Ubuntu precise (development branch) (GNU/Linux 3.2.0-23-generic x86_64)
    
     * Documentation:  https://help.ubuntu.com/
    
      System information as of Wed May 16 21:55:30 MDT 2012
    
      System load:  0.0               Users logged in:       1
      Usage of /:   45.0% of 5.85GB   IP address for xenbr0: 155.98.34.2
      Memory usage: 24%               IP address for xenbr1: 10.1.1.254
      Swap usage:   0%                IP address for xenbr2: 10.2.1.254
      Processes:    150               IP address for xenbr3: 10.3.1.254
    
      Graph this data and manage this system at https://landscape.canonical.com/
    Last login: Wed May 16 21:45:14 2012 from capybara.bbn.com
    control,[~],21:55(0)$ 
    
  • Password-based login should fail because i don't have a password set, and because /etc/ssh/sshd_config contains:
    ChallengeResponseAuthentication no
    PasswordAuthentication no
    
    and it does:
    capybara,[~],23:56(0)$ ssh -o PubkeyAuthentication=no utah.control.geniracks.net 
    Permission denied (publickey).
    

Step 5B: verify the absence of common unencrypted login protocols

Using:

  • Use netstat to enumerate the network-listening processes running on the infrastructure VM server
  • Identify each process and determine whether it is a common unencrypted login protocol
  • For any unencrypted login protocols found to be listening, try to access the relevant port remotely and determine whether login is possible

Verify:

  • No unencrypted login protocols are listening on accessible networks
  • Login does not succeed via any unencrypted login protocol

Results of testing step 5B: 2012-05-17

  • On Ubuntu, get a list of listeners using:
    sudo netstat -anp | grep LISTEN
    
  • This reveals that only sshd is listening for remote connections on non-localhost interfaces.

Step 5C: verify sudo and sudo logging

Using:

  • On the infrastructure VM server, run: sudo whoami
  • Look for a syslog file containing a record of the sudo command which was run

Verify:

  • The sudo command should succeed
  • The command which was run should be recorded in a log

Results of testing step 5C: 2012-05-17

  • The command succeeds:
    control,[~],22:05(0)$ sudo whoami
    root
    
  • A record of sudo is found in /var/log/auth.log:
    control,[~],22:05(0)$ sudo tail /var/log/auth.log
    ...
    May 16 22:05:17 control sudo:    chaos : TTY=pts/5 ; PWD=/home/chaos ; USER=root ; COMMAND=/usr/bin/whoami
    ...
    

Step 6: verify access to experimental OpenVZ node

Step 6A: verify that SSH to experimental OpenVZ node succeeds and allows public keys only on public IPs

Using:

  • SSH to root on the experimental node from boss using public-key SSH
  • View sshd_config file and determine whether password-based login is allowed

Verify:

  • Public-key SSH succeeds from boss
  • Password-based SSH does not succeed from outside of the rack

Results of testing step 6A: 2012-05-17

Testing with Utah rack, using shared OpenVZ host pc5.utah.geniracks.net.

  • SSH to root from boss does work:
    boss,[~],22:12(0)$ sudo ssh pc5
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    @    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
    @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
    IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
    Someone could be eavesdropping on you right now (man-in-the-middle attack)!
    It is also possible that the RSA host key has just been changed.
    The fingerprint for the RSA key sent by the remote host is
    46:63:92:67:c8:75:20:4e:52:9f:2d:f6:cb:58:16:77.
    Please contact your system administrator.
    Add correct host key in /root/.ssh/known_hosts to get rid of this message.
    Offending key in /root/.ssh/known_hosts:18
    Password authentication is disabled to avoid man-in-the-middle attacks.
    Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
    Last login: Wed May 16 22:10:34 2012 from boss.utah.geniracks.net
    [root@vhost1 ~]# 
    
  • Incidentally, SSH as myself also works:
    capybara,[~/src/git/tango-monitor],00:11(0)$ ssh pc5.utah.geniracks.net
    The authenticity of host 'pc5.utah.geniracks.net (155.98.34.15)' can't be established.
    RSA key fingerprint is 46:63:92:67:c8:75:20:4e:52:9f:2d:f6:cb:58:16:77.
    Are you sure you want to continue connecting (yes/no)? yes
    Warning: Permanently added 'pc5.utah.geniracks.net,155.98.34.15' (RSA) to the list of known hosts.
    Last login: Wed May 16 22:11:24 2012 from boss.utah.geniracks.net
    vhost1,[~],22:11(0)$
    
  • Hmm, password-based SSH as root (using the password listed for the node within the Emulab admin UI) also works:
    capybara,[~],00:15(0)$ ssh root@pc5.utah.geniracks.net
    root@pc5.utah.geniracks.net's password: 
    Last login: Wed May 16 22:12:21 2012 from boss.utah.geniracks.net
    [root@vhost1 ~]# 
    
    However, given the size of the pseudorandom passwords used for Emulab experimental hosts, i think it's fine to assert that root password guessing is not a realistic threat, so i am not opening a ticket.

Step 6B: verify the absence of common unencrypted login protocols

Using:

  • Use netstat to enumerate the network-listening processes running on the OpenVZ node
  • Identify each process and determine whether it is a common unencrypted login protocol
  • For any unencrypted login protocols found to be listening, try to access the relevant port remotely and determine whether login is possible

Verify:

  • No unencrypted login protocols are listening on accessible networks
  • Login does not succeed via any unencrypted login protocol

Results of testing step 6B: 2012-05-17

  • On RHEL-like OSes, get a list of listeners using:
    sudo netstat -anp | grep LISTEN
    
  • This reveals that the following processes, which are not concerning, are listening for remote connections:
    sshd
    rpc.statd
    rpcbind
    emulab-syncd (emulab)
    pubsubd (emulab)
    
  • There is also an unidentifiable process listening on this port:
    tcp        0      0 0.0.0.0:58441               0.0.0.0:*                   LISTEN      -                   
    
    I was unable to turn up any further information about it from lsof, so i tried connecting to it (perhaps unreasonably). I got:
    $ telnet pc5.utah.geniracks.net 58441
    Trying 155.98.34.15...
    Connected to pc5.utah.geniracks.net.
    Escape character is '^]'.
    HELP
    Connection closed by foreign host.
    
    and in the logs:
    May 16 22:39:28 localhost kernel: [27502.595007] RPC: multiple fragments per record not supported
    
    So i think this is an RPC-based thing, and that this kind of testing is not very useful.

Step 6C: verify sudo and sudo logging

Using:

  • On the OpenVZ node, run: sudo whoami
  • Look for a syslog file containing a record of the sudo command which was run

Verify:

  • The sudo command should succeed
  • The command which was run should be recorded in a log

Results of testing step 6C: 2012-05-17

  • The command succeeds and the log entry is created:
    vhost1,[/var/log],22:40(0)$ sudo whoami
    root
    
    vhost1,[/var/log],22:43(0)$ sudo tail /var/log/secure 
    ...
    May 16 22:43:48 localhost sudo:    chaos : TTY=pts/2 ; PWD=/var/log ; USER=root ; COMMAND=/usr/bin/whoami
    ...
    

Step 7: verify access to control network switch

Step 7A: verify SSH access

Using:

  • Login to the control network switch via SSH

Verify:

  • SSH login succeeds

Results of testing step 7A: 2012-05-17

Tested using Utah rack.

  • SSH login does not succeed:
    boss,[~],22:46(0)$ ssh procurve1
    ssh: connect to host procurve1 port 22: Connection refused
    
  • Telnet succeeds instead, using the password in /usr/testbed/etc/switch.pswd:
    boss,[~],22:47(0)$ telnet procurve1
    Trying 10.1.1.253...
    Connected to procurve1.
    Escape character is '^]'.
    ...
    Password: 
    ...
    ProCurve Switch 2610-24#  
    

This is presumed to be acceptable as long as only server nodes (such as boss) have interfaces on 10.1.1.0/24.

Step 7B: verify privileged access to the control network switch

Using:

  • On the control network switch, run the enable command to enter privileged mode
  • On the control network switch, view the running configuration
  • On the control network switch, view the MAC address table

Verify:

  • Entering privileged mode should succeed
  • Viewing the running configuration should succeed
  • Viewing the MAC address table should succeed

Results of testing step 7B: 2012-05-17

  • Enable command isn't needed, since login is privileged already
  • Running config is viewable:
    ProCurve Switch 2610-24# show running-config 
    
    Running configuration:
    
    ; J9085A Configuration Editor; Created on release #R.11.70
    ...
    
  • MAC address table is viewable:
    ProCurve Switch 2610-24# show mac-address 
    
     Status and Counters - Port Address Table
    ...
    

Step 7C: verify absence of unencrypted login access

Using:

  • From boss, attempt to connect via telnet (port 23) to the control network switch
  • From boss, attempt to connect via http (port 80) to the control network switch
  • If a port 80 connection is successful, determine whether login is allowed via that interface
  • Look at the control switch running configuration, and identify any configuration lines which might represent login services
  • Determine whether any such services are encrypted or disabled

Verify:

  • The device does not allow telnet access on the standard port
  • The device does not allow unencrypted http authentication
  • No other services appear to allow remote unencrypted authentication

Results of testing step 7C: 2012-05-17

Not applicable: given results of 7A, not bothering to test for other instances of unencrypted login.

Step 7D: verify serial console access to the device

Using:

  • Access the control network switch via the serial console
  • Authenticate to the control network switch if necessary
  • Enter enable mode if necessary
  • View the running configuration of the control network switch

Verify:

  • The control network switch should be accessible via serial console
  • It should be possible to perform privileged operations via the serial console
  • It should be possible to view the running configuration via the serial console

Step 8: verify access to dataplane switch

Step 8A: verify SSH access

Using:

  • Login to the dataplane switch via SSH

Verify:

  • SSH login succeeds

Results of testing step 8A: 2012-05-17

Tested using Utah rack.

  • SSH login succeeds using the password in /etc/testbed/etc/switch.pswd:
    boss,[~],22:56(0)$ ssh elabman@procurve2
    We'd like to keep you up to date about:
      * Software feature updates
      * New product announcements
      * Special events
    
    Please register your products now at:  www.ProCurve.com
    
    elabman@procurve2's password: 
    ...
    ProCurve Switch 6600ml-48G-4XG#  
    
  • Telnet succeeds as well:
    boss,[~],22:57(0)$ telnet procurve2
    Trying 10.2.1.253...
    Connected to procurve2.
    Escape character is '^]'.
    ...
    Password: 
    ...
    ProCurve Switch 6600ml-48G-4XG# 
    

This is presumed to be acceptable as long as only server nodes (such as boss) have interfaces on 10.2.1.0/24.

Step 8B: verify privileged access to the dataplane switch

Using:

  • On the dataplane switch, run the enable command to enter privileged mode
  • On the dataplane switch, view the running configuration
  • On the dataplane switch, view the MAC address table

Verify:

  • Entering privileged mode should succeed
  • Viewing the running configuration should succeed
  • Viewing the MAC address table should succeed

Results of testing step 8B: 2012-05-17

  • Enable command isn't needed, since login is privileged already
  • Running config is viewable:
    ProCurve Switch 6600ml-48G-4XG# show running-config
    
    Running configuration:
    
    ; J9452A Configuration Editor; Created on release #K.14.41
    ...
    
  • MAC address table is viewable:
    ProCurve Switch 6600ml-48G-4XG# show mac-address 
    
     Status and Counters - Port Address Table
    ...
    

Step 8C: verify absence of unencrypted login access

Using:

  • From boss, attempt to connect via telnet (port 23) to the dataplane switch
  • From boss, attempt to connect via http (port 80) to the dataplane switch
  • If a port 80 connection is successful, determine whether login is allowed via that interface
  • Look at the dataplane switch running configuration, and identify any configuration lines which might represent login services
  • Determine whether any such services are encrypted or disabled

Verify:

  • The device does not allow telnet access on the standard port
  • The device does not allow unencrypted http authentication
  • No other services appear to allow remote unencrypted authentication

Results of testing step 8C: 2012-05-17

Not applicable: given results of 8A, not bothering to test for other instances of unencrypted login.

Step 8D: verify serial console access to the device

Using:

  • Access the dataplane switch via the serial console
  • Authenticate to the dataplane switch if necessary
  • Enter enable mode if necessary
  • View the running configuration of the dataplane switch

Verify:

  • The dataplane switch should be accessible via serial console
  • It should be possible to perform privileged operations via the serial console
  • It should be possible to view the running configuration via the serial console

Step 9: verify that iLO is not accessible via unencrypted protocols

Using:

  • From boss, attempt to connect via telnet (port 23) to the pc1 iLO IP
  • From boss, attempt to connect via http (port 80) to the the pc1 iLO IP
  • If a port 80 connection is successful, determine whether login is allowed via that interface
  • If any prospective unencrypted protocols were identified in the iLO configurations during IG-ADM-1 step 3G, attempt to connect to those ports from boss

Verify:

  • The pc1 iLO does not allow telnet access on the standard port
  • The device does not allow unencrypted http authentication
  • No other services appear to allow remote unencrypted authentication

Results of testing step 9: 2012-05-17

  • The pc1.utah.geniracks.net iLO is at 155.98.34.103
  • Telnet fails:
    boss,[~],23:06(0)$ telnet 155.98.34.103
    Trying 155.98.34.103...
    telnet: connect to address 155.98.34.103: Connection refused
    telnet: Unable to connect to remote host
    
  • Unencrypted telnet to port 80 succeeds, but closes the connection immediately:
    boss,[~],23:08(1)$ telnet 155.98.34.103 80
    Trying 155.98.34.103...
    Connected to 155.98.34.103.
    Escape character is '^]'.
    Connection closed by foreign host.
    
    In a web browser, this connection is redirected from port 80 to port 443
  • According to Administration -> Access Settings in iLO, the only additional ports this thing uses are remote console, virtual media, and IPMI/DCMI, all expected iLO functions which i don't need to test this way.