wiki:GENIRacksHome/ExogeniRacks/AcceptanceTestStatus/EG-ADM-2

Version 14 (modified by chaos@bbn.com, 7 years ago) (diff)

--

Detailed test plan for EG-ADM-2: Rack Administrator Access Test

This page is GPO's working page for performing EG-ADM-2. It is public for informational purposes, but it is not an official status report. See GENIRacksHome/ExogeniRacks/AcceptanceTestStatus for the current status of ExoGENI acceptance tests.

Last substantive edit of this page: 2012-05-27

Page format

  • The status chart summarizes the state of this test
  • The high-level description from test plan contains text copied exactly from the public test plan and acceptance criteria pages.
  • The steps contain things i will actually do/verify:
    • Steps may be composed of related substeps where i find this useful for clarity
    • Each step is identified as either "(prep)" or "(verify)":
      • Prep steps are just things we have to do. They're not tests of the rack, but are prerequisites for subsequent verification steps
      • Verify steps are steps in which we will actually look at rack output and make sure it is as expected. They contain a Using: block, which lists the steps to run the verification, and an Expect: block which lists what outcome is expected for the test to pass.

Status of test

Step State Date completed Open Tickets Closed Tickets/Comments
1A Color(orange,Blocked)? 34 blocked on information about bbn-hn configuration
1B Color(green,Pass)? 2012-05-27
1C Color(green,Pass)? 2012-05-27
2A Color(yellow,Complete)? retest when experiments are known to be running on the worker
2B Color(orange,Blocked)? blocked on retest of 2A; this is n/a if no public IPs
2C Color(green,Pass)? 2012-05-27
3A Color(green,Pass)? 2012-05-27 (10) ready to test
3B Color(light-green,Pass: most criteria)? 2012-05-27 enable mode on switch not available to site admins, but available information appears sufficient
3C ready to test
3D Color(orange,Blocked)? 19 (10) blocked on serial access to switches
4A (10) ready to test
4B ready to test
4C ready to test
4D Color(orange,Blocked)? 19 (10) blocked on serial access to switches
5

High-level description from test plan

This test verifies local and remote administrative access to rack devices.

Procedure

  1. For each type of rack infrastructure node, including the head node and a worker node configured for OpenStack, use a site administrator account to test:
    • Login to the node using public-key SSH.
    • Verify that you cannot login to the node using password-based SSH, nor via any unencrypted login protocol.
    • When logged in, run a command via sudo to verify root privileges.
  2. For each rack infrastructure device (switches, remote PDUs if any), use a site administrator account to test:
    • Login via SSH.
    • Login via a serial console (if the device has one).
    • Verify that you cannot login to the device via an unencrypted login protocol.
    • Use the "enable" command or equivalent to verify privileged access.
  3. Test that IMM (the ExoGENI remote console solution for rack hosts) can be used to access the consoles of the head node and a worker node:
    • Login via SSH or other encrypted protocol.
    • Verify that you cannot login via an unencrypted login protocol.

Criteria to verify as part of this test

  • V.01. For all rack infrastructure Unix hosts, including rack servers (both physical and VM) and experimental VM servers, site administrators should be able to login at a console (physical or virtual). (C.3.a)
  • V.02. Site administrators can login to all rack infrastructure Unix hosts using public-key SSH. (C.3.a, C.3.b)
  • V.03. Site administrators cannot login to any rack infrastructure Unix hosts using password-based SSH, nor via any unencrypted login protocol. (C.3.a)
  • V.04. Site administrators can run any command with root privileges on all rack infrastructure Unix hosts. (C.3.a)
  • V.05. Site administrators can login to all network-accessible rack infrastructure devices (network devices, remote KVMs, remote PDUs, etc) via serial console and via SSH. (C.3.a, C.3.b)
  • V.06. Site administrators cannot login to any network-accessible rack device via an unencrypted login protocol. (C.3.a)

Step 1: verify access to rack head node

Step 1A: verify that SSH to bbn-hn succeeds and allows public keys only

Using:

  • SSH to chaos@bbn-hn.exogeni.gpolab.bbn.com from outside of the rack using public-key SSH
  • SSH to chaos@bbn-hn.exogeni.gpolab.bbn.com from outside of the rack using password-based SSH

Verify:

  • Public-key SSH succeeds
  • Password-based SSH does not succeed

Results of testing step 1A: 2012-05-27

  • From the BBN internal network (128.89.68.0/23), with pubkey:
    $ ssh -o PubkeyAuthentication=yes chaos@bbn-hn.exogeni.gpolab.bbn.com
    Last login: Sat May 26 14:41:55 2012 from capybara.bbn.com
    ...
    bbn-hn,[~],20:25(0)$ 
    
  • With password:
    $ ssh -o PubkeyAuthentication=no chaos@bbn-hn.exogeni.gpolab.bbn.com
    chaos@bbn-hn.exogeni.gpolab.bbn.com's password: 
    Last login: Sun May 27 20:25:48 2012 from capybara.bbn.com
    ...
    bbn-hn,[~],20:27(0)$ 
    

I opened exoticket:34 to request explanation of the protection mechanisms on bbn-hn, so i can redo this test and verify that they work and seem likely to be sufficient.

Step 1B: verify the absence of common unencrypted login protocols

Using:

  • Use netstat to enumerate the network-listening processes running on bbn-hn
  • Identify each process and determine whether it is a common unencrypted login protocol
  • For any unencrypted login protocols found to be listening, try to access the relevant port remotely and determine whether login is possible

Verify:

  • No unencrypted login protocols are listening on accessible networks
  • Login does not succeed via any unencrypted login protocol

Results of testing step 1B: 2012-05-27

  • First ran sudo netstat -anp | grep LISTEN, and got rid of:
    • things bound to 10.100.x.x interfaces
    • things bound to 127.0.0.1
  • That left too many things to easily look at, so i also looked at the incoming-to-host firewall rules:
    sudo iptables -L -v -n
    
    • Ignore things with source addresses in private or exogeni-server-old spaces
    • Ignore things which reject incoming connections on the public interface before allowing them on private interfaces
  • Finally, looking at what's left from all that, i identify the following listening programs:
    /usr/sbin/sshd
    /usr/sbin/httpd
    /usr/sbin/nginx
    /usr/java/latest/bin/java (tomcat, in particular `/opt/orca*/tomcat/`)
    

Since httpd redirects to the SSL port and nginx is only serving the foam site, i don't see any obvious unencrypted login options.

Step 1C: verify sudo and sudo logging

Using:

  • On bbn-hn, run: sudo whoami
  • Look for a syslog file containing a record of the sudo command which was run

Verify:

  • The sudo command should succeed
  • The command which was run should be recorded in a log

Results of testing step 1C: 2012-05-27

  • Running as xoadmin chaos:
    bbn-hn,[~],21:11(0)$ sudo whoami
    [sudo] password for chaos: 
    root
    
  • Running as bbnadmin cgolubit:
    (cgolubit) bbn-hn,[~],21:11(0)$ sudo whoami
    
    We trust you have received the usual lecture from the local System
    Administrator. It usually boils down to these three things:
    
        #1) Respect the privacy of others.
        #2) Think before you type.
        #3) With great power comes great responsibility.
    
    [sudo] password for cgolubit: 
    root
    
  • Looking in the log:
    bbn-hn,[/var/log],21:12(0)$ sudo grep sudo: /var/log/secure
    ...
    May 27 21:11:06 bbn-hn sudo:    chaos : TTY=pts/0 ; PWD=/home/chaos ; USER=root ; COMMAND=/usr/bin/whoami
    ...
    May 27 21:11:41 bbn-hn sudo: cgolubit : TTY=pts/1 ; PWD=/home/cgolubit ; USER=root ; COMMAND=/usr/bin/whoami
    

Step 2: verify access to rack worker node

Step 2A: verify that SSH to bbn-w1 succeeds and allows public keys only on public IPs

Using:

  • SSH to chaos@bbn-w1 from bbn-hn using public-key SSH
  • Determine whether any routable IP addresses are defined on bbn-w1 at the time of testing
  • If so, SSH to chaos@bbn-w1 using password-based SSH from outside of the rack

Verify:

  • Public-key SSH succeeds from bbn-hn
  • Password-based SSH does not succeed from outside of the rack

Results of testing step 2A: 2012-05-27

  • SSH from bbn-hn to chaos@bbn-w1 using public-key:
    bbn-hn,[~],21:28(0)$ ssh bbn-w1
    Last login: Sun May 27 21:21:44 2012 from bbn-hn.local
    ...
    bbn-w1,[~],21:28(0)$ 
    
  • That are no public IPs at this time:
    bbn-w1,[~],21:28(0)$ ifconfig | grep "inet addr"
              inet addr:10.100.0.11  Bcast:10.100.0.255  Mask:255.255.255.0
              inet addr:127.0.0.1  Mask:255.0.0.0
    bbn-w1,[~],21:29(0)$ 
    
  • Also verifying that cgolubit can SSH:
    (cgolubit) bbn-hn,[~],21:31(0)$ ssh bbn-w1
    cgolubit@bbn-w1's password: 
    Last login: Sun May 27 21:30:03 2012 from bbn-hn.local
    ...
    (cgolubit) bbn-w1,[~],21:31(0)$ 
    

I'll want to recheck this when i know some experiments are running on a worker node, to verify that there are no public IPs bound to the node itself in that situation.

Step 2B: verify the absence of common unencrypted login protocols

Using:

  • Use netstat to enumerate the network-listening processes running on bbn-w1
  • Identify each process and determine whether it is a common unencrypted login protocol
  • For any unencrypted login protocols found to be listening, try to access the relevant port remotely and determine whether login is possible

Verify:

  • No unencrypted login protocols are listening on accessible networks
  • Login does not succeed via any unencrypted login protocol

Step 2C: verify sudo and sudo logging

Using:

  • On bbn-w1, run: sudo whoami
  • Look for a syslog file containing a record of the sudo command which was run

Verify:

  • The sudo command should succeed
  • The command which was run should be recorded in a log

Results of testing step 2C: 2012-05-27

  • Tested as site admin cgolubit:
    (cgolubit) bbn-w1,[~],21:31(0)$ sudo whoami
    
    We trust you have received the usual lecture from the local System
    Administrator. It usually boils down to these three things:
    
        #1) Respect the privacy of others.
        #2) Think before you type.
        #3) With great power comes great responsibility.
    
    [sudo] password for cgolubit: 
    root
    
  • Look in the logs:
    $ sudo grep sudo: /var/log/secure
    ...
    May 27 21:36:42 bbn-w1 sudo: cgolubit : TTY=pts/1 ; PWD=/home/cgolubit ; USER=root ; COMMAND=/usr/bin/whoami
    

Step 3: verify access to 8052 (management) switch

Step 3A: verify SSH access

Using:

  • From bbn-hn, login to chaos@8052 via SSH

Verify:

  • SSH login succeeds

Results of testing step 3A: 2012-05-27

Notes:

  • Actually testing as cgolubit for now
  • Actually testing from BBN internal network on the public IP (192.1.242.4)

Login succeeds:

$ ssh cgolubit@192.1.242.4
Enter radius password: 

IBM Networking Operating System RackSwitch G8052.

8052.bbn.xo>

Step 3B: verify privileged access to the 8052 switch

Using:

  • On the 8052, run the enable command to enter privileged mode
  • On the 8052, view the running configuration
  • On the 8052, view the MAC address table

Verify:

  • Entering privileged mode should succeed
  • Viewing the running configuration should succeed
  • Viewing the MAC address table should succeed

Results of testing step 3B: 2012-05-27

As determined in EG-ADM-1:

  • Enable mode does not succeed:
    8052.bbn.xo>en
    
    Enable access using (oper) credentials restricted to admin accounts only.
    
  • Viewing the running-config does not succeed:
    8052.bbn.xo>show running-config
                      ^
    % Invalid input detected at '^' marker.
    
  • However viewing the mac address table does succeed:
    8052.bbn.xo>show mac-address-table 
    Mac address Aging Time: 300 
    
    Total number of FDB entries : 23
    ...
    
  • And viewing the interface information does succeed:
    8052.bbn.xo>show interface status 
    ------------------------------------------------------------------
    Alias   Port   Speed    Duplex     Flow Ctrl      Link
    ------- ----   -----   --------  --TX-----RX--   ------
    ...
    

Step 3C: verify absence of unencrypted login access

Using:

  • From bbn-hn, attempt to connect via telnet (port 23) to the 8052
  • From bbn-hn, attempt to connect via http (port 80) to the 8052
  • If a port 80 connection is successful, determine whether login is allowed via that interface
  • Look at the 8052 running configuration, and identify any configuration lines which might represent login services
  • Determine whether any such services are encrypted or disabled

Verify:

  • The device does not allow telnet access on the standard port
  • The device does not allow unencrypted http authentication
  • No other services appear to allow remote unencrypted authentication

Step 3D: verify serial console access to the device

Using:

  • From bbn-hn, access the 8264 via the serial console
  • Authenticate to the 8264 if necessary
  • Enter enable mode if necessary
  • View the running configuration of the 8264

Verify:

  • The 8264 should be accessible via serial console
  • It should be possible to perform privileged operations via the serial console
  • It should be possible to view the running configuration via the serial console

Step 4: verify access to 8264 (dataplane) switch

Step 4A: verify SSH access

Using:

  • From bbn-hn, login to chaos@8264 via SSH

Verify:

  • SSH login succeeds

Step 4B: verify privileged access to the 8264 switch

Using:

  • On the 8264, run the enable command to enter privileged mode
  • On the 8264, view the running configuration
  • On the 8264, view the MAC address table

Verify:

  • Entering privileged mode should succeed
  • Viewing the running configuration should succeed
  • Viewing the MAC address table should succeed

Step 4C: verify absence of unencrypted login access

Using:

  • From bbn-hn, attempt to connect via telnet (port 23) to the 8264
  • From bbn-hn, attempt to connect via http (port 80) to the 8264
  • If a port 80 connection is successful, determine whether login is allowed via that interface
  • Look at the 8264 running configuration, and identify any configuration lines which might represent login services
  • Determine whether any such services are encrypted or disabled

Verify:

  • The device does not allow telnet access on the standard port
  • The device does not allow unencrypted http authentication
  • No other services appear to allow remote unencrypted authentication

Step 4D: verify serial console access to the device

Using:

  • From bbn-hn, access the 8264 via the serial console
  • Authenticate to the 8264 if necessary
  • Enter enable mode if necessary
  • View the running configuration of the 8264

Verify:

  • The 8264 should be accessible via serial console
  • It should be possible to perform privileged operations via the serial console
  • It should be possible to view the running configuration via the serial console

Step 5: verify that IMM/IPMI is not accessible without the VPN

Using:

  • From bbn-hn, attempt to connect via telnet (port 23) to the bbn-w1 IMM IP
  • From bbn-hn, attempt to connect via http (port 80) to the the bbn-w1 IMM IP
  • If a port 80 connection is successful, determine whether login is allowed via that interface
  • If any prospective unencrypted protocols were identified in the IMM configurations during EG-ADM-1 step 3D, attempt to connect to those ports from bbn-hn

Verify:

  • The bbn-w1 IMM does not allow telnet access on the standard port
  • The device does not allow unencrypted http authentication
  • No other services appear to allow remote unencrypted authentication