Changes between Version 7 and Version 8 of GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-ADM-2


Ignore:
Timestamp:
05/18/12 05:22:07 (12 years ago)
Author:
chaos@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-ADM-2

    v7 v8  
    55''This page is GPO's working page for performing IG-ADM-2.  It is public for informational purposes, but it is not an official status report.  See [wiki:GENIRacksHome/InstageniRacks/AcceptanceTestStatus] for the current status of InstaGENI acceptance tests.''
    66
    7 ''Last substantive edit of this page: 2012-05-16''
     7''Last substantive edit of this page: 2012-05-17''
    88
    99== Page format ==
    … …  
    1919== Status of test ==
    2020
    21 || '''Step''' || '''State'''               || '''Date completed''' || '''Tickets'''  || '''Comments'''                                   ||
    22 || 1A         || [[Color(green,Pass)]] ||                      || instaticket:18 || question about root SSH access was resolved satisfactorily with no change to rack ||
    23 || 1B         || [[Color(green,Pass)]] ||                      ||                || ||
    24 || 1C         ||                           ||                      ||                || ready to test                                    ||
    25 || 2A         ||                           ||                      ||                || ready to test                                    ||
    26 || 2B         ||                           ||                      ||                || ready to test                                    ||
    27 || 2C         ||                           ||                      ||                || ready to test                                    ||
    28 || 3A         || [[Color(orange,Blocked)]] ||                      ||                || blocked on access to FOAM VM                     ||
    29 || 3B         || [[Color(orange,Blocked)]] ||                      ||                || blocked on 3A                                    ||
    30 || 3C         || [[Color(orange,Blocked)]] ||                      ||                || blocked on 3A                                    ||
    31 || 4A         || [[Color(orange,Blocked)]] ||                      ||                || blocked on access to FlowVisor VM                ||
    32 || 4B         || [[Color(orange,Blocked)]] ||                      ||                || blocked on 4A                                    ||
    33 || 4C         || [[Color(orange,Blocked)]] ||                      ||                || blocked on 4A                                    ||
    34 || 5A         ||                           ||                      ||                || ready to test                                    ||
    35 || 5B         || [[Color(orange,Blocked)]] ||                      ||                || blocked on 5A                                    ||
    36 || 5C         || [[Color(orange,Blocked)]] ||                      ||                || blocked on 5A                                    ||
    37 || 6A         || [[Color(orange,Blocked)]] ||                      ||                || blocked on allocation of OpenVZ node            ||
    38 || 6B         || [[Color(orange,Blocked)]] ||                      ||                || blocked on 6A                                    ||
    39 || 6C         || [[Color(orange,Blocked)]] ||                      ||                || blocked on 6A                                    ||
    40 || 7A         ||                           ||                      ||                || ready to test                                    ||
    41 || 7B         ||                           ||                      ||                || ready to test                                    ||
    42 || 7C         ||                           ||                      ||                || ready to test                                    ||
    43 || 7D         || [[Color(orange,Blocked)]] ||                      ||                || blocked on serial access to switches             ||
    44 || 8A         || [[Color(orange,Blocked)]] ||                      ||                || blocked on access to dataplane switch            ||
    45 || 8B         || [[Color(orange,Blocked)]] ||                      ||                || blocked on 8A                                    ||
    46 || 8C         || [[Color(orange,Blocked)]] ||                      ||                || blocked on 8A                                    ||
    47 || 8D         || [[Color(orange,Blocked)]] ||                      ||                || blocked on serial access to switches             ||
    48 || 9          || [[Color(orange,Blocked)]] ||                      ||                || blocked on access to rack iLO                    ||
     21|| '''Step''' || '''State'''                            || '''Date completed''' || '''Tickets'''  || '''Comments'''                                                                                                            ||
     22|| 1A         || [[Color(green,Pass)]]                  ||                      || instaticket:18 || question about root SSH access was resolved satisfactorily with no change to rack                                        ||
     23|| 1B         || [[Color(green,Pass)]]                  ||                      ||                ||                                                                                                                          ||
     24|| 1C         || [[Color(green,Pass)]]                  ||                      ||                || ready to test                                                                                                             ||
     25|| 2A         || [[Color(orange,Blocked)]]              ||                      || instaticket:22 || resolve question about password-based login to ops                                                                        ||
     26|| 2B         || [[Color(green,Pass)]]                  ||                      ||                || ready to test                                                                                                             ||
     27|| 2C         || [[Color(green,Pass)]]                  ||                      ||                || ready to test                                                                                                             ||
     28|| 3A         || [[Color(orange,Blocked)]]              ||                      ||                || blocked on access to FOAM VM                                                                                              ||
     29|| 3B         || [[Color(orange,Blocked)]]              ||                      ||                || blocked on 3A                                                                                                             ||
     30|| 3C         || [[Color(orange,Blocked)]]              ||                      ||                || blocked on 3A                                                                                                             ||
     31|| 4A         || [[Color(orange,Blocked)]]              ||                      ||                || blocked on access to FlowVisor VM                                                                                         ||
     32|| 4B         || [[Color(orange,Blocked)]]              ||                      ||                || blocked on 4A                                                                                                             ||
     33|| 4C         || [[Color(orange,Blocked)]]              ||                      ||                || blocked on 4A                                                                                                             ||
     34|| 5A         || [[Color(green,Pass)]]                  ||                      ||                ||                                                                                                                           ||
     35|| 5B         || [[Color(green,Pass)]]                  ||                      ||                ||                                                                                                                           ||
     36|| 5C         || [[Color(green,Pass)]]                  ||                      ||                ||                                                                                                                           ||
     37|| 6A         || [[Color(green,Pass)]]                  ||                      ||                || password-based login is allowed, but passwords are pseudorandom and considered sufficiently strong to discourage guessing ||
     38|| 6B         || [[Color(green,Pass)]]                  ||                      ||                ||                                                                                                                           ||
     39|| 6C         || [[Color(green,Pass)]]                  ||                      ||                ||                                                                                                                           ||
     40|| 7A         || [[Color(#98FB98,Pass: most criteria)]] ||                      ||                || unencrypted login is assumed to be okay provided only boss has an interface on the switch control network                 ||
     41|| 7B         || [[Color(green,Pass)]]                  ||                      ||                ||                                                                                                                           ||
     42|| 7C         || N/A                                    ||                      ||                || per 7A, device allows telnet from private network, so not testing this step                                               ||
     43|| 7D         || [[Color(orange,Blocked)]]              ||                      ||                || blocked on serial access to switches                                                                                      ||
     44|| 8A         || [[Color(#98FB98,Pass: most criteria)]] ||                      ||                || unencrypted login is assumed to be okay provided only boss has an interface on the switch control network                 ||
     45|| 8B         || [[Color(green,Pass)]]                  ||                      ||                ||                                                                                                                           ||
     46|| 8C         || N/A                                    ||                      ||                || per 8A, device allows telnet from private network, so not testing this step                                               ||
     47|| 8D         || [[Color(orange,Blocked)]]              ||                      ||                || blocked on serial access to switches                                                                                      ||
     48|| 9          || [[Color(green,Pass)]]                  ||                      ||                ||                                                                                                                           ||
    4949
    5050== High-level description from test plan ==
    … …  
    179179 * The command which was run should be recorded in a log
    180180
     181==== Results of testing: 2012-05-16 ====
     182
     183 * Command succeeds:
     184{{{
     185boss,[~],20:49(0)$ sudo whoami
     186root
     187}}}
     188 * Record of command shows up in `/var/log/messages`:
     189{{{
     190boss,[~],20:57(0)$ tail -1 /var/log/messages
     191May 16 20:56:31 boss sudo:    chaos : TTY=pts/6 ; PWD=/users/chaos ; USER=root ; COMMAND=/usr/bin/whoami
     192}}}
     193
    181194== Step 2: verify access to rack ops node ==
    182195
    … …  
    190203 * Public-key SSH succeeds
    191204 * Password-based SSH does not succeed
     205
     206==== Results of testing: 2012-05-16 ====
     207
     208 * Public key authentication succeeds:
     209{{{
     210capybara,[~],22:58(0)$ ssh -o PubkeyAuthentication=yes ops.utah.geniracks.net
     211Last login: Wed May 16 09:23:18 2012 from capybara.bbn.co
     212Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
     213        The Regents of the University of California.  All rights reserved.
     214
     215FreeBSD 8.3-RC1 (XEN) #0: Tue Mar 13 16:27:12 MDT 2012
     216
     217Welcome to FreeBSD!
     218
     219ops,[~],20:58(0)$
     220}}}
     221 * Password-based authentication also succeeds:
     222{{{
     223capybara,[~],22:59(0)$ ssh -o PubkeyAuthentication=no ops.utah.geniracks.net
     224Password:
     225Last login: Wed May 16 20:58:43 2012 from capybara.bbn.co
     226Copyright (c) 1980, 1983, 1986, 1988, 1990, 1991, 1993, 1994
     227        The Regents of the University of California.  All rights reserved.
     228
     229FreeBSD 8.3-RC1 (XEN) #0: Tue Mar 13 16:27:12 MDT 2012
     230
     231Welcome to FreeBSD!
     232
     233ops,[~],20:59(0)$
     234}}}
     235 * Given that password-based authentication works, i am worried about the setting:
     236{{{
     237$ grep "^PermitRootLogin" /etc/ssh/sshd_config
     238PermitRootLogin yes
     239PermitRootLogin yes
     240}}}
    192241
    193242=== Step 2B: verify the absence of common unencrypted login protocols ===
    … …  
    202251 * Login does not succeed via any unencrypted login protocol
    203252
     253==== Results of testing: 2012-05-16 ====
     254
     255On FreeBSD, `sockstat -lL46` shows IPv4 and IPv6 listeners on non-loopback networks.
     256
     257I found the following listeners, none of which are problematic for our purposes here:
     258{{{
     259httpd
     260sshd
     261sendmail
     262pubsubd (emulab)
     263ntpd
     264nfsd
     265mountd
     266rpcbind
     267syslogd
     268mysqld
     269}}}
     270
    204271=== Step 2C: verify sudo and sudo logging ===
    205272
    … …  
    211278 * The sudo command should succeed
    212279 * The command which was run should be recorded in a log
     280
     281==== Results of testing: 2012-05-16 ====
     282
     283 * Command succeeds:
     284{{{
     285ops,[~],21:27(0)$ sudo whoami
     286root
     287}}}
     288 * Command is logged:
     289{{{
     290$ tail -1 /var/log/messages
     291May 16 21:37:55 ops sudo:    chaos : TTY=pts/2 ; PWD=/q/users/chaos ; USER=root ; COMMAND=/usr/bin/whoami
     292}}}
    213293
    214294== Step 3: verify access to rack foam node ==
    … …  
    280360== Step 5: verify access to rack infrastructure VM server host ==
    281361
    282 === Step 5A: verify that SSH to foam succeeds and allows public keys only ===
     362=== Step 5A: verify that SSH to infrastructure host succeeds and allows public keys only ===
    283363
    284364'''Using:'''
    … …  
    289369 * Public-key SSH succeeds
    290370 * Password-based SSH does not succeed
     371
     372==== Results of testing: 2012-05-16 ====
     373
     374''Testing with Utah rack, whose control node is utah.control.geniracks.net (which believes its hostname is control.utah.geniracks.net, see instaticket:23).''
     375
     376 * Public key login succeeds:
     377{{{
     378capybara,[~],23:55(0)$ ssh -o PubkeyAuthentication=yes utah.control.geniracks.net
     379Welcome to Ubuntu precise (development branch) (GNU/Linux 3.2.0-23-generic x86_64)
     380
     381 * Documentation:  https://help.ubuntu.com/
     382
     383  System information as of Wed May 16 21:55:30 MDT 2012
     384
     385  System load:  0.0               Users logged in:       1
     386  Usage of /:   45.0% of 5.85GB   IP address for xenbr0: 155.98.34.2
     387  Memory usage: 24%               IP address for xenbr1: 10.1.1.254
     388  Swap usage:   0%                IP address for xenbr2: 10.2.1.254
     389  Processes:    150               IP address for xenbr3: 10.3.1.254
     390
     391  Graph this data and manage this system at https://landscape.canonical.com/
     392Last login: Wed May 16 21:45:14 2012 from capybara.bbn.com
     393control,[~],21:55(0)$
     394}}}
     395 * Password-based login should fail because i don't have a password set, and because `/etc/ssh/sshd_config` contains:
     396{{{
     397ChallengeResponseAuthentication no
     398PasswordAuthentication no
     399}}}
     400 and it does:
     401{{{
     402capybara,[~],23:56(0)$ ssh -o PubkeyAuthentication=no utah.control.geniracks.net
     403Permission denied (publickey).
     404}}}
    291405
    292406=== Step 5B: verify the absence of common unencrypted login protocols ===
    … …  
    301415 * Login does not succeed via any unencrypted login protocol
    302416
     417==== Results of testing: 2012-05-17 ====
     418
     419 * On Ubuntu, get a list of listeners using:
     420{{{
     421sudo netstat -anp | grep LISTEN
     422}}}
     423 * This reveals that only sshd is listening for remote connections on non-localhost interfaces.
     424
    303425=== Step 5C: verify sudo and sudo logging ===
    304426
    … …  
    311433 * The command which was run should be recorded in a log
    312434
     435==== Results of testing: 2012-05-17 ====
     436
     437 * The command succeeds:
     438{{{
     439control,[~],22:05(0)$ sudo whoami
     440root
     441}}}
     442 * A record of sudo is found in `/var/log/auth.log`:
     443{{{
     444control,[~],22:05(0)$ sudo tail /var/log/auth.log
     445...
     446May 16 22:05:17 control sudo:    chaos : TTY=pts/5 ; PWD=/home/chaos ; USER=root ; COMMAND=/usr/bin/whoami
     447...
     448}}}
     449
    313450== Step 6: verify access to experimental OpenVZ node ==
    314451
    … …  
    322459 * Public-key SSH succeeds from boss
    323460 * Password-based SSH does not succeed from outside of the rack
     461
     462==== Results of testing: 2012-05-17 ====
     463
     464''Testing with Utah rack, using shared OpenVZ host pc5.utah.geniracks.net.''
     465
     466 * SSH to `root` from boss does work:
     467{{{
     468boss,[~],22:12(0)$ sudo ssh pc5
     469@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
     470@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
     471@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
     472IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
     473Someone could be eavesdropping on you right now (man-in-the-middle attack)!
     474It is also possible that the RSA host key has just been changed.
     475The fingerprint for the RSA key sent by the remote host is
     47646:63:92:67:c8:75:20:4e:52:9f:2d:f6:cb:58:16:77.
     477Please contact your system administrator.
     478Add correct host key in /root/.ssh/known_hosts to get rid of this message.
     479Offending key in /root/.ssh/known_hosts:18
     480Password authentication is disabled to avoid man-in-the-middle attacks.
     481Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
     482Last login: Wed May 16 22:10:34 2012 from boss.utah.geniracks.net
     483[root@vhost1 ~]#
     484}}}
     485 * Incidentally, SSH as myself also works:
     486{{{
     487capybara,[~/src/git/tango-monitor],00:11(0)$ ssh pc5.utah.geniracks.net
     488The authenticity of host 'pc5.utah.geniracks.net (155.98.34.15)' can't be established.
     489RSA key fingerprint is 46:63:92:67:c8:75:20:4e:52:9f:2d:f6:cb:58:16:77.
     490Are you sure you want to continue connecting (yes/no)? yes
     491Warning: Permanently added 'pc5.utah.geniracks.net,155.98.34.15' (RSA) to the list of known hosts.
     492Last login: Wed May 16 22:11:24 2012 from boss.utah.geniracks.net
     493vhost1,[~],22:11(0)$
     494}}}
     495 * Hmm, password-based SSH as root (using the password listed for the node within the Emulab admin UI) also works:
     496{{{
     497capybara,[~],00:15(0)$ ssh root@pc5.utah.geniracks.net
     498root@pc5.utah.geniracks.net's password:
     499Last login: Wed May 16 22:12:21 2012 from boss.utah.geniracks.net
     500[root@vhost1 ~]#
     501}}}
     502 However, given the size of the pseudorandom passwords used for Emulab experimental hosts, i think it's fine to assert that root password guessing is not a realistic threat, so i am not opening a ticket.
    324503
    325504=== Step 6B: verify the absence of common unencrypted login protocols ===
    … …  
    334513 * Login does not succeed via any unencrypted login protocol
    335514
     515==== Results of testing: 2012-05-17 ====
     516
     517 * On RHEL-like OSes, get a list of listeners using:
     518{{{
     519sudo netstat -anp | grep LISTEN
     520}}}
     521 * This reveals that the following processes, which are not concerning, are listening for remote connections:
     522{{{
     523sshd
     524rpc.statd
     525rpcbind
     526emulab-syncd (emulab)
     527pubsubd (emulab)
     528}}}
     529 * There is also an unidentifiable process listening on this port:
     530{{{
     531tcp        0      0 0.0.0.0:58441               0.0.0.0:*                   LISTEN      -                   
     532}}}
     533 I was unable to turn up any further information about it from lsof, so i tried connecting to it (perhaps unreasonably).  I got:
     534{{{
     535$ telnet pc5.utah.geniracks.net 58441
     536Trying 155.98.34.15...
     537Connected to pc5.utah.geniracks.net.
     538Escape character is '^]'.
     539HELP
     540Connection closed by foreign host.
     541}}}
     542 and in the logs:
     543{{{
     544May 16 22:39:28 localhost kernel: [27502.595007] RPC: multiple fragments per record not supported
     545}}}
     546 So i think this is an RPC-based thing, and that this kind of testing is not very useful.
     547
    336548=== Step 6C: verify sudo and sudo logging ===
    337549
    … …  
    344556 * The command which was run should be recorded in a log
    345557
     558==== Results of testing: 2012-05-17 ====
     559
     560 * The command succeeds and the log entry is created:
     561{{{
     562vhost1,[/var/log],22:40(0)$ sudo whoami
     563root
     564
     565vhost1,[/var/log],22:43(0)$ sudo tail /var/log/secure
     566...
     567May 16 22:43:48 localhost sudo:    chaos : TTY=pts/2 ; PWD=/var/log ; USER=root ; COMMAND=/usr/bin/whoami
     568...
     569}}}
     570
    346571== Step 7: verify access to control network switch ==
    347572
    … …  
    353578'''Verify:'''
    354579 * SSH login succeeds
     580
     581==== Results of testing: 2012-05-17 ====
     582
     583''Tested using Utah rack.''
     584
     585 * SSH login does not succeed:
     586{{{
     587boss,[~],22:46(0)$ ssh procurve1
     588ssh: connect to host procurve1 port 22: Connection refused
     589}}}
     590 * Telnet succeeds instead, using the password in `/usr/testbed/etc/switch.pswd`:
     591{{{
     592boss,[~],22:47(0)$ telnet procurve1
     593Trying 10.1.1.253...
     594Connected to procurve1.
     595Escape character is '^]'.
     596...
     597Password:
     598...
     599ProCurve Switch 2610-24# 
     600}}}
     601
     602This is presumed to be acceptable as long as only server nodes (such as boss) have interfaces on 10.1.1.0/24.
    355603
    356604=== Step 7B: verify privileged access to the control network switch ===
    … …  
    365613 * Viewing the running configuration should succeed
    366614 * Viewing the MAC address table should succeed
     615
     616==== Results of testing: 2012-05-17 ====
     617
     618 * Enable command isn't needed, since login is privileged already
     619 * Running config is viewable:
     620{{{
     621ProCurve Switch 2610-24# show running-config
     622
     623Running configuration:
     624
     625; J9085A Configuration Editor; Created on release #R.11.70
     626...
     627}}}
     628 * MAC address table is viewable:
     629{{{
     630ProCurve Switch 2610-24# show mac-address
     631
     632 Status and Counters - Port Address Table
     633...
     634}}}
    367635
    368636=== Step 7C: verify absence of unencrypted login access ===
    … …  
    380648 * No other services appear to allow remote unencrypted authentication
    381649
     650==== Results of testing: 2012-05-17 ====
     651
     652Not applicable: given results of 7A, not bothering to test for other instances of unencrypted login.
     653
    382654=== Step 7D: verify serial console access to the device ===
    383655
    … …  
    403675 * SSH login succeeds
    404676
     677==== Results of testing: 2012-05-17 ====
     678
     679''Tested using Utah rack.''
     680
     681 * SSH login succeeds using the password in `/etc/testbed/etc/switch.pswd`:
     682{{{
     683boss,[~],22:56(0)$ ssh elabman@procurve2
     684We'd like to keep you up to date about:
     685  * Software feature updates
     686  * New product announcements
     687  * Special events
     688
     689Please register your products now at:  www.ProCurve.com
     690
     691elabman@procurve2's password:
     692...
     693ProCurve Switch 6600ml-48G-4XG# 
     694}}}
     695 * Telnet succeeds as well:
     696{{{
     697boss,[~],22:57(0)$ telnet procurve2
     698Trying 10.2.1.253...
     699Connected to procurve2.
     700Escape character is '^]'.
     701...
     702Password:
     703...
     704ProCurve Switch 6600ml-48G-4XG#
     705}}}
     706
     707This is presumed to be acceptable as long as only server nodes (such as boss) have interfaces on 10.2.1.0/24.
     708
    405709=== Step 8B: verify privileged access to the dataplane switch ===
    406710
    … …  
    414718 * Viewing the running configuration should succeed
    415719 * Viewing the MAC address table should succeed
     720
     721==== Results of testing: 2012-05-17 ====
     722
     723 * Enable command isn't needed, since login is privileged already
     724 * Running config is viewable:
     725{{{
     726ProCurve Switch 6600ml-48G-4XG# show running-config
     727
     728Running configuration:
     729
     730; J9452A Configuration Editor; Created on release #K.14.41
     731...
     732}}}
     733 * MAC address table is viewable:
     734{{{
     735ProCurve Switch 6600ml-48G-4XG# show mac-address
     736
     737 Status and Counters - Port Address Table
     738...
     739}}}
    416740
    417741=== Step 8C: verify absence of unencrypted login access ===
    … …  
    429753 * No other services appear to allow remote unencrypted authentication
    430754
     755==== Results of testing: 2012-05-17 ====
     756
     757Not applicable: given results of 8A, not bothering to test for other instances of unencrypted login.
     758
    431759=== Step 8D: verify serial console access to the device ===
    432760
    … …  
    448776 * From boss, attempt to connect via http (port 80) to the the pc1 iLO IP
    449777 * If a port 80 connection is successful, determine whether login is allowed via that interface
    450  * If any prospective unencrypted protocols were identified in the iLO configurations during IG-ADM-1 step 3F, attempt to connect to those ports from boss
     778 * If any prospective unencrypted protocols were identified in the iLO configurations during IG-ADM-1 step 3G, attempt to connect to those ports from boss
    451779
    452780'''Verify:'''
    … …  
    455783 * No other services appear to allow remote unencrypted authentication
    456784
     785==== Results of testing: 2012-05-17 ====
     786
     787 * The pc1.utah.geniracks.net iLO is at 155.98.34.103
     788 * Telnet fails:
     789{{{
     790boss,[~],23:06(0)$ telnet 155.98.34.103
     791Trying 155.98.34.103...
     792telnet: connect to address 155.98.34.103: Connection refused
     793telnet: Unable to connect to remote host
     794}}}
     795 * Unencrypted telnet to port 80 succeeds, but closes the connection immediately:
     796{{{
     797boss,[~],23:08(1)$ telnet 155.98.34.103 80
     798Trying 155.98.34.103...
     799Connected to 155.98.34.103.
     800Escape character is '^]'.
     801Connection closed by foreign host.
     802}}}
     803 In a web browser, this connection is redirected from port 80 to port 443
     804 * According to Administration -> Access Settings in iLO, the only additional ports this thing uses are remote console, virtual media, and IPMI/DCMI, all expected iLO functions which i don't need to test this way.