Changes between Version 20 and Version 21 of GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-MON-1


Ignore:
Timestamp:
01/07/13 17:46:14 (6 years ago)
Author:
chaos@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-MON-1

    v20 v21  
    2020
    2121|| '''Step''' || '''State'''                    || '''Date completed''' || '''Open Tickets''' || '''Closed Tickets/Comments'''                                                             ||
    22 || 1          || [[Color(yellow,Complete)]]     ||                      ||                    || ([instaticket:24]) retest on BBN rack to verify availability of Emulab source/object tree ||
     22|| 1          || [[Color(orange,Blocked)]]      ||                      || [instaticket:80]   || ([instaticket:24]) blocked on availability of Emulab source/object tree on BBN boss      ||
    2323|| 2          || [[Color(yellow,Complete)]]     ||                      ||                    || ([instaticket:24]) retest on BBN rack to verify availability of Emulab source/object tree ||
    2424|| 3          || [[Color(green,Pass)]]          || 2012-11-21           ||                    ||                                                                                           ||
     
    515515=== Results of testing step 1: 2013-01-07 ===
    516516
    517  * In fact `sockstat -lL46` is my goto on FreeBSD.  Get a full list of all binaries of processes which are listening on IPv4 or IPv6 sockets from non-localhost addresses:
    518 {{{
    519 boss,[~],09:05(0)$ for pid in $(sockstat -lL46 | awk '{print $3}' | grep -v PID | sort -u); do procstat -b $pid; done | awk '{print $3}' | sort -u
    520 procstat: sysctl: kern.proc.pathname: 1340: No such file or directory
    521 procstat: sysctl: kern.proc.pathname: 1341: No such file or directory
    522 procstat: sysctl: kern.proc.pathname: 1342: No such file or directory
    523 procstat: sysctl: kern.proc.pathname: 1343: No such file or directory
    524 procstat: sysctl: kern.proc.pathname: 1344: No such file or directory
    525 procstat: sysctl: kern.proc.pathname: 1345: No such file or directory
    526 procstat: sysctl: kern.proc.pathname: 1346: No such file or directory
    527 procstat: sysctl: kern.proc.pathname: 1347: No such file or directory
    528 procstat: sysctl: kern.proc.pathname: 1348: No such file or directory
    529 procstat: sysctl: kern.proc.pathname: 1349: No such file or directory
    530 procstat: sysctl: kern.proc.pathname: 1350: No such file or directory
    531 procstat: sysctl: kern.proc.pathname: 1351: No such file or directory
    532 procstat: sysctl: kern.proc.pathname: 1352: No such file or directory
    533 procstat: sysctl: kern.proc.pathname: 1353: No such file or directory
     517Testing on `boss.instageni.gpolab.bbn.com`:
     518
     519 * `sockstat -lL46` is my goto on FreeBSD.  Get a full list of all binaries of processes which are listening on IPv4 or IPv6 sockets from non-localhost addresses:
     520{{{
     521boss,[~],17:07(0)$ for pid in $(sockstat -lL46 | awk '{print $3}' | grep -v PID | sort -u); do procstat -b $pid; done | awk '{print $3}' | sort -u
     522usage: procstat [-h] [-C] [-M core] [-N system] [-w interval]
     523                [-b | -c | -e | -f | -i | -j | -k | -l | -s | -t | -v | -x] [-a | pid ...]
    534524/usr/libexec/sendmail/sendmail
    535 /usr/local/bin/python2.6
     525/usr/local/bin/perl
     526/usr/local/bin/python2.7
    536527/usr/local/libexec/pubsubd
    537528/usr/local/libexec/tftpd
     
    549540/usr/testbed/sbin/mfrisbeed
    550541/usr/testbed/sbin/sdcollectd
     542/usr/testbed/sbin/tmcd
    551543PATH
    552544}}}
    553  * Huh, well, that's weird.  All the PIDs procstat choked on seem to be tmcd and its children.  So we can call the list:
    554 {{{
    555 /usr/testbed/sbin/tmcd
     545 * The errors were a couple of pids which get reported as `?`:
     546{{{
     547boss,[~],17:14(0)$ sockstat -lL46
     548USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS     
     549...
     550?        ?          ?     ?  udp4   *:2049                *:*
     551?        ?          ?     ?  udp6   *:2049                *:*
     552}}}
     553 * Hmm, that's pretty strange, since nfsd is also represented:
     554{{{
     555boss,[~],17:18(0)$ sockstat -lL46 | grep 2049
     556root     nfsd       1441  5  tcp4   *:2049                *:*
     557root     nfsd       1441  6  tcp6   *:2049                *:*
     558?        ?          ?     ?  udp4   *:2049                *:*
     559?        ?          ?     ?  udp6   *:2049                *:*
     560}}}
     561 The Utah rack shows the same thing, though, so i'm going to guess it's normal, and just has to do with the strange kernel dependencies of NFS.
     562 * So, the list of listeners is:
     563{{{
    556564/usr/libexec/sendmail/sendmail
    557 /usr/local/bin/python2.6
     565/usr/local/bin/perl
     566/usr/local/bin/python2.7
    558567/usr/local/libexec/pubsubd
    559568/usr/local/libexec/tftpd
     
    571580/usr/testbed/sbin/mfrisbeed
    572581/usr/testbed/sbin/sdcollectd
     582/usr/testbed/sbin/tmcd
    573583}}}
    574584 * The following commands are sourced from freebsd packages:
    575585{{{
    576 $ pkg_info -W /usr/local/bin/python2.6
    577 /usr/local/bin/python2.6 was installed by package python26-2.6.6
     586$ pkg_info -W /usr/local/bin/perl
     587/usr/local/bin/perl was installed by package perl-5.12.4_3
     588
     589$ pkg_info -W /usr/local/bin/python2.7
     590/usr/local/bin/python2.7 was installed by package python27-2.7.2_4
    578591
    579592$ pkg_info -W /usr/local/libexec/pubsubd
     
    584597
    585598$ pkg_info -W /usr/local/sbin/dhcpd
    586 /usr/local/sbin/dhcpd was installed by package isc-dhcp42-server-4.2.3_1
     599/usr/local/sbin/dhcpd was installed by package isc-dhcp42-server-4.2.3_2
    587600
    588601$ pkg_info -W /usr/local/sbin/httpd
    589 /usr/local/sbin/httpd was installed by package apache-2.2.21
     602/usr/local/sbin/httpd was installed by package apache-2.2.22_5
    590603}}}
    591604   * So the summary of sourced packages here is:
    592605{{{
    593 apache-2.2.21
     606apache-2.2.22_5
    594607emulab-tftp-hpa-0.48
    595 isc-dhcp42-server-4.2.3_1
     608isc-dhcp42-server-4.2.3_2
     609perl-5.12.4_3
    596610pubsub-0.95
    597 python26-2.6.6
     611python27-2.7.2_4
    598612}}}
    599613   * For each of these, i can verify using the distinfo that /usr/ports seems to have the same version as is installed:
    600614{{{
    601615$ cat /usr/ports/www/apache22/distinfo
    602 SHA256 (apache22/httpd-2.2.21.tar.bz2) = 18d5591fe48cfbac44fc20316036ffe17456df60bc3a2aaad238d56c6445577f
    603 SIZE (apache22/httpd-2.2.21.tar.bz2) = 5324905
     616SHA256 (apache22/httpd-2.2.22.tar.bz2) = dcdc9f1dc722f84798caf69d69dca78daa5e09a4269060045aeca7e4f44cb231
     617SIZE (apache22/httpd-2.2.22.tar.bz2) = 5378934
    604618
    605619$ cat /usr/ports/misc/emulab-tftp-hpa/distinfo
     
    609623
    610624$ cat /usr/ports/net/isc-dhcp42-server/distinfo
    611 SHA256 (dhcp-4.2.3-P1.tar.gz) = 4860625e07c290768d513b9a2260e655ae4bad4cc5c4c8f5eaaeb4f9cbfa96af
    612 SIZE (dhcp-4.2.3-P1.tar.gz) = 9525670
     625SHA256 (dhcp-4.2.3-P2.tar.gz) = 5cf7ae2cad9c4ca0103748b2476ec8ea78484e408f8fe597e4e0a4afb051b469
     626SIZE (dhcp-4.2.3-P2.tar.gz) = 9527546
     627
     628$ cat /usr/ports/lang/perl5.12/distinfo
     629SHA256 (perl/perl-5.12.4.tar.bz2) = c920fa01a34b7752e8cbbc99c6921b908df60611d84a29c4377e4a798604767e
     630SIZE (perl/perl-5.12.4.tar.bz2) = 12350353
     631SHA256 (perl/BSDPAN-20111107.tar.bz2) = 2f03218a592dc65ebfdc3c6b9394d91dcf4c53aa5290a08458b837baad5a21f9
     632SIZE (perl/BSDPAN-20111107.tar.bz2) = 8448
    613633
    614634$ cat /usr/ports/misc/emulab-pubsub/distinfo
     
    617637SHA256 (pubsub-0.95.tar.gz) = 7b8eb4f495a531454b16abb8e8399e031fa17a31e09c4ce47486cedaa3ae4410
    618638
    619 $ cat /usr/ports/lang/python26/distinfo
    620 MD5 (python/Python-2.6.6.tgz) = b2f209df270a33315e62c1ffac1937f0
    621 SHA256 (python/Python-2.6.6.tgz) = 372f66db46d773214e4619df1794a26449158f626138d4d2141a64c2f017fae1
    622 SIZE (python/Python-2.6.6.tgz) = 13318547
     639$ cat /usr/ports/lang/python27/distinfo
     640SHA256 (python/Python-2.7.2.tgz) = 1d54b7096c17902c3f40ffce7e5b84e0072d0144024184fff184a84d563abbb3
     641SIZE (python/Python-2.7.2.tgz) = 14091337
    623642}}}
    624643
    625644 * The following commands aren't part of packages (`pkg_info -W` reports nothing):
    626645{{{
    627 /usr/testbed/sbin/tmcd
    628 /usr/testbed/sbin/bootinfo
    629 /usr/testbed/sbin/mfrisbeed
    630 /usr/testbed/sbin/sdcollectd
    631646/usr/libexec/sendmail/sendmail
    632647/usr/sbin/inetd
    633 /usr/sbin/lwresd             # hard link from /usr/sbin/named
     648/usr/sbin/lwresd
    634649/usr/sbin/mountd
    635650/usr/sbin/nfsd
     
    638653/usr/sbin/sshd
    639654/usr/sbin/syslogd
     655/usr/testbed/sbin/bootinfo
     656/usr/testbed/sbin/mfrisbeed
     657/usr/testbed/sbin/sdcollectd
     658/usr/testbed/sbin/tmcd
     659}}}
    640660}}}
    641661 * The assumption is that these are either part of the FreeBSD base system, or are part of Emulab.  How do we find out which?
    642    * Since Emulab has been compiled recently, i can look in the canonical source of that compile, which i believe is `/users/stoller/testbed/obj`.  Here's a process for checking various things in `/usr/testbed/sbin`, on the suspicion that they are probably Emulab binaries:
    643 {{{
     662   * I can look in the canonical source of the emulab compile, which should be `/usr/testbed/obj/testbed`.  Here's a process for checking various things in `/usr/testbed/sbin`, on the suspicion that they are probably Emulab binaries:
     663{{{
     664cd /usr/testbed/obj/testbed
    644665shortname=bootinfo    # or whatever
    645666shortmd5=$(md5 /usr/testbed/sbin/$shortname | awk '{print $4}')
    646667for path in $(find . -type f -name $shortname 2> /dev/null); do md5 $path; done | grep $shortmd5
    647668}}}
    648    * That finds that the following items are indeed from Emulab:
    649 {{{
    650 /usr/testbed/sbin/bootinfo:            ./pxe/bootinfo
    651 /usr/testbed/sbin/mfrisbeed:           ./clientside/os/frisbee.redux/mfrisbeed
    652 /usr/testbed/sbin/sdcollectd:          ./clientside/sensors/slothd/sdcollectd
    653 /usr/testbed/sbin/tmcd:                ./tmcd/tmcd
    654 }}}
    655    * Note that, in instaticket:24, we found that on a non-development rack, source and object files will be found in `/usr/testbed/{src,obj}/emulab-stable`.
    656  * So the following items are not from Emulab, and we assume they would be part of the base install:
    657 {{{
    658 /usr/libexec/sendmail/sendmail
    659 /usr/sbin/inetd
    660 /usr/sbin/lwresd             # hard link from /usr/sbin/named
    661 /usr/sbin/mountd
    662 /usr/sbin/nfsd
    663 /usr/sbin/ntpd
    664 /usr/sbin/rpcbind
    665 /usr/sbin/sshd
    666 /usr/sbin/syslogd
    667 }}}
    668    * If the OS had been compiled recently, i could look in /usr/obj for binaries which were identical to things on the system.  Since it's running the base install, find out what version that is:
    669 {{{
    670 $ uname -r
    671 8.3-RC1
    672 }}}
    673    * Hmm, so it looks like [http://lists.freebsd.org/pipermail/freebsd-stable/2012-March/066722.html 8.3-RC1] is not exactly the version which became [http://lists.freebsd.org/pipermail/freebsd-announce/2012-April/001417.html 8.3-RELEASE].  I did a bit of poking around to find the i386 install disc for 8.3-RC1, eventually locating a copy at [http://ftp.at.freebsd.org/pub/FreeBSD/ISO-IMAGES-i386/8.3/FreeBSD-8.3-RC1-i386-disc1.iso]
    674    * Loopback-mounted that on my desktop:
    675 {{{
    676 sudo mount -o loop FreeBSD-8.3-RC1-i386-disc1.iso /mnt/iso
    677 mkdir -p ~/tmp/8.3-RC1/{base,src}
    678 cd ~/tmp/8.3-RC1/base
    679 cat /mnt/iso/8.3-RC1/base/base.?? | tar xvzf -
    680 cd ~/tmp/8.3-RC1/src
    681 dists="base bin cddl contrib crypto etc games gnu include krb5 lib libexec release rescue sbin secure share sys tools ubin usbin"
    682 for dist in $dists; do
    683   cat /mnt/iso/8.3-RC1/src/s${dist}.?? | tar xvzf -
    684 done
    685 }}}
    686    * So, this isn't perfect, in that at a glance, the object checksums don't match between what is on boss and what i found in this iso.  However:
    687      * It definitely would tell someone which items are part of the freebsd base distribution (if i find `~/tmp/8.3-RC1/base/usr/sbin/inetd`, then `boss:/usr/sbin/inetd` has been verified to come from FreeBSD)
    688      * I can look at the source code from which the object was presumably taken, even if i can't verify that the object is unmodified.
    689    * Using this, i verified that all the non-Emulab-sourced, non-packaged listeners on boss are indeed from the FreeBSD base distribution.
    690 
    691 All that remains is to retest on the BBN rack once it arrives, and verify that the Emulab source distribution lands in a reasonable place.
     669   * Hmm, the first thing i tried that on, bootinfo, did not match:
     670{{{
     671boss,[~],15:32(0)$ md5 /usr/testbed/sbin/bootinfo /usr/testbed/obj/testbed/pxe/bootinfo
     672MD5 (/usr/testbed/sbin/bootinfo) = 131c06c10ca3b7438c85aafa93ccc93b
     673MD5 (/usr/testbed/obj/testbed/pxe/bootinfo) = a98b23f4598f9c2d268b614cbdbc8e30
     674
     675boss,[~],15:32(0)$ ls -l /usr/testbed/sbin/bootinfo /usr/testbed/obj/testbed/pxe/
     676bootinfo
     677-rwxr-xr-x  1 root  wheel    136499 Mar  8  2012 /usr/testbed/obj/testbed/pxe/bootinfo*
     678-rwxr-xr-x  1 root  tbadmin  140415 Oct 29 16:39 /usr/testbed/sbin/bootinfo*
     679}}}
     680   Note that, [instaticket:24] claimed that, on a non-development rack, source and object files will be found in `/usr/testbed/{src,obj}/emulab-stable`.
     681   * I'm not going to bother retesting the things from the FreeBSD base install --- as mentioned when i tested at Utah, the MD5 sums don't match, so i'd just be using the FreeBSD base install ISO to verify the identities of the binaries.  Since the listed binaries are the same this time as last time, i'm going to assume they are still reflected in the FreeBSD base distro.
     682
     683[instaticket:80] covers the question of the locations of the Emulab src/obj trees on the BBN boss node.
    692684
    693685== Step 2: identify network-listening software on the ops node ==