Changes between Version 11 and Version 12 of GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-MON-1


Ignore:
Timestamp:
05/27/12 12:18:22 (7 years ago)
Author:
chaos@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-MON-1

    v11 v12  
    55''This page is GPO's working page for performing IG-MON-1.  It is public for informational purposes, but it is not an official status report.  See [wiki:GENIRacksHome/InstageniRacks/AcceptanceTestStatus] for the current status of InstaGENI acceptance tests.''
    66
    7 ''Last substantive edit of this page: 2012-05-26''
     7''Last substantive edit of this page: 2012-05-27''
    88
    99== Page format ==
     
    1919== Status of test ==
    2020
    21 || '''Step''' || '''State'''                    || '''Date completed''' || '''Open Tickets''' || '''Closed Tickets/Comments'''                                                                                           ||
    22 || 1          || [[Color(#63B8FF,In Progress)]] ||                      ||                    || (instaticket:24) [[BR]] need to retest and verify that resolution of instaticket:24 gives me all the information i need ||
    23 || 2          || [[Color(#63B8FF,In Progress)]] ||                      ||                    || (instaticket:24) [[BR]] need to retest and verify that resolution of instaticket:24 gives me all the information i need ||
    24 || 3          || [[Color(orange,Blocked)]]      ||                      ||                    || blocked on sudo access to foam VM                                                                                       ||
    25 || 4          || [[Color(orange,Blocked)]]      ||                      ||                    || blocked on sudo access to flowvisor VM                                                                                  ||
    26 || 5          || [[Color(green,Pass)]]          ||                      ||                    ||                                                                                                                         ||
    27 || 6          || [[Color(orange,Blocked)]]      ||                      || instaticket:25     || blocked on resolution of "how to get Emulab version" question for Emulab images                                         ||
    28 || 7          || [[Color(yellow,Blocked-site)]] ||                      ||                    || blocked on receipt of the BBN rack                                                                                      ||
     21|| '''Step''' || '''State'''                    || '''Date completed''' || '''Open Tickets''' || '''Closed Tickets/Comments'''                                                                  ||
     22|| 1          || [[Color(yellow,Complete)]]     ||                      ||                    || (instaticket:24) [[BR]] retest on BBN rack to verify availability of Emulab source/object tree ||
     23|| 2          || [[Color(yellow,Complete)]]     ||                      ||                    || (instaticket:24) [[BR]] retest on BBN rack to verify availability of Emulab source/object tree ||
     24|| 3          || [[Color(orange,Blocked)]]      ||                      ||                    || blocked on sudo access to foam VM                                                              ||
     25|| 4          || [[Color(orange,Blocked)]]      ||                      ||                    || blocked on sudo access to flowvisor VM                                                         ||
     26|| 5          || [[Color(green,Pass)]]          ||                      ||                    ||                                                                                                ||
     27|| 6          || [[Color(orange,Blocked)]]      ||                      || instaticket:25     || blocked on resolution of "how to get Emulab version" question for Emulab images                ||
     28|| 7          || [[Color(yellow,Blocked-site)]] ||                      ||                    || blocked on receipt of the BBN rack                                                             ||
    2929
    3030== High-level description from test plan ==
     
    335335 3. Ask someone where the source for the FreeBSD packages installed on the system, some of which are Emulab-specific, come from.
    336336
     337=== Results of testing step 1: 2012-05-27 ===
     338
     339 * In fact `sockstat -lL46` is my goto on FreeBSD.  Get a full list of all binaries of processes which are listening on IPv4 or IPv6 sockets from non-localhost addresses:
     340{{{
     341boss,[~],09:05(0)$ for pid in $(sockstat -lL46 | awk '{print $3}' | grep -v PID | sort -u); do procstat -b $pid; done | awk '{print $3}' | sort -u
     342procstat: sysctl: kern.proc.pathname: 1340: No such file or directory
     343procstat: sysctl: kern.proc.pathname: 1341: No such file or directory
     344procstat: sysctl: kern.proc.pathname: 1342: No such file or directory
     345procstat: sysctl: kern.proc.pathname: 1343: No such file or directory
     346procstat: sysctl: kern.proc.pathname: 1344: No such file or directory
     347procstat: sysctl: kern.proc.pathname: 1345: No such file or directory
     348procstat: sysctl: kern.proc.pathname: 1346: No such file or directory
     349procstat: sysctl: kern.proc.pathname: 1347: No such file or directory
     350procstat: sysctl: kern.proc.pathname: 1348: No such file or directory
     351procstat: sysctl: kern.proc.pathname: 1349: No such file or directory
     352procstat: sysctl: kern.proc.pathname: 1350: No such file or directory
     353procstat: sysctl: kern.proc.pathname: 1351: No such file or directory
     354procstat: sysctl: kern.proc.pathname: 1352: No such file or directory
     355procstat: sysctl: kern.proc.pathname: 1353: No such file or directory
     356/usr/libexec/sendmail/sendmail
     357/usr/local/bin/python2.6
     358/usr/local/libexec/pubsubd
     359/usr/local/libexec/tftpd
     360/usr/local/sbin/dhcpd
     361/usr/local/sbin/httpd
     362/usr/sbin/inetd
     363/usr/sbin/lwresd
     364/usr/sbin/mountd
     365/usr/sbin/nfsd
     366/usr/sbin/ntpd
     367/usr/sbin/rpcbind
     368/usr/sbin/sshd
     369/usr/sbin/syslogd
     370/usr/testbed/sbin/bootinfo
     371/usr/testbed/sbin/mfrisbeed
     372/usr/testbed/sbin/sdcollectd
     373PATH
     374}}}
     375 * Huh, well, that's weird.  All the PIDs procstat choked on seem to be tmcd and its children.  So we can call the list:
     376{{{
     377/usr/testbed/sbin/tmcd
     378/usr/libexec/sendmail/sendmail
     379/usr/local/bin/python2.6
     380/usr/local/libexec/pubsubd
     381/usr/local/libexec/tftpd
     382/usr/local/sbin/dhcpd
     383/usr/local/sbin/httpd
     384/usr/sbin/inetd
     385/usr/sbin/lwresd
     386/usr/sbin/mountd
     387/usr/sbin/nfsd
     388/usr/sbin/ntpd
     389/usr/sbin/rpcbind
     390/usr/sbin/sshd
     391/usr/sbin/syslogd
     392/usr/testbed/sbin/bootinfo
     393/usr/testbed/sbin/mfrisbeed
     394/usr/testbed/sbin/sdcollectd
     395}}}
     396 * The following commands are sourced from freebsd packages:
     397{{{
     398$ pkg_info -W /usr/local/bin/python2.6
     399/usr/local/bin/python2.6 was installed by package python26-2.6.6
     400
     401$ pkg_info -W /usr/local/libexec/pubsubd
     402/usr/local/libexec/pubsubd was installed by package pubsub-0.95
     403
     404$ pkg_info -W /usr/local/libexec/tftpd
     405/usr/local/libexec/tftpd was installed by package emulab-tftp-hpa-0.48
     406
     407$ pkg_info -W /usr/local/sbin/dhcpd
     408/usr/local/sbin/dhcpd was installed by package isc-dhcp42-server-4.2.3_1
     409
     410$ pkg_info -W /usr/local/sbin/httpd
     411/usr/local/sbin/httpd was installed by package apache-2.2.21
     412}}}
     413   * So the summary of sourced packages here is:
     414{{{
     415apache-2.2.21
     416emulab-tftp-hpa-0.48
     417isc-dhcp42-server-4.2.3_1
     418pubsub-0.95
     419python26-2.6.6
     420}}}
     421   * For each of these, i can verify using the distinfo that /usr/ports seems to have the same version as is installed:
     422{{{
     423$ cat /usr/ports/www/apache22/distinfo
     424SHA256 (apache22/httpd-2.2.21.tar.bz2) = 18d5591fe48cfbac44fc20316036ffe17456df60bc3a2aaad238d56c6445577f
     425SIZE (apache22/httpd-2.2.21.tar.bz2) = 5324905
     426
     427$ cat /usr/ports/misc/emulab-tftp-hpa/distinfo
     428MD5 (tftp-hpa-0.48.tar.gz) = 73dd29dfa9062a174163f6a267e0b5f7
     429SHA256 (tftp-hpa-0.48.tar.gz) = 0bba7dc69a0174b8a9d5b340fa79c9c4874c61a880fe1cd8129e7644dd0967bb
     430SIZE (tftp-hpa-0.48.tar.gz) = 104844
     431
     432$ cat /usr/ports/net/isc-dhcp42-server/distinfo
     433SHA256 (dhcp-4.2.3-P1.tar.gz) = 4860625e07c290768d513b9a2260e655ae4bad4cc5c4c8f5eaaeb4f9cbfa96af
     434SIZE (dhcp-4.2.3-P1.tar.gz) = 9525670
     435
     436$ cat /usr/ports/misc/emulab-pubsub/distinfo
     437MD5 (pubsub-0.9.tar.gz) = 25216f20821bb7094a1853b1771cb5b8
     438MD5 (pubsub-0.95.tar.gz) = 9e012e65ebb4979e99f686e5f8fa8104
     439SHA256 (pubsub-0.95.tar.gz) = 7b8eb4f495a531454b16abb8e8399e031fa17a31e09c4ce47486cedaa3ae4410
     440
     441$ cat /usr/ports/lang/python26/distinfo
     442MD5 (python/Python-2.6.6.tgz) = b2f209df270a33315e62c1ffac1937f0
     443SHA256 (python/Python-2.6.6.tgz) = 372f66db46d773214e4619df1794a26449158f626138d4d2141a64c2f017fae1
     444SIZE (python/Python-2.6.6.tgz) = 13318547
     445}}}
     446
     447 * The following commands aren't part of packages (`pkg_info -W` reports nothing):
     448{{{
     449/usr/testbed/sbin/tmcd
     450/usr/testbed/sbin/bootinfo
     451/usr/testbed/sbin/mfrisbeed
     452/usr/testbed/sbin/sdcollectd
     453/usr/libexec/sendmail/sendmail
     454/usr/sbin/inetd
     455/usr/sbin/lwresd             # hard link from /usr/sbin/named
     456/usr/sbin/mountd
     457/usr/sbin/nfsd
     458/usr/sbin/ntpd
     459/usr/sbin/rpcbind
     460/usr/sbin/sshd
     461/usr/sbin/syslogd
     462}}}
     463 * The assumption is that these are either part of the FreeBSD base system, or are part of Emulab.  How do we find out which?
     464   * Since Emulab has been compiled recently, i can look in the canonical source of that compile, which i believe is `/users/stoller/testbed/obj`.  Here's a process for checking various things in `/usr/testbed/sbin`, on the suspicion that they are probably Emulab binaries:
     465{{{
     466shortname=bootinfo    # or whatever
     467shortmd5=$(md5 /usr/testbed/sbin/$shortname | awk '{print $4}')
     468for path in $(find . -type f -name $shortname 2> /dev/null); do md5 $path; done | grep $shortmd5
     469}}}
     470   * That finds that the following items are indeed from Emulab:
     471{{{
     472/usr/testbed/sbin/bootinfo:            ./pxe/bootinfo
     473/usr/testbed/sbin/mfrisbeed:           ./clientside/os/frisbee.redux/mfrisbeed
     474/usr/testbed/sbin/sdcollectd:          ./clientside/sensors/slothd/sdcollectd
     475/usr/testbed/sbin/tmcd:                ./tmcd/tmcd
     476}}}
     477   * Note that, in instaticket:24, we found that on a non-development rack, source and object files will be found in `/usr/testbed/{src,obj}/emulab-stable`.
     478 * So the following items are not from Emulab, and we assume they would be part of the base install:
     479{{{
     480/usr/libexec/sendmail/sendmail
     481/usr/sbin/inetd
     482/usr/sbin/lwresd             # hard link from /usr/sbin/named
     483/usr/sbin/mountd
     484/usr/sbin/nfsd
     485/usr/sbin/ntpd
     486/usr/sbin/rpcbind
     487/usr/sbin/sshd
     488/usr/sbin/syslogd
     489}}}
     490   * If the OS had been compiled recently, i could look in /usr/obj for binaries which were identical to things on the system.  Since it's running the base install, find out what version that is:
     491{{{
     492$ uname -r
     4938.3-RC1
     494}}}
     495   * Hmm, so it looks like [http://lists.freebsd.org/pipermail/freebsd-stable/2012-March/066722.html 8.3-RC1] is not exactly the version which became [http://lists.freebsd.org/pipermail/freebsd-announce/2012-April/001417.html 8.3-RELEASE].  I did a bit of poking around to find the i386 install disc for 8.3-RC1, eventually locating a copy at [http://ftp.at.freebsd.org/pub/FreeBSD/ISO-IMAGES-i386/8.3/FreeBSD-8.3-RC1-i386-disc1.iso]
     496   * Loopback-mounted that on my desktop:
     497{{{
     498sudo mount -o loop FreeBSD-8.3-RC1-i386-disc1.iso /mnt/iso
     499mkdir -p ~/tmp/8.3-RC1/{base,src}
     500cd ~/tmp/8.3-RC1/base
     501cat /mnt/iso/8.3-RC1/base/base.?? | tar xvzf -
     502cd ~/tmp/8.3-RC1/src
     503dists="base bin cddl contrib crypto etc games gnu include krb5 lib libexec release rescue sbin secure share sys tools ubin usbin"
     504for dist in $dists; do
     505  cat /mnt/iso/8.3-RC1/src/s${dist}.?? | tar xvzf -
     506done
     507}}}
     508   * So, this isn't perfect, in that at a glance, the object checksums don't match between what is on boss and what i found in this iso.  However:
     509     * It definitely would tell someone which items are part of the freebsd base distribution (if i find `~/tmp/8.3-RC1/base/usr/sbin/inetd`, then `boss:/usr/sbin/inetd` has been verified to come from FreeBSD)
     510     * I can look at the source code from which the object was presumably taken, even if i can't verify that the object is unmodified.
     511   * Using this, i verified that all the non-Emulab-sourced, non-packaged listeners on boss are indeed from the FreeBSD base distribution.
     512
     513All that remains is to retest on the BBN rack once it arrives, and verify that the Emulab source distribution lands in a reasonable place.
     514
    337515== Step 2: identify network-listening software on the ops node ==
    338516
     
    350528 * FreeBSD ports can be identified for each FreeBSD-sourced package
    351529 * The source code and identifiable version (e.g. a git tag) can be found for each non-FreeBSD software source
     530
     531=== Results of testing step 2: 2012-05-27 ===
     532
     533 * In fact `sockstat -lL46` is my goto on FreeBSD.  Get a full list of all binaries of processes which are listening on IPv4 or IPv6 sockets from non-localhost addresses:
     534{{{
     535ops,[~],09:33(0)$ for pid in $(sockstat -lL46 | awk '{print $3}' | grep -v PID | sort -u); do procstat -b $pid; done | awk '{print $3}' | sort -u
     536/usr/libexec/sendmail/sendmail
     537/usr/local/libexec/mysqld
     538/usr/local/libexec/pubsubd
     539/usr/local/sbin/httpd
     540/usr/sbin/mountd
     541/usr/sbin/nfsd
     542/usr/sbin/ntpd
     543/usr/sbin/rpcbind
     544/usr/sbin/sshd
     545/usr/sbin/syslogd
     546PATH
     547}}}
     548 * The following commands are sourced from freebsd packages:
     549{{{
     550$ pkg_info -W /usr/local/libexec/mysqld
     551/usr/local/libexec/mysqld was installed by package mysql-server-5.0.92
     552
     553$ pkg_info -W /usr/local/libexec/pubsubd
     554/usr/local/libexec/pubsubd was installed by package pubsub-0.95
     555
     556$ pkg_info -W /usr/local/sbin/httpd
     557/usr/local/sbin/httpd was installed by package apache-2.2.21
     558}}}
     559   * So the summary of sourced packages here is:
     560{{{
     561apache-2.2.21
     562pubsub-0.95
     563mysql-server-5.0.92
     564}}}
     565   * There is no /usr/ports on ops, so i will assume that all needed information can be found in /usr/ports on boss.  Given that, we already checked apache and pubsub, so the only remaining item is mysql-server:
     566{{{
     567$ cat /usr/ports/databases/mysql50-server/distinfo
     568SHA256 (mysql-5.0.92.tar.gz) = 223530c4cd6d45e2906e6c7d33615c5824816dfdb772d104f4165a483ea78a55
     569SIZE (mysql-5.0.92.tar.gz) = 22440590
     570}}}
     571 * The following commands aren't part of packages (`pkg_info -W` reports nothing):
     572{{{
     573/usr/libexec/sendmail/sendmail
     574/usr/sbin/mountd
     575/usr/sbin/nfsd
     576/usr/sbin/ntpd
     577/usr/sbin/rpcbind
     578/usr/sbin/sshd
     579/usr/sbin/syslogd
     580}}}
     581 * The assumption is that these are either part of the FreeBSD base system, or are part of Emulab.  I'll assume at a guess that all of these are part of the base system, which is the same release and architecture as boss:
     582{{{
     583ops,[~],09:35(0)$ uname -r
     5848.3-RC1
     585
     586ops,[~],09:38(0)$ uname -m
     587i386
     588}}}
     589 * Using the FreeBSD base distribution downloaded in step 1, i verified that those commands all appear in the FreeBSD 8.3-RC1 base distribution.
    352590
    353591== Step 3: identify network-listening software on the foam node ==