Changes between Version 6 and Version 7 of GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-MON-1


Ignore:
Timestamp:
05/18/12 05:23:32 (12 years ago)
Author:
chaos@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-MON-1

    v6 v7  
    1919== Status of test ==
    2020
    21 || '''Step''' || '''State'''               || '''Date completed''' || '''Tickets''' || '''Comments'''                         ||
    22 || 1          ||                           ||                      ||               || ready to test                          ||
    23 || 2          ||                           ||                      ||               || ready to test                          ||
    24 || 3          || [[Color(orange,Blocked)]] ||                      ||               || blocked on sudo access to foam VM      ||
    25 || 4          || [[Color(orange,Blocked)]] ||                      ||               || blocked on sudo access to flowvisor VM ||
    26 || 5          ||                           ||                      ||               || ready to test                          ||
    27 || 6          ||                           ||                      ||               || ready to test                          ||
    28 || 7          ||                           ||                      ||               || ready to test                          ||
     21|| '''Step''' || '''State'''               || '''Date completed''' || '''Tickets'''  || '''Comments'''                                                                    ||
     22|| 1          || [[Color(orange,Blocked)]] ||                      || instaticket:24 || blocked on resolution of "how to get source" questions for Emulab FreeBSD systems ||
     23|| 2          || [[Color(orange,Blocked)]] ||                      || instaticket:24 || blocked on resolution of "how to get source" questions for Emulab FreeBSD systems ||
     24|| 3          || [[Color(orange,Blocked)]] ||                      ||                || blocked on sudo access to foam VM                                                 ||
     25|| 4          || [[Color(orange,Blocked)]] ||                      ||                || blocked on sudo access to flowvisor VM                                            ||
     26|| 5          || [[Color(green,Pass)]]     ||                      ||                ||                                                                                   ||
     27|| 6          || [[Color(orange,Blocked)]] ||                      ||                || blocked on resolution of "how to get Emulab version" question for Emulab images   ||
     28|| 7          || [[Color(yellow,Blocked-site)]] ||                      ||                || blocked on receipt of the BBN rack ||
    2929
    3030== High-level description from test plan ==
    … …  
    5151 * VII.13. A site administrator can locate recent syslogs for all infrastructure and experimental hosts. (D.6.b)
    5252
     53=== Results of experiment setup on rack: 2012-05-17 ===
     54
     55''Preparation: i wrote these tests on the assumption that there would be some active experiments on the rack while i was testing, and there aren't.  So i wanted some running experiments to look at.''
     56
     57 * First attempt (didn't work):
     58   * Here is my rspec:
     59{{{
     60jericho,[~],10:00(0)$ cat ~/IG-MON-nodes-A.rspec
     61<?xml version="1.0" encoding="UTF-8"?>
     62<!-- This rspec will reserve one physical node and one openvz node, each
     63     with no OS specified, and create a single dataplane link between
     64     them.  It should work on any Emulab which has nodes available and
     65     supports OpenVZ.  -->
     66<rspec xmlns="http://protogeni.net/resources/rspec/0.2">
     67  <node client_id="phys1" exclusive="true">
     68    <sliver_type name="raw" />
     69    <interface client_id="phys1:if0" />
     70  </node>
     71  <node client_id="virt1" exclusive="false">
     72    <sliver_type name="emulab-openvz" />
     73    <interface client_id="virt1:if0" />
     74  </node>
     75
     76  <link client_id="phys1-virt1-0">
     77    <interface_ref client_id="phys1:if0"/>
     78    <interface_ref client_id="virt1:if0"/>
     79    <property source_id="phys1:if0" dest_id="virt1:if0"/>
     80    <property source_id="virt1:if0" dest_id="phys1:if0"/>
     81  </link>
     82</rspec>
     83}}}
     84   * Make sure i have a long enough slice:
     85{{{
     86omni renewslice ecgtest 2012-05-18  # Hmm, maybe not long enough if i work this evening?
     87omni renewslice ecgtest 2012-05-19
     88}}}
     89   * Now try creating the sliver:
     90{{{
     91jericho,[~],10:03(0)$ omni -a http://www.utah.geniracks.net/protogeni/xmlrpc/am createsliver ecgtest ~/IG-MON-nodes-A.rspec
     92INFO:omni:Loading config file /home/chaos/omni/omni_pgeni
     93INFO:omni:Using control framework pg
     94INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest expires on 2012-05-19 00:00:00 UTC
     95INFO:omni:Creating sliver(s) from rspec file /home/chaos/IG-MON-nodes-A.rspec for slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest
     96ERROR:omni.protogeni:Call for Create Sliver urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest at http://www.utah.geniracks.net/protogeni/xmlrpc/am failed. Server says: <Fault 1: 'Must provide a virtualization_type'>
     97INFO:omni:Asked http://www.utah.geniracks.net/protogeni/xmlrpc/am to reserve resources. Result:
     98INFO:omni:<!-- Reserved resources for:
     99        Slice: ecgtest
     100        At AM:
     101        URL: http://www.utah.geniracks.net/protogeni/xmlrpc/am
     102 -->
     103INFO:omni: ------------------------------------------------------------
     104INFO:omni: Completed createsliver:
     105
     106  Options as run:
     107                aggregate: http://www.utah.geniracks.net/protogeni/xmlrpc/am
     108                configfile: /home/chaos/omni/omni_pgeni
     109                framework: pg
     110                native: True
     111
     112  Args: createsliver ecgtest /home/chaos/IG-MON-nodes-A.rspec
     113
     114  Result Summary: Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest expires on 2012-05-19 00:00:00 UTC
     115Asked http://www.utah.geniracks.net/protogeni/xmlrpc/am to reserve resources. No manifest Rspec returned. <Fault 1: 'Must provide a virtualization_type'>
     116INFO:omni: ============================================================
     117}}}
     118 * Is this just an rspec versioning issue?
     119   * Take 2 rspec:
     120{{{
     121jericho,[~],10:11(0)$ cat IG-MON-nodes-B.rspec
     122<?xml version="1.0" encoding="UTF-8"?>
     123<!-- This rspec will reserve one physical node and one openvz node, each
     124     with no OS specified, and create a single dataplane link between
     125     them.  It should work on any Emulab which has nodes available and
     126     supports OpenVZ.  -->
     127<rspec xmlns="http://www.geni.net/resources/rspec/3"
     128       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     129       xsi:schemaLocation="http://www.geni.net/resources/rspec/3
     130                           http://www.geni.net/resources/rspec/3/request.xsd"
     131       type="request">
     132
     133  <node client_id="phys1" exclusive="true">
     134    <sliver_type name="raw" />
     135    <interface client_id="phys1:if0" />
     136  </node>
     137  <node client_id="virt1" exclusive="false">
     138    <sliver_type name="emulab-openvz" />
     139    <interface client_id="virt1:if0" />
     140  </node>
     141
     142  <link client_id="phys1-virt1-0">
     143    <interface_ref client_id="phys1:if0"/>
     144    <interface_ref client_id="virt1:if0"/>
     145    <property source_id="phys1:if0" dest_id="virt1:if0"/>
     146    <property source_id="virt1:if0" dest_id="phys1:if0"/>
     147  </link>
     148</rspec>
     149}}}
     150   * Try creating the sliver:
     151{{{
     152jericho,[~],10:12(0)$ omni -a http://www.utah.geniracks.net/protogeni/xmlrpc/am createsliver ecgtest ~/IG-MON-nodes-B.rspec
     153INFO:omni:Loading config file /home/chaos/omni/omni_pgeni
     154INFO:omni:Using control framework pg
     155INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest expires on 2012-05-19 00:00:00 UTC
     156INFO:omni:Creating sliver(s) from rspec file /home/chaos/IG-MON-nodes-B.rspec for slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest
     157INFO:omni:Asked http://www.utah.geniracks.net/protogeni/xmlrpc/am to reserve resources. Result:
     158INFO:omni:<?xml version="1.0" ?>
     159INFO:omni:<!-- Reserved resources for:
     160        Slice: ecgtest
     161        At AM:
     162        URL: http://www.utah.geniracks.net/protogeni/xmlrpc/am
     163 -->
     164INFO:omni:<rspec type="manifest" xmlns="http://www.geni.net/resources/rspec/3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.geni.net/resources/rspec/3                            http://www.geni.net/resources/rspec/3/manifest.xsd"> 
     165
     166    <node client_id="phys1" component_id="urn:publicid:IDN+utah.geniracks.net+node+pc3" component_manager_id="urn:publicid:IDN+utah.geniracks.net+authority+cm" exclusive="true" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+366">   
     167        <sliver_type name="raw-pc"/>   
     168        <interface client_id="phys1:if0" component_id="urn:publicid:IDN+utah.geniracks.net+interface+pc3:eth1" mac_address="e83935b14e8a" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+369">      <ip address="10.10.1.1" type="ipv4"/>    </interface>   
     169      <rs:vnode name="pc3" xmlns:rs="http://www.protogeni.net/resources/rspec/ext/emulab/1"/>    <host name="phys1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net"/>    <services>      <login authentication="ssh-keys" hostname="pc3.utah.geniracks.net" port="22" username="chaos"/>    </services>  </node> 
     170    <node client_id="virt1" component_id="urn:publicid:IDN+utah.geniracks.net+node+pc5" component_manager_id="urn:publicid:IDN+utah.geniracks.net+authority+cm" exclusive="false" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+367">   
     171        <sliver_type name="emulab-openvz"/>   
     172        <interface client_id="virt1:if0" component_id="urn:publicid:IDN+utah.geniracks.net+interface+pc5:eth1" mac_address="00000a0a0102" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+370">      <ip address="10.10.1.2" type="ipv4"/>    </interface>   
     173      <rs:vnode name="pcvm5-1" xmlns:rs="http://www.protogeni.net/resources/rspec/ext/emulab/1"/>    <host name="virt1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net"/>    <services>      <login authentication="ssh-keys" hostname="pc5.utah.geniracks.net" port="30010" username="chaos"/>    </services>  </node> 
     174
     175    <link client_id="phys1-virt1-0" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+368" vlantag="259">   
     176        <interface_ref client_id="phys1:if0" component_id="urn:publicid:IDN+utah.geniracks.net+interface+pc3:eth1" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+369"/>   
     177        <interface_ref client_id="virt1:if0" component_id="urn:publicid:IDN+utah.geniracks.net+interface+pc5:eth1" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+370"/>   
     178        <property dest_id="virt1:if0" source_id="phys1:if0"/>   
     179        <property dest_id="phys1:if0" source_id="virt1:if0"/>   
     180    </link> 
     181</rspec>
     182INFO:omni: ------------------------------------------------------------
     183INFO:omni: Completed createsliver:
     184
     185  Options as run:
     186                aggregate: http://www.utah.geniracks.net/protogeni/xmlrpc/am
     187                configfile: /home/chaos/omni/omni_pgeni
     188                framework: pg
     189                native: True
     190
     191  Args: createsliver ecgtest /home/chaos/IG-MON-nodes-B.rspec
     192
     193  Result Summary: Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest expires on 2012-05-19 00:00:00 UTC
     194Reserved resources on http://www.utah.geniracks.net/protogeni/xmlrpc/am. 
     195INFO:omni: ============================================================
     196}}}
     197 * The sliverstatus output is very long, but the relevant information is:
     198   * pc3.utah.geniracks.net has been assigned to be phys1, and it is ready
     199   * virt1 is hosted on pc5.utah.geniracks.net, and my login port is 30010
     200   * When i login to phys1, it has active interfaces:
     201{{{
     202eth0    E8:39:35:B1:4E:88  155.98.34.13/24
     203eth1    E8:39:35:B1:4E:8A  10.10.1.1/24
     204}}}
     205   * When i login to virt1, it has active interfaces:
     206{{{
     207eth999  00:00:AC:11:05:01  172.17.5.1/12
     208mv1.1   82:01:0A:0A:01:02  10.10.1.2/24
     209}}}
     210   * I can ping from phys1 to 10.10.1.2
     211   * I can ping from virt1 to 10.10.1.1
     212
    53213== Step 1: identify network-listening software on the boss node ==
    54214
    … …  
    67227 * The source code and identifiable version (e.g. a git tag) can be found for each non-FreeBSD software source
    68228
     229=== Results of testing: 2012-05-17 ===
     230
     231 * I said `netstat` a bunch in these test definitions, but in fact `sockstat -lL46` is my goto on FreeBSD.  Get a full list of all binaries of processes which are listening on IPv4 or IPv6 sockets from non-localhost addresses:
     232{{{
     233$ for pid in $(sockstat -lL46 | awk '{print $3}' | grep -v PID | sort -u); do procstat -b $pid; done | awk '{print $3}' | sort -u
     234/usr/libexec/sendmail/sendmail
     235/usr/local/bin/python2.6     # running /usr/testbed/sbin/sslxmlrpc_server.py
     236/usr/local/libexec/pubsubd
     237/usr/local/libexec/tftpd
     238/usr/local/sbin/dhcpd
     239/usr/local/sbin/httpd
     240/usr/sbin/inetd
     241/usr/sbin/lwresd             # hard link from /usr/sbin/named
     242/usr/sbin/mountd
     243/usr/sbin/nfsd
     244/usr/sbin/ntpd
     245/usr/sbin/rpcbind
     246/usr/sbin/sshd
     247/usr/sbin/syslogd
     248/usr/testbed/sbin/bootinfo
     249/usr/testbed/sbin/capserver
     250/usr/testbed/sbin/mfrisbeed
     251/usr/testbed/sbin/sdcollectd
     252/usr/testbed/sbin/tmcd
     253PATH
     254}}}
     255 * The following commands are sourced from freebsd packages:
     256{{{
     257$ pkg_info -W /usr/local/bin/python2.6
     258/usr/local/bin/python2.6 was installed by package python26-2.6.6
     259
     260$ pkg_info -W /usr/local/libexec/pubsubd
     261/usr/local/libexec/pubsubd was installed by package pubsub-0.95
     262
     263$ pkg_info -W /usr/local/libexec/tftpd
     264/usr/local/libexec/tftpd was installed by package emulab-tftp-hpa-0.48
     265
     266$ pkg_info -W /usr/local/libexec/tftpd
     267/usr/local/libexec/tftpd was installed by package emulab-tftp-hpa-0.48
     268
     269$ pkg_info -W /usr/local/sbin/dhcpd
     270/usr/local/sbin/dhcpd was installed by package isc-dhcp42-server-4.2.3_1
     271
     272$ pkg_info -W /usr/local/sbin/httpd
     273/usr/local/sbin/httpd was installed by package apache-2.2.21
     274}}}
     275   * So the summary of sourced packages here is:
     276{{{
     277apache-2.2.21
     278emulab-tftp-hpa-0.48
     279isc-dhcp42-server-4.2.3_1
     280pubsub-0.95
     281python26-2.6.6
     282}}}
     283
     284 * The following commands aren't part of packages (`pkg_info -W` reports nothing):
     285{{{
     286/usr/libexec/sendmail/sendmail
     287/usr/sbin/inetd
     288/usr/sbin/lwresd             # hard link from /usr/sbin/named
     289/usr/sbin/mountd
     290/usr/sbin/nfsd
     291/usr/sbin/ntpd
     292/usr/sbin/rpcbind
     293/usr/sbin/sshd
     294/usr/sbin/syslogd
     295/usr/testbed/sbin/bootinfo
     296/usr/testbed/sbin/capserver
     297/usr/testbed/sbin/mfrisbeed
     298/usr/testbed/sbin/sdcollectd
     299/usr/testbed/sbin/sslxmlrpc_server.py
     300/usr/testbed/sbin/tmcd
     301}}}
     302 * The assumption is that these are either part of the FreeBSD base system, or are part of Emulab.  How do we find out which?
     303   * If the OS had been compiled recently, i could look in /usr/obj for binaries which were identical to things on the system.  However, it's currently running the base install (afaict), so that won't work.
     304   * Since Emulab has been compiled recently, i can look in the canonical source of that compile, which i believe is `/users/stoller/testbed/obj`.  Here's a process for checking various things in `/usr/testbed/sbin`, on the suspicion that they are probably Emulab binaries:
     305{{{
     306shortname=bootinfo    # or whatever
     307shortmd5=$(md5 /usr/testbed/sbin/$shortname | awk '{print $4}')
     308for path in $(find . -type f -name $shortname 2> /dev/null); do md5 $path; done | grep $shortmd5
     309}}}
     310 * That finds that the following items are from Emulab:
     311{{{
     312/usr/testbed/sbin/bootinfo:            ./pxe/bootinfo
     313/usr/testbed/sbin/capserver:           ./capture/capserver
     314/usr/testbed/sbin/mfrisbeed:           ./clientside/os/frisbee.redux/mfrisbeed
     315/usr/testbed/sbin/sdcollectd:          ./clientside/sensors/slothd/sdcollectd
     316/usr/testbed/sbin/sslxmlrpc_server.py: ./xmlrpc/sslxmlrpc_server.py
     317/usr/testbed/sbin/tmcd:                ./tmcd/tmcd
     318}}}
     319 * So the following items are not from Emulab, and we assume they would be part of the base install:
     320{{{
     321/usr/libexec/sendmail/sendmail
     322/usr/sbin/inetd
     323/usr/sbin/lwresd             # hard link from /usr/sbin/named
     324/usr/sbin/mountd
     325/usr/sbin/nfsd
     326/usr/sbin/ntpd
     327/usr/sbin/rpcbind
     328/usr/sbin/sshd
     329/usr/sbin/syslogd
     330}}}
     331
     332So, what's needed to be able to finalize this?
     333 1. Ask InstaGENI to provide a reliable way for site admins to find out the .../src and .../obj directories which correspond to the installed software.
     334 2. Ask someone to come up with a suggestion for how to reverse engineer from installed software which is assumed to be part of the FreeBSD base, to the version/source code which was used to generate it.
     335 3. Ask someone where the source for the FreeBSD packages installed on the system, some of which are Emulab-specific, come from.
     336
    69337== Step 2: identify network-listening software on the ops node ==
    70338
    … …  
    131399 * The source code and identifiable version (e.g. a git tag) can be found for each non-Debian software source
    132400
     401=== Results of testing: 2012-05-17 ===
     402
     403 * Here's the netstat invocation to get all the IPv4/IPv6 listeners:
     404{{{
     405control,[~],12:02(0)$ sudo netstat -lnp46
     406Active Internet connections (only servers)
     407Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
     408tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1007/sshd       
     409tcp        0      0 127.0.0.1:6010          0.0.0.0:*               LISTEN      1650/0         
     410tcp        0      0 127.0.0.1:6011          0.0.0.0:*               LISTEN      3936/3         
     411tcp6       0      0 :::22                   :::*                    LISTEN      1007/sshd       
     412tcp6       0      0 ::1:6010                :::*                    LISTEN      1650/0         
     413tcp6       0      0 ::1:6011                :::*                    LISTEN      3936/3         
     414}}}
     415 * Looking up those binaries:
     416{{{
     417control,[~],12:11(0)$ sudo ls -l /proc/{1007,1650,3936}/exe
     418lrwxrwxrwx 1 root root 0 May 10 18:16 /proc/1007/exe -> /usr/sbin/sshd
     419lrwxrwxrwx 1 root root 0 May 17 12:16 /proc/1650/exe -> /usr/sbin/sshd
     420lrwxrwxrwx 1 root root 0 May 17 12:16 /proc/3936/exe -> /usr/sbin/sshd
     421}}}
     422 * So the only thing listening is sshd.  Find out what package sshd is from:
     423{{{
     424control,[~],12:19(1)$ dpkg -S /usr/sbin/sshd
     425openssh-server: /usr/sbin/sshd
     426
     427control,[~],12:19(0)$ dpkg -s openssh-server
     428Package: openssh-server
     429Status: install ok installed
     430Multi-Arch: foreign
     431Priority: optional
     432Section: net
     433Installed-Size: 807
     434Maintainer: Colin Watson <cjwatson@ubuntu.com>
     435Architecture: amd64
     436Source: openssh
     437Version: 1:5.9p1-5ubuntu1
     438...
     439}}}
     440 * Testing download of deb source for this apt-provided package:
     441{{{
     442control,[~],12:30(0)$ mkdir tmp
     443control,[~],12:34(0)$ cd tmp/
     444
     445control,[~/tmp],12:34(0)$ apt-get --download-only source openssh-server=1:5.9p1-5ubuntu1
     446Reading package lists... Done
     447Building dependency tree       
     448Reading state information... Done
     449Picking 'openssh' as source package instead of 'openssh-server'
     450NOTICE: 'openssh' packaging is maintained in the 'Bzr' version control system at:
     451http://anonscm.debian.org/bzr/pkg-ssh/openssh/trunk
     452Please use:
     453bzr branch http://anonscm.debian.org/bzr/pkg-ssh/openssh/trunk
     454to retrieve the latest (possibly unreleased) updates to the package.
     455Need to get 1,363 kB of source archives.
     456Get:1 http://us.archive.ubuntu.com/ubuntu/ precise/main openssh 1:5.9p1-5ubuntu1 (dsc) [2,651 B]
     457Get:2 http://us.archive.ubuntu.com/ubuntu/ precise/main openssh 1:5.9p1-5ubuntu1 (tar) [1,110 kB]
     458Get:3 http://us.archive.ubuntu.com/ubuntu/ precise/main openssh 1:5.9p1-5ubuntu1 (diff) [251 kB]
     459Fetched 1,363 kB in 1s (827 kB/s)
     460Download complete and in download only mode
     461
     462control,[~/tmp],12:34(0)$ ls
     463openssh_5.9p1-5ubuntu1.debian.tar.gz  openssh_5.9p1.orig.tar.gz
     464openssh_5.9p1-5ubuntu1.dsc
     465
     466control,[~/tmp],12:34(0)$ file *
     467openssh_5.9p1-5ubuntu1.debian.tar.gz: gzip compressed data, from Unix, max compression
     468openssh_5.9p1-5ubuntu1.dsc:           ASCII text
     469openssh_5.9p1.orig.tar.gz:            gzip compressed data, extra field, from Unix
     470}}}
     471 The first tarball is the control files and sources patches.  The second tarball is the original OpenSSH source code which was used.
     472
    133473== Step 6: identify network-listening software on an OpenVZ experimental node ==
    134474
    … …  
    147487 * The source code and identifiable version (e.g. a git tag) can be found for each non-RPM software source
    148488
     489=== Results of testing: 2012-05-17 ===
     490
     491''Testing on pc5.utah.geniracks.net, which is operating as an OpenVZ host right now.''
     492
     493 * List listening processes:
     494{{{
     495vhost1,[~],12:47(4)$ sudo netstat -lnp -A inet > netstat.listeners
     496vhost1,[~],12:47(0)$ sudo netstat -lnp -A inet6 >> netstat.listeners
     497
     498vhost1,[~],12:49(0)$ cut -b88- netstat.listeners | awk '{print $1}' | sort -u
     499-
     5001286/rpcbind
     5011301/sshd
     5021410/rpc.statd
     5031551/emulab-syncd
     50417497/sshd
     50517506/pubsubd
     506862/ntpd
     507PID/Program
     508}}}
     509 * Find the binaries for the processes which are identified:
     510{{{
     511vhost1,[~],12:59(0)$ sudo ls -l /proc/{862,1286,1301,1410,1551,17497,17506}/exe
     512lrwxrwxrwx 1 root root 0 May 17 12:59 /proc/1286/exe -> /sbin/rpcbind
     513lrwxrwxrwx 1 root root 0 May 17 12:59 /proc/1301/exe -> /usr/sbin/sshd
     514lrwxrwxrwx 1 root root 0 May 17 12:59 /proc/1410/exe -> /sbin/rpc.statd
     515lrwxrwxrwx 1 root root 0 May 17 12:59 /proc/1551/exe -> /usr/local/etc/emulab/emulab-syncd
     516lrwxrwxrwx 1 root root 0 May 17 12:59 /proc/17497/exe -> /mnt/pcvm5-1/root/usr/sbin/sshd
     517lrwxrwxrwx 1 root root 0 May 17 12:59 /proc/17506/exe -> /mnt/pcvm5-1/root/usr/local/libexec/pubsubd
     518lrwxrwxrwx 1 root root 0 May 16 15:00 /proc/862/exe -> /usr/sbin/ntpd
     519}}}
     520 * Some of those processes are running inside an OpenVZ container, but are identical to binaries in the base system:
     521{{{
     522$ md5sum /mnt/pcvm5-1/root/usr/sbin/sshd /usr/sbin/sshd
     52339aceab46fa9705600dc8b194649bf9c  /mnt/pcvm5-1/root/usr/sbin/sshd
     52439aceab46fa9705600dc8b194649bf9c  /usr/sbin/sshd
     525
     526$ md5sum /mnt/pcvm5-1/root/usr/local/libexec/pubsubd /usr/local/libexec/pubsubd
     527823239d468e277b7c634d34d82c6049a  /mnt/pcvm5-1/root/usr/local/libexec/pubsubd
     528823239d468e277b7c634d34d82c6049a  /usr/local/libexec/pubsubd
     529}}}
     530 * Some of those processes are from RPM packages:
     531   * Determine which RPM each package is from:
     532{{{
     533$ rpm -qf /sbin/rpcbind
     534rpcbind-0.2.0-10.fc15.x86_64
     535
     536$ rpm -qf /usr/sbin/sshd
     537openssh-server-5.6p1-34.fc15.1.x86_64
     538
     539$ rpm -qf /sbin/rpc.statd
     540nfs-utils-1.2.4-1.fc15.x86_64
     541
     542$ rpm -qf /usr/sbin/ntpd
     543ntp-4.2.6p3-4.fc15.x86_64
     544}}}
     545   * Determine which repo each of these packages came from:
     546{{{
     547$ yum list installed rpcbind openssh-server nfs-utils ntp
     548Installed Packages
     549nfs-utils.x86_64                       1:1.2.4-1.fc15                   @updates
     550ntp.x86_64                             4.2.6p3-4.fc15                   @fedora
     551openssh-server.x86_64                  5.6p1-34.fc15.1                  @updates
     552rpcbind.x86_64                         0.2.0-10.fc15                    @fedora
     553}}}
     554   * Look in `/etc/yum.repos.d/*.repo` for information about the locations of RPM files:
     555{{{
     556$ cat /etc/yum.repos.d/fedora.repo
     557[fedora]
     558mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch
     559...
     560
     561$ cat /etc/yum.repos.d/fedora-updates.repo
     562[updates]
     563mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=updates-released-f$releasever&arch=$basearch
     564
     565$ cat /etc/redhat-release
     566Fedora release 15 (Lovelock)
     567
     568$ uname -m
     569x86_64
     570}}}
     571   * Browse to [https://mirrors.fedoraproject.org/]
     572   * Click to [https://mirrors.fedoraproject.org/publiclist/Fedora/15/x86_64/]
     573   * Browse to a reasonable-looking mirror
     574   * Click through to [http://mirrors.kernel.org/fedora/releases/15/Everything/source/SRPMS/]
     575   * Download [http://mirrors.kernel.org/fedora/releases/15/Everything/source/SRPMS/ntp-4.2.6p3-4.fc15.src.rpm]
     576   * Unpack the source file with rpm2cpio:
     577{{{
     578rpm2cpio ntp-4.2.6p3-4.fc15.src.rpm | cpio -idmv
     579}}}
     580   * This yields a bunch of patches and a tarball of the source used.
     581   * The same procedure should work for other things from fedora, and for things from updates (for the latter `s/releases/updates/` when browsing through the tree of available packages).
     582 * Some of those files are not from RPMs:
     583{{{
     584/usr/local/etc/emulab/emulab-syncd
     585/usr/local/libexec/pubsubd
     586}}}
     587 * Netstat also reports some listeners with no processes:
     588{{{
     589Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
     590tcp        0      0 0.0.0.0:58441               0.0.0.0:*                   LISTEN      -                   
     591udp        0      0 0.0.0.0:990                 0.0.0.0:*                               -                   
     592udp        0      0 0.0.0.0:45938               0.0.0.0:*                               -                   
     593tcp        0      0 :::57373                    :::*                        LISTEN      -                   
     594udp        0      0 :::47886                    :::*                                    -                   
     595}}}
     596 However, `lsof -i` does not report these ports, so i am inclined not to worry about them, not understanding what has caused netstat to report them.
     597
     598Unresolved question:
     599 * How do i determine what version of the Emulab source was used to create a particular image?
     600
    149601== Step 7: verify VLANs on the rack management switch ==
    150602
    … …  
    160612 * It is possible to identify and classify every MAC address visible on the control switch
    161613
     614=== Results of testing: 2012-05-17 ===
     615
     616 * Telnet to `procurve1` from boss
     617 * Look at the VLAN mappings for interfaces:
     618{{{
     619ProCurve Switch 2610-24# show running-config
     620...
     621vlan 1
     622   untagged 1-24,26-27
     623   no untagged 25,28
     624
     625vlan 257
     626   untagged 25
     627
     628vlan 260
     629   untagged 28
     630
     631}}}
     632 * Then use `show interfaces brief` to find all up interfaces and their VLANs:
     633{{{
     634ProCurve Switch 2610-24# show interfaces brief
     635
     636 Status and Counters - Port Status
     637
     638                  | Intrusion                           MDI   Flow  Bcast
     639  Port  Type      | Alert     Enabled Status Mode       Mode  Ctrl  Limit
     640  ----- --------- + --------- ------- ------ ---------- ----- ----- ------
     641  1     10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
     642  2     10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
     643  3     10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
     644  4     10/100TX  | No        Yes     Up     100FDx     MDI   off   0     (vlan 1)
     645  5     10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
     646  6     10/100TX  | No        Yes     Up     100FDx     MDI   off   0     (vlan 1)
     647  7     10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
     648  8     10/100TX  | No        Yes     Up     100FDx     MDI   off   0     (vlan 1)
     649  9     10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
     650  10    10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
     651...
     652  21    10/100TX  | No        Yes     Up     100FDx     MDI   off   0     (vlan 1)
     653  22    10/100TX  | No        Yes     Up     100FDx     MDI   off   0     (vlan 1)
     654  23    10/100TX  | No        Yes     Up     100FDx     MDI   off   0     (vlan 1)
     655  24    10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
     656...
     657  26    100/1000T | No        Yes     Up     1000FDx    MDIX  off   0     (vlan 1)
     658...
     659}}}
     660 * As you'd expect, there are no mac addresses on vlans 257 and 260:
     661{{{
     662ProCurve Switch 2610-24# show mac-address vlan 257
     663
     664 Status and Counters - Address Table - VLAN 257
     665
     666  MAC Address   Port
     667  ------------- -----
     668 
     669ProCurve Switch 2610-24# show mac-address vlan 260
     670
     671 Status and Counters - Address Table - VLAN 260
     672
     673  MAC Address   Port
     674  ------------- -----
     675 
     676}}}
     677 * Here are some MACs on vlan 1, reordered by port:
     678{{{
     679ProCurve Switch 2610-24# show mac-address vlan 1
     680
     681 Status and Counters - Address Table - VLAN 1
     682
     683  MAC Address   Port
     684  ------------- -----
     685  e4115b-ed1cb4 1    (pc5[eth0]: per login to pc5)
     686  e83935-ae8586 2    (pc5[iLO]: per iLO information)
     687  e83935-ae35a6 4    (pc2[iLO]: per iLO information)
     688  e83935-aec97c 6    (pc1[iLO]: per iLO information)
     689  e83935-b14e88 7    (pc3[eth0]: per login to pc3)
     690  e83935-ae8b2a 8    (pc3[iLO]: per iLO information)
     691  e83935-ae5566 10   (pc4[iLO]: per iLO information)
     692  00009b-6201df 21   
     693  00009b-6224df 23   
     694  00009b-6224e0 23   
     695  e4115b-eae224 23   (control[eth0]: per login to control)
     696  e4115b-e6580c 24   
     697  0024a8-547c5e 26   
     698  00d0bc-f414f8 26   (155.98.31.1: per control's arp table)
     699}}}
     700 * On reflection, i don't think tracking down the rest of this makes much sense until we have our own rack to look at.  I will ask for a wiring diagram of the Utah rack, but it's probably better to wait for the BBN rack to assess what is plugged into the control network.