wiki:GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-MON-1

Version 27 (modified by chaos@bbn.com, 11 years ago) (diff)

--

Detailed test plan for IG-MON-1: Control Network Software and VLAN Inspection Test

This page is GPO's working page for performing IG-MON-1. It is public for informational purposes, but it is not an official status report. See GENIRacksHome/InstageniRacks/AcceptanceTestStatus for the current status of InstaGENI acceptance tests.

Last substantive edit of this page: 2013-03-12

Page format

  • The status chart summarizes the state of this test
  • The high-level description from test plan contains text copied exactly from the public test plan and acceptance criteria pages.
  • The steps contain things i will actually do/verify:
    • Steps may be composed of related substeps where i find this useful for clarity
    • Each step is either a preparatory step (identified by "(prep)") or a verification step (the default):
      • Preparatory steps are just things we have to do. They're not tests of the rack, but are prerequisites for subsequent verification steps
      • Verification steps are steps in which we will actually look at rack output and make sure it is as expected. They contain a Using: block, which lists the steps to run the verification, and an Expect: block which lists what outcome is expected for the test to pass.

Status of test

Step State Date completed Open Tickets Closed Tickets/Comments
1 Color(green,Pass)? 2013-03-12 (24, 80)
2 Color(green,Pass)? 2012-05-27 (24)
3 Color(green,Pass)? 2012-11-21
4 Color(green,Pass)? 2013-02-28
5 Color(green,Pass)? 2012-05-17
6 Color(green,Pass)? 2013-01-07 (25) verified ability to find Emulab version within Emulab images at BBN
7 Color(#63B8FF,In Progress)? need to map and report on control network topology

High-level description from test plan

This test inspects the state of the rack control network, infrastructure nodes, and system software.

Procedure

  • A site administrator enumerates processes on each of the server host, the boss VM, the ops VM, the FOAM VM, and an experimental node configured for OpenVZ, which listen for network connections from other nodes, identifies what version of what software package is in use for each, and verifies that we know the source of each piece of software and could get access to its source code.
  • A site administrator reviews the configuration of the rack control plane switch and verifies that each experimental node's control and iLO interfaces are on the expected VLANs.
  • A site administrator reviews the MAC address table on the control plane switch, and verifies that all entries are identifiable and expected.

Criteria to verify as part of this test

  • VI.09. A public document explains how to identify the software versions and system file configurations running on the rack, and how to get information about recent changes to the rack software and configuration. (F.5)
  • VI.11. A public document describes the GENI software running on the rack, and explains how to get access to the source code of each piece of GENI software. (F.6)
  • VII.03. Site administrators can understand the expected control and dataplane network behavior of their rack. (F.2)
  • VII.04. Site administrators can view and investigate current system and network activity on their rack. (F.2)
  • VII.06. A site administrator can verify the control software and configurations on the rack at some point in time. (F.5)
  • VII.08. A site administrator can get access to source code for the version of each piece of GENI code installed on their site rack at some point in time. (F.6)
  • VII.09. A site administrator can determine the MAC addresses of all physical host interfaces, all network device interfaces, all active experimental VMs, and all recently-terminated experimental VMs. (C.3.f)
  • VII.10. A site administrator can locate current and recent CPU and memory utilization for each rack network device, and can find recent changes or errors in a log. (D.6.a)
  • VII.12. For each infrastructure and experimental host, a site administrator can locate current and recent uptime, CPU, disk, and memory utilization, interface traffic counters, process counts, and active user counts. (D.6.b)
  • VII.13. A site administrator can locate recent syslogs for all infrastructure and experimental hosts. (D.6.b)

Results of experiment setup on rack: 2012-05-17

Preparation: i wrote these tests on the assumption that there would be some active experiments on the rack while i was testing, and there aren't. So i wanted some running experiments to look at.

  • First attempt (didn't work):
    • Here is my rspec:
      jericho,[~],10:00(0)$ cat ~/IG-MON-nodes-A.rspec 
      <?xml version="1.0" encoding="UTF-8"?>
      <!-- This rspec will reserve one physical node and one openvz node, each
           with no OS specified, and create a single dataplane link between
           them.  It should work on any Emulab which has nodes available and
           supports OpenVZ.  -->
      <rspec xmlns="http://protogeni.net/resources/rspec/0.2">
        <node client_id="phys1" exclusive="true">
          <sliver_type name="raw" />
          <interface client_id="phys1:if0" />
        </node>
        <node client_id="virt1" exclusive="false">
          <sliver_type name="emulab-openvz" />
          <interface client_id="virt1:if0" />
        </node>
      
        <link client_id="phys1-virt1-0">
          <interface_ref client_id="phys1:if0"/>
          <interface_ref client_id="virt1:if0"/>
          <property source_id="phys1:if0" dest_id="virt1:if0"/>
          <property source_id="virt1:if0" dest_id="phys1:if0"/>
        </link>
      </rspec>
      
    • Make sure i have a long enough slice:
      omni renewslice ecgtest 2012-05-18  # Hmm, maybe not long enough if i work this evening?
      omni renewslice ecgtest 2012-05-19
      
    • Now try creating the sliver:
      jericho,[~],10:03(0)$ omni -a http://www.utah.geniracks.net/protogeni/xmlrpc/am createsliver ecgtest ~/IG-MON-nodes-A.rspec
      INFO:omni:Loading config file /home/chaos/omni/omni_pgeni
      INFO:omni:Using control framework pg
      INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest expires on 2012-05-19 00:00:00 UTC
      INFO:omni:Creating sliver(s) from rspec file /home/chaos/IG-MON-nodes-A.rspec for slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest
      ERROR:omni.protogeni:Call for Create Sliver urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest at http://www.utah.geniracks.net/protogeni/xmlrpc/am failed. Server says: <Fault 1: 'Must provide a virtualization_type'>
      INFO:omni:Asked http://www.utah.geniracks.net/protogeni/xmlrpc/am to reserve resources. Result:
      INFO:omni:<!-- Reserved resources for:
              Slice: ecgtest
              At AM:
              URL: http://www.utah.geniracks.net/protogeni/xmlrpc/am
       -->
      INFO:omni: ------------------------------------------------------------
      INFO:omni: Completed createsliver:
      
        Options as run:
                      aggregate: http://www.utah.geniracks.net/protogeni/xmlrpc/am
                      configfile: /home/chaos/omni/omni_pgeni
                      framework: pg
                      native: True
      
        Args: createsliver ecgtest /home/chaos/IG-MON-nodes-A.rspec
      
        Result Summary: Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest expires on 2012-05-19 00:00:00 UTC
      Asked http://www.utah.geniracks.net/protogeni/xmlrpc/am to reserve resources. No manifest Rspec returned. <Fault 1: 'Must provide a virtualization_type'> 
      INFO:omni: ============================================================
      
  • Is this just an rspec versioning issue?
    • Take 2 rspec:
      jericho,[~],10:11(0)$ cat IG-MON-nodes-B.rspec 
      <?xml version="1.0" encoding="UTF-8"?>
      <!-- This rspec will reserve one physical node and one openvz node, each
           with no OS specified, and create a single dataplane link between
           them.  It should work on any Emulab which has nodes available and
           supports OpenVZ.  -->
      <rspec xmlns="http://www.geni.net/resources/rspec/3"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
             xsi:schemaLocation="http://www.geni.net/resources/rspec/3
                                 http://www.geni.net/resources/rspec/3/request.xsd" 
             type="request">
      
        <node client_id="phys1" exclusive="true">
          <sliver_type name="raw" />
          <interface client_id="phys1:if0" />
        </node>
        <node client_id="virt1" exclusive="false">
          <sliver_type name="emulab-openvz" />
          <interface client_id="virt1:if0" />
        </node>
      
        <link client_id="phys1-virt1-0">
          <interface_ref client_id="phys1:if0"/>
          <interface_ref client_id="virt1:if0"/>
          <property source_id="phys1:if0" dest_id="virt1:if0"/>
          <property source_id="virt1:if0" dest_id="phys1:if0"/>
        </link>
      </rspec>
      
    • Try creating the sliver:
      jericho,[~],10:12(0)$ omni -a http://www.utah.geniracks.net/protogeni/xmlrpc/am createsliver ecgtest ~/IG-MON-nodes-B.rspec
      INFO:omni:Loading config file /home/chaos/omni/omni_pgeni
      INFO:omni:Using control framework pg
      INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest expires on 2012-05-19 00:00:00 UTC
      INFO:omni:Creating sliver(s) from rspec file /home/chaos/IG-MON-nodes-B.rspec for slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest
      INFO:omni:Asked http://www.utah.geniracks.net/protogeni/xmlrpc/am to reserve resources. Result:
      INFO:omni:<?xml version="1.0" ?>
      INFO:omni:<!-- Reserved resources for:
              Slice: ecgtest
              At AM:
              URL: http://www.utah.geniracks.net/protogeni/xmlrpc/am
       -->
      INFO:omni:<rspec type="manifest" xmlns="http://www.geni.net/resources/rspec/3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.geni.net/resources/rspec/3                            http://www.geni.net/resources/rspec/3/manifest.xsd">  
      
          <node client_id="phys1" component_id="urn:publicid:IDN+utah.geniracks.net+node+pc3" component_manager_id="urn:publicid:IDN+utah.geniracks.net+authority+cm" exclusive="true" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+366">    
              <sliver_type name="raw-pc"/>    
              <interface client_id="phys1:if0" component_id="urn:publicid:IDN+utah.geniracks.net+interface+pc3:eth1" mac_address="e83935b14e8a" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+369">      <ip address="10.10.1.1" type="ipv4"/>    </interface>    
            <rs:vnode name="pc3" xmlns:rs="http://www.protogeni.net/resources/rspec/ext/emulab/1"/>    <host name="phys1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net"/>    <services>      <login authentication="ssh-keys" hostname="pc3.utah.geniracks.net" port="22" username="chaos"/>    </services>  </node>  
          <node client_id="virt1" component_id="urn:publicid:IDN+utah.geniracks.net+node+pc5" component_manager_id="urn:publicid:IDN+utah.geniracks.net+authority+cm" exclusive="false" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+367">    
              <sliver_type name="emulab-openvz"/>    
              <interface client_id="virt1:if0" component_id="urn:publicid:IDN+utah.geniracks.net+interface+pc5:eth1" mac_address="00000a0a0102" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+370">      <ip address="10.10.1.2" type="ipv4"/>    </interface>    
            <rs:vnode name="pcvm5-1" xmlns:rs="http://www.protogeni.net/resources/rspec/ext/emulab/1"/>    <host name="virt1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net"/>    <services>      <login authentication="ssh-keys" hostname="pc5.utah.geniracks.net" port="30010" username="chaos"/>    </services>  </node>  
      
          <link client_id="phys1-virt1-0" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+368" vlantag="259">    
              <interface_ref client_id="phys1:if0" component_id="urn:publicid:IDN+utah.geniracks.net+interface+pc3:eth1" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+369"/>    
              <interface_ref client_id="virt1:if0" component_id="urn:publicid:IDN+utah.geniracks.net+interface+pc5:eth1" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+370"/>    
              <property dest_id="virt1:if0" source_id="phys1:if0"/>    
              <property dest_id="phys1:if0" source_id="virt1:if0"/>    
          </link>  
      </rspec>
      INFO:omni: ------------------------------------------------------------
      INFO:omni: Completed createsliver:
      
        Options as run:
                      aggregate: http://www.utah.geniracks.net/protogeni/xmlrpc/am
                      configfile: /home/chaos/omni/omni_pgeni
                      framework: pg
                      native: True
      
        Args: createsliver ecgtest /home/chaos/IG-MON-nodes-B.rspec
      
        Result Summary: Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest expires on 2012-05-19 00:00:00 UTC
      Reserved resources on http://www.utah.geniracks.net/protogeni/xmlrpc/am.  
      INFO:omni: ============================================================
      
  • The sliverstatus output is very long, but the relevant information is:
    • pc3.utah.geniracks.net has been assigned to be phys1, and it is ready
    • virt1 is hosted on pc5.utah.geniracks.net, and my login port is 30010
    • When i login to phys1, it has active interfaces:
      eth0    E8:39:35:B1:4E:88  155.98.34.13/24
      eth1    E8:39:35:B1:4E:8A  10.10.1.1/24
      
    • When i login to virt1, it has active interfaces:
      eth999  00:00:AC:11:05:01  172.17.5.1/12
      mv1.1   82:01:0A:0A:01:02  10.10.1.2/24
      
    • I can ping from phys1 to 10.10.1.2
    • I can ping from virt1 to 10.10.1.1

Step 1: identify network-listening software on the boss node

Using:

  • Using netstat, enumerate processes on boss which listen for network connections from outside the node
  • For each process found:
    • Use the command-line to determine what executable file is running
    • Use pkg_info to determine whether the executable file is part of a FreeBSD package/port
    • Otherwise, use documentation or iterate with the InstaGENI team to determine the origin of the software
  • For each FreeBSD package found, identify a location from which the port source can be obtained
  • For each non-FreeBSD software source found, identify a location from which the source code for that version can be obtained.

Verify:

  • The source of each network-listening file can be identified
  • FreeBSD ports can be identified for each FreeBSD-sourced package
  • The source code and identifiable version (e.g. a git tag) can be found for each non-FreeBSD software source

Results of testing step 1: 2012-05-17

  • I said netstat a bunch in these test definitions, but in fact sockstat -lL46 is my goto on FreeBSD. Get a full list of all binaries of processes which are listening on IPv4 or IPv6 sockets from non-localhost addresses:
    $ for pid in $(sockstat -lL46 | awk '{print $3}' | grep -v PID | sort -u); do procstat -b $pid; done | awk '{print $3}' | sort -u
    /usr/libexec/sendmail/sendmail
    /usr/local/bin/python2.6     # running /usr/testbed/sbin/sslxmlrpc_server.py
    /usr/local/libexec/pubsubd
    /usr/local/libexec/tftpd
    /usr/local/sbin/dhcpd
    /usr/local/sbin/httpd
    /usr/sbin/inetd
    /usr/sbin/lwresd             # hard link from /usr/sbin/named
    /usr/sbin/mountd
    /usr/sbin/nfsd
    /usr/sbin/ntpd
    /usr/sbin/rpcbind
    /usr/sbin/sshd
    /usr/sbin/syslogd
    /usr/testbed/sbin/bootinfo
    /usr/testbed/sbin/capserver
    /usr/testbed/sbin/mfrisbeed
    /usr/testbed/sbin/sdcollectd
    /usr/testbed/sbin/tmcd
    PATH
    
  • The following commands are sourced from freebsd packages:
    $ pkg_info -W /usr/local/bin/python2.6
    /usr/local/bin/python2.6 was installed by package python26-2.6.6
    
    $ pkg_info -W /usr/local/libexec/pubsubd
    /usr/local/libexec/pubsubd was installed by package pubsub-0.95
    
    $ pkg_info -W /usr/local/libexec/tftpd
    /usr/local/libexec/tftpd was installed by package emulab-tftp-hpa-0.48
    
    $ pkg_info -W /usr/local/libexec/tftpd
    /usr/local/libexec/tftpd was installed by package emulab-tftp-hpa-0.48
    
    $ pkg_info -W /usr/local/sbin/dhcpd
    /usr/local/sbin/dhcpd was installed by package isc-dhcp42-server-4.2.3_1
    
    $ pkg_info -W /usr/local/sbin/httpd
    /usr/local/sbin/httpd was installed by package apache-2.2.21
    
    • So the summary of sourced packages here is:
      apache-2.2.21
      emulab-tftp-hpa-0.48
      isc-dhcp42-server-4.2.3_1
      pubsub-0.95
      python26-2.6.6
      
  • The following commands aren't part of packages (pkg_info -W reports nothing):
    /usr/libexec/sendmail/sendmail
    /usr/sbin/inetd
    /usr/sbin/lwresd             # hard link from /usr/sbin/named
    /usr/sbin/mountd
    /usr/sbin/nfsd
    /usr/sbin/ntpd
    /usr/sbin/rpcbind
    /usr/sbin/sshd
    /usr/sbin/syslogd
    /usr/testbed/sbin/bootinfo
    /usr/testbed/sbin/capserver
    /usr/testbed/sbin/mfrisbeed
    /usr/testbed/sbin/sdcollectd
    /usr/testbed/sbin/sslxmlrpc_server.py
    /usr/testbed/sbin/tmcd
    
  • The assumption is that these are either part of the FreeBSD base system, or are part of Emulab. How do we find out which?
    • If the OS had been compiled recently, i could look in /usr/obj for binaries which were identical to things on the system. However, it's currently running the base install (afaict), so that won't work.
    • Since Emulab has been compiled recently, i can look in the canonical source of that compile, which i believe is /users/stoller/testbed/obj. Here's a process for checking various things in /usr/testbed/sbin, on the suspicion that they are probably Emulab binaries:
      shortname=bootinfo    # or whatever
      shortmd5=$(md5 /usr/testbed/sbin/$shortname | awk '{print $4}')
      for path in $(find . -type f -name $shortname 2> /dev/null); do md5 $path; done | grep $shortmd5
      
  • That finds that the following items are from Emulab:
    /usr/testbed/sbin/bootinfo:            ./pxe/bootinfo
    /usr/testbed/sbin/capserver:           ./capture/capserver
    /usr/testbed/sbin/mfrisbeed:           ./clientside/os/frisbee.redux/mfrisbeed
    /usr/testbed/sbin/sdcollectd:          ./clientside/sensors/slothd/sdcollectd
    /usr/testbed/sbin/sslxmlrpc_server.py: ./xmlrpc/sslxmlrpc_server.py
    /usr/testbed/sbin/tmcd:                ./tmcd/tmcd
    
  • So the following items are not from Emulab, and we assume they would be part of the base install:
    /usr/libexec/sendmail/sendmail
    /usr/sbin/inetd
    /usr/sbin/lwresd             # hard link from /usr/sbin/named
    /usr/sbin/mountd
    /usr/sbin/nfsd
    /usr/sbin/ntpd
    /usr/sbin/rpcbind
    /usr/sbin/sshd
    /usr/sbin/syslogd
    

So, what's needed to be able to finalize this?

  1. Ask InstaGENI to provide a reliable way for site admins to find out the .../src and .../obj directories which correspond to the installed software.
  2. Ask someone to come up with a suggestion for how to reverse engineer from installed software which is assumed to be part of the FreeBSD base, to the version/source code which was used to generate it.
  3. Ask someone where the source for the FreeBSD packages installed on the system, some of which are Emulab-specific, come from.

Results of testing step 1: 2012-05-27

  • In fact sockstat -lL46 is my goto on FreeBSD. Get a full list of all binaries of processes which are listening on IPv4 or IPv6 sockets from non-localhost addresses:
    boss,[~],09:05(0)$ for pid in $(sockstat -lL46 | awk '{print $3}' | grep -v PID | sort -u); do procstat -b $pid; done | awk '{print $3}' | sort -u
    procstat: sysctl: kern.proc.pathname: 1340: No such file or directory
    procstat: sysctl: kern.proc.pathname: 1341: No such file or directory
    procstat: sysctl: kern.proc.pathname: 1342: No such file or directory
    procstat: sysctl: kern.proc.pathname: 1343: No such file or directory
    procstat: sysctl: kern.proc.pathname: 1344: No such file or directory
    procstat: sysctl: kern.proc.pathname: 1345: No such file or directory
    procstat: sysctl: kern.proc.pathname: 1346: No such file or directory
    procstat: sysctl: kern.proc.pathname: 1347: No such file or directory
    procstat: sysctl: kern.proc.pathname: 1348: No such file or directory
    procstat: sysctl: kern.proc.pathname: 1349: No such file or directory
    procstat: sysctl: kern.proc.pathname: 1350: No such file or directory
    procstat: sysctl: kern.proc.pathname: 1351: No such file or directory
    procstat: sysctl: kern.proc.pathname: 1352: No such file or directory
    procstat: sysctl: kern.proc.pathname: 1353: No such file or directory
    /usr/libexec/sendmail/sendmail
    /usr/local/bin/python2.6
    /usr/local/libexec/pubsubd
    /usr/local/libexec/tftpd
    /usr/local/sbin/dhcpd
    /usr/local/sbin/httpd
    /usr/sbin/inetd
    /usr/sbin/lwresd
    /usr/sbin/mountd
    /usr/sbin/nfsd
    /usr/sbin/ntpd
    /usr/sbin/rpcbind
    /usr/sbin/sshd
    /usr/sbin/syslogd
    /usr/testbed/sbin/bootinfo
    /usr/testbed/sbin/mfrisbeed
    /usr/testbed/sbin/sdcollectd
    PATH
    
  • Huh, well, that's weird. All the PIDs procstat choked on seem to be tmcd and its children. So we can call the list:
    /usr/testbed/sbin/tmcd
    /usr/libexec/sendmail/sendmail
    /usr/local/bin/python2.6
    /usr/local/libexec/pubsubd
    /usr/local/libexec/tftpd
    /usr/local/sbin/dhcpd
    /usr/local/sbin/httpd
    /usr/sbin/inetd
    /usr/sbin/lwresd
    /usr/sbin/mountd
    /usr/sbin/nfsd
    /usr/sbin/ntpd
    /usr/sbin/rpcbind
    /usr/sbin/sshd
    /usr/sbin/syslogd
    /usr/testbed/sbin/bootinfo
    /usr/testbed/sbin/mfrisbeed
    /usr/testbed/sbin/sdcollectd
    
  • The following commands are sourced from freebsd packages:
    $ pkg_info -W /usr/local/bin/python2.6
    /usr/local/bin/python2.6 was installed by package python26-2.6.6
    
    $ pkg_info -W /usr/local/libexec/pubsubd
    /usr/local/libexec/pubsubd was installed by package pubsub-0.95
    
    $ pkg_info -W /usr/local/libexec/tftpd
    /usr/local/libexec/tftpd was installed by package emulab-tftp-hpa-0.48
    
    $ pkg_info -W /usr/local/sbin/dhcpd
    /usr/local/sbin/dhcpd was installed by package isc-dhcp42-server-4.2.3_1
    
    $ pkg_info -W /usr/local/sbin/httpd
    /usr/local/sbin/httpd was installed by package apache-2.2.21
    
    • So the summary of sourced packages here is:
      apache-2.2.21
      emulab-tftp-hpa-0.48
      isc-dhcp42-server-4.2.3_1
      pubsub-0.95
      python26-2.6.6
      
    • For each of these, i can verify using the distinfo that /usr/ports seems to have the same version as is installed:
      $ cat /usr/ports/www/apache22/distinfo 
      SHA256 (apache22/httpd-2.2.21.tar.bz2) = 18d5591fe48cfbac44fc20316036ffe17456df60bc3a2aaad238d56c6445577f
      SIZE (apache22/httpd-2.2.21.tar.bz2) = 5324905
      
      $ cat /usr/ports/misc/emulab-tftp-hpa/distinfo 
      MD5 (tftp-hpa-0.48.tar.gz) = 73dd29dfa9062a174163f6a267e0b5f7
      SHA256 (tftp-hpa-0.48.tar.gz) = 0bba7dc69a0174b8a9d5b340fa79c9c4874c61a880fe1cd8129e7644dd0967bb
      SIZE (tftp-hpa-0.48.tar.gz) = 104844
      
      $ cat /usr/ports/net/isc-dhcp42-server/distinfo 
      SHA256 (dhcp-4.2.3-P1.tar.gz) = 4860625e07c290768d513b9a2260e655ae4bad4cc5c4c8f5eaaeb4f9cbfa96af
      SIZE (dhcp-4.2.3-P1.tar.gz) = 9525670
      
      $ cat /usr/ports/misc/emulab-pubsub/distinfo 
      MD5 (pubsub-0.9.tar.gz) = 25216f20821bb7094a1853b1771cb5b8
      MD5 (pubsub-0.95.tar.gz) = 9e012e65ebb4979e99f686e5f8fa8104
      SHA256 (pubsub-0.95.tar.gz) = 7b8eb4f495a531454b16abb8e8399e031fa17a31e09c4ce47486cedaa3ae4410
      
      $ cat /usr/ports/lang/python26/distinfo 
      MD5 (python/Python-2.6.6.tgz) = b2f209df270a33315e62c1ffac1937f0
      SHA256 (python/Python-2.6.6.tgz) = 372f66db46d773214e4619df1794a26449158f626138d4d2141a64c2f017fae1
      SIZE (python/Python-2.6.6.tgz) = 13318547
      
  • The following commands aren't part of packages (pkg_info -W reports nothing):
    /usr/testbed/sbin/tmcd
    /usr/testbed/sbin/bootinfo
    /usr/testbed/sbin/mfrisbeed
    /usr/testbed/sbin/sdcollectd
    /usr/libexec/sendmail/sendmail
    /usr/sbin/inetd
    /usr/sbin/lwresd             # hard link from /usr/sbin/named
    /usr/sbin/mountd
    /usr/sbin/nfsd
    /usr/sbin/ntpd
    /usr/sbin/rpcbind
    /usr/sbin/sshd
    /usr/sbin/syslogd
    
  • The assumption is that these are either part of the FreeBSD base system, or are part of Emulab. How do we find out which?
    • Since Emulab has been compiled recently, i can look in the canonical source of that compile, which i believe is /users/stoller/testbed/obj. Here's a process for checking various things in /usr/testbed/sbin, on the suspicion that they are probably Emulab binaries:
      shortname=bootinfo    # or whatever
      shortmd5=$(md5 /usr/testbed/sbin/$shortname | awk '{print $4}')
      for path in $(find . -type f -name $shortname 2> /dev/null); do md5 $path; done | grep $shortmd5
      
    • That finds that the following items are indeed from Emulab:
      /usr/testbed/sbin/bootinfo:            ./pxe/bootinfo
      /usr/testbed/sbin/mfrisbeed:           ./clientside/os/frisbee.redux/mfrisbeed
      /usr/testbed/sbin/sdcollectd:          ./clientside/sensors/slothd/sdcollectd
      /usr/testbed/sbin/tmcd:                ./tmcd/tmcd
      
    • Note that, in instaticket:24, we found that on a non-development rack, source and object files will be found in /usr/testbed/{src,obj}/emulab-stable.
  • So the following items are not from Emulab, and we assume they would be part of the base install:
    /usr/libexec/sendmail/sendmail
    /usr/sbin/inetd
    /usr/sbin/lwresd             # hard link from /usr/sbin/named
    /usr/sbin/mountd
    /usr/sbin/nfsd
    /usr/sbin/ntpd
    /usr/sbin/rpcbind
    /usr/sbin/sshd
    /usr/sbin/syslogd
    
    • If the OS had been compiled recently, i could look in /usr/obj for binaries which were identical to things on the system. Since it's running the base install, find out what version that is:
      $ uname -r
      8.3-RC1
      
    • Hmm, so it looks like 8.3-RC1 is not exactly the version which became 8.3-RELEASE. I did a bit of poking around to find the i386 install disc for 8.3-RC1, eventually locating a copy at http://ftp.at.freebsd.org/pub/FreeBSD/ISO-IMAGES-i386/8.3/FreeBSD-8.3-RC1-i386-disc1.iso
    • Loopback-mounted that on my desktop:
      sudo mount -o loop FreeBSD-8.3-RC1-i386-disc1.iso /mnt/iso
      mkdir -p ~/tmp/8.3-RC1/{base,src}
      cd ~/tmp/8.3-RC1/base
      cat /mnt/iso/8.3-RC1/base/base.?? | tar xvzf -
      cd ~/tmp/8.3-RC1/src
      dists="base bin cddl contrib crypto etc games gnu include krb5 lib libexec release rescue sbin secure share sys tools ubin usbin"
      for dist in $dists; do
        cat /mnt/iso/8.3-RC1/src/s${dist}.?? | tar xvzf -
      done
      
    • So, this isn't perfect, in that at a glance, the object checksums don't match between what is on boss and what i found in this iso. However:
      • It definitely would tell someone which items are part of the freebsd base distribution (if i find ~/tmp/8.3-RC1/base/usr/sbin/inetd, then boss:/usr/sbin/inetd has been verified to come from FreeBSD)
      • I can look at the source code from which the object was presumably taken, even if i can't verify that the object is unmodified.
    • Using this, i verified that all the non-Emulab-sourced, non-packaged listeners on boss are indeed from the FreeBSD base distribution.

All that remains is to retest on the BBN rack once it arrives, and verify that the Emulab source distribution lands in a reasonable place.

Results of testing step 1: 2013-01-07

Testing on boss.instageni.gpolab.bbn.com:

  • sockstat -lL46 is my goto on FreeBSD. Get a full list of all binaries of processes which are listening on IPv4 or IPv6 sockets from non-localhost addresses:
    boss,[~],17:07(0)$ for pid in $(sockstat -lL46 | awk '{print $3}' | grep -v PID | sort -u); do procstat -b $pid; done | awk '{print $3}' | sort -u
    usage: procstat [-h] [-C] [-M core] [-N system] [-w interval] 
                    [-b | -c | -e | -f | -i | -j | -k | -l | -s | -t | -v | -x] [-a | pid ...]
    /usr/libexec/sendmail/sendmail
    /usr/local/bin/perl
    /usr/local/bin/python2.7
    /usr/local/libexec/pubsubd
    /usr/local/libexec/tftpd
    /usr/local/sbin/dhcpd
    /usr/local/sbin/httpd
    /usr/sbin/inetd
    /usr/sbin/lwresd
    /usr/sbin/mountd
    /usr/sbin/nfsd
    /usr/sbin/ntpd
    /usr/sbin/rpcbind
    /usr/sbin/sshd
    /usr/sbin/syslogd
    /usr/testbed/sbin/bootinfo
    /usr/testbed/sbin/mfrisbeed
    /usr/testbed/sbin/sdcollectd
    /usr/testbed/sbin/tmcd
    PATH
    
  • The errors were a couple of pids which get reported as ?:
    boss,[~],17:14(0)$ sockstat -lL46
    USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS      
    ...
    ?        ?          ?     ?  udp4   *:2049                *:*
    ?        ?          ?     ?  udp6   *:2049                *:*
    
  • Hmm, that's pretty strange, since nfsd is also represented:
    boss,[~],17:18(0)$ sockstat -lL46 | grep 2049
    root     nfsd       1441  5  tcp4   *:2049                *:*
    root     nfsd       1441  6  tcp6   *:2049                *:*
    ?        ?          ?     ?  udp4   *:2049                *:*
    ?        ?          ?     ?  udp6   *:2049                *:*
    
    The Utah rack shows the same thing, though, so i'm going to guess it's normal, and just has to do with the strange kernel dependencies of NFS.
  • So, the list of listeners is:
    /usr/libexec/sendmail/sendmail
    /usr/local/bin/perl
    /usr/local/bin/python2.7
    /usr/local/libexec/pubsubd
    /usr/local/libexec/tftpd
    /usr/local/sbin/dhcpd
    /usr/local/sbin/httpd
    /usr/sbin/inetd
    /usr/sbin/lwresd
    /usr/sbin/mountd
    /usr/sbin/nfsd
    /usr/sbin/ntpd
    /usr/sbin/rpcbind
    /usr/sbin/sshd
    /usr/sbin/syslogd
    /usr/testbed/sbin/bootinfo
    /usr/testbed/sbin/mfrisbeed
    /usr/testbed/sbin/sdcollectd
    /usr/testbed/sbin/tmcd
    
  • The following commands are sourced from freebsd packages:
    $ pkg_info -W /usr/local/bin/perl
    /usr/local/bin/perl was installed by package perl-5.12.4_3
    
    $ pkg_info -W /usr/local/bin/python2.7
    /usr/local/bin/python2.7 was installed by package python27-2.7.2_4
    
    $ pkg_info -W /usr/local/libexec/pubsubd
    /usr/local/libexec/pubsubd was installed by package pubsub-0.95
    
    $ pkg_info -W /usr/local/libexec/tftpd
    /usr/local/libexec/tftpd was installed by package emulab-tftp-hpa-0.48
    
    $ pkg_info -W /usr/local/sbin/dhcpd
    /usr/local/sbin/dhcpd was installed by package isc-dhcp42-server-4.2.3_2
    
    $ pkg_info -W /usr/local/sbin/httpd
    /usr/local/sbin/httpd was installed by package apache-2.2.22_5
    
    • So the summary of sourced packages here is:
      apache-2.2.22_5
      emulab-tftp-hpa-0.48
      isc-dhcp42-server-4.2.3_2
      perl-5.12.4_3
      pubsub-0.95
      python27-2.7.2_4
      
    • For each of these, i can verify using the distinfo that /usr/ports seems to have the same version as is installed:
      $ cat /usr/ports/www/apache22/distinfo 
      SHA256 (apache22/httpd-2.2.22.tar.bz2) = dcdc9f1dc722f84798caf69d69dca78daa5e09a4269060045aeca7e4f44cb231
      SIZE (apache22/httpd-2.2.22.tar.bz2) = 5378934
      
      $ cat /usr/ports/misc/emulab-tftp-hpa/distinfo 
      MD5 (tftp-hpa-0.48.tar.gz) = 73dd29dfa9062a174163f6a267e0b5f7
      SHA256 (tftp-hpa-0.48.tar.gz) = 0bba7dc69a0174b8a9d5b340fa79c9c4874c61a880fe1cd8129e7644dd0967bb
      SIZE (tftp-hpa-0.48.tar.gz) = 104844
      
      $ cat /usr/ports/net/isc-dhcp42-server/distinfo 
      SHA256 (dhcp-4.2.3-P2.tar.gz) = 5cf7ae2cad9c4ca0103748b2476ec8ea78484e408f8fe597e4e0a4afb051b469
      SIZE (dhcp-4.2.3-P2.tar.gz) = 9527546
      
      $ cat /usr/ports/lang/perl5.12/distinfo 
      SHA256 (perl/perl-5.12.4.tar.bz2) = c920fa01a34b7752e8cbbc99c6921b908df60611d84a29c4377e4a798604767e
      SIZE (perl/perl-5.12.4.tar.bz2) = 12350353
      SHA256 (perl/BSDPAN-20111107.tar.bz2) = 2f03218a592dc65ebfdc3c6b9394d91dcf4c53aa5290a08458b837baad5a21f9
      SIZE (perl/BSDPAN-20111107.tar.bz2) = 8448
      
      $ cat /usr/ports/misc/emulab-pubsub/distinfo 
      MD5 (pubsub-0.9.tar.gz) = 25216f20821bb7094a1853b1771cb5b8
      MD5 (pubsub-0.95.tar.gz) = 9e012e65ebb4979e99f686e5f8fa8104
      SHA256 (pubsub-0.95.tar.gz) = 7b8eb4f495a531454b16abb8e8399e031fa17a31e09c4ce47486cedaa3ae4410
      
      $ cat /usr/ports/lang/python27/distinfo 
      SHA256 (python/Python-2.7.2.tgz) = 1d54b7096c17902c3f40ffce7e5b84e0072d0144024184fff184a84d563abbb3
      SIZE (python/Python-2.7.2.tgz) = 14091337
      
  • The following commands aren't part of packages (pkg_info -W reports nothing):
    /usr/libexec/sendmail/sendmail
    /usr/sbin/inetd
    /usr/sbin/lwresd
    /usr/sbin/mountd
    /usr/sbin/nfsd
    /usr/sbin/ntpd
    /usr/sbin/rpcbind
    /usr/sbin/sshd
    /usr/sbin/syslogd
    /usr/testbed/sbin/bootinfo
    /usr/testbed/sbin/mfrisbeed
    /usr/testbed/sbin/sdcollectd
    /usr/testbed/sbin/tmcd
    

}}}

  • The assumption is that these are either part of the FreeBSD base system, or are part of Emulab. How do we find out which?
    • I can look in the canonical source of the emulab compile, which should be /usr/testbed/obj/testbed. Here's a process for checking various things in /usr/testbed/sbin, on the suspicion that they are probably Emulab binaries:
      cd /usr/testbed/obj/testbed
      shortname=bootinfo    # or whatever
      shortmd5=$(md5 /usr/testbed/sbin/$shortname | awk '{print $4}')
      for path in $(find . -type f -name $shortname 2> /dev/null); do md5 $path; done | grep $shortmd5
      
    • Hmm, the first thing i tried that on, bootinfo, did not match:
      boss,[~],15:32(0)$ md5 /usr/testbed/sbin/bootinfo /usr/testbed/obj/testbed/pxe/bootinfo
      MD5 (/usr/testbed/sbin/bootinfo) = 131c06c10ca3b7438c85aafa93ccc93b
      MD5 (/usr/testbed/obj/testbed/pxe/bootinfo) = a98b23f4598f9c2d268b614cbdbc8e30
      
      boss,[~],15:32(0)$ ls -l /usr/testbed/sbin/bootinfo /usr/testbed/obj/testbed/pxe/
      bootinfo
      -rwxr-xr-x  1 root  wheel    136499 Mar  8  2012 /usr/testbed/obj/testbed/pxe/bootinfo*
      -rwxr-xr-x  1 root  tbadmin  140415 Oct 29 16:39 /usr/testbed/sbin/bootinfo*
      
      Note that, 24 claimed that, on a non-development rack, source and object files will be found in /usr/testbed/{src,obj}/emulab-stable.
    • I'm not going to bother retesting the things from the FreeBSD base install --- as mentioned when i tested at Utah, the MD5 sums don't match, so i'd just be using the FreeBSD base install ISO to verify the identities of the binaries. Since the listed binaries are the same this time as last time, i'm going to assume they are still reflected in the FreeBSD base distro.

Results of testing step 1: 2013-02-28

80 covers the question of the locations of the Emulab src/obj trees on the BBN boss node: it turns out i was looking at the wrong boss node for much of the previous test, so i'm just going to redo it entirely.

Testing on boss.instageni.gpolab.bbn.com:

  • sockstat -lL46 is my goto on FreeBSD. Get a full list of all binaries of processes which are listening on IPv4 or IPv6 sockets from non-localhost addresses:
    boss.instageni.gpolab.bbn.com,[~],07:57(0)$ for pid in $(sockstat -lL46 | awk '{print $3}' | grep -v PID | sort -u); do procstat -b $pid; done | awk '{print $3}' | sort -u
    usage: procstat [-h] [-C] [-M core] [-N system] [-w interval] 
                    [-b | -c | -e | -f | -i | -j | -k | -l | -s | -t | -v | -x] [-a | pid ...]
    /usr/libexec/sendmail/sendmail
    /usr/local/bin/perl
    /usr/local/bin/python2.7
    /usr/local/libexec/pubsubd
    /usr/local/libexec/tftpd
    /usr/local/sbin/dhcpd
    /usr/local/sbin/httpd
    /usr/sbin/inetd
    /usr/sbin/lwresd
    /usr/sbin/mountd
    /usr/sbin/nfsd
    /usr/sbin/ntpd
    /usr/sbin/rpcbind
    /usr/sbin/sshd
    /usr/sbin/syslogd
    /usr/testbed/sbin/bootinfo
    /usr/testbed/sbin/mfrisbeed
    /usr/testbed/sbin/sdcollectd
    /usr/testbed/sbin/tmcd
    PATH
    
  • The errors were a couple of pids which get reported as ?:
    boss.instageni.gpolab.bbn.com,[~],07:57(0)$ sockstat -lL46
    USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS      
    ...
    ?        ?          ?     ?  udp4   *:2049                *:*
    ?        ?          ?     ?  udp6   *:2049                *:*
    
    In the previous test, i concluded that this is normal nfsd weirdness and no cause for alarm.
  • So, the list of listeners is:
    /usr/libexec/sendmail/sendmail
    /usr/local/bin/perl
    /usr/local/bin/python2.7
    /usr/local/libexec/pubsubd
    /usr/local/libexec/tftpd
    /usr/local/sbin/dhcpd
    /usr/local/sbin/httpd
    /usr/sbin/inetd
    /usr/sbin/lwresd
    /usr/sbin/mountd
    /usr/sbin/nfsd
    /usr/sbin/ntpd
    /usr/sbin/rpcbind
    /usr/sbin/sshd
    /usr/sbin/syslogd
    /usr/testbed/sbin/bootinfo
    /usr/testbed/sbin/mfrisbeed
    /usr/testbed/sbin/sdcollectd
    /usr/testbed/sbin/tmcd
    
  • The following commands are sourced from freebsd packages:
    boss.instageni.gpolab.bbn.com,[~],08:00(0)$ pkg_info -W /usr/local/bin/perl
    /usr/local/bin/perl was installed by package perl-5.12.4_3
    
    boss.instageni.gpolab.bbn.com,[~],08:00(0)$ pkg_info -W /usr/local/bin/python2.7
    /usr/local/bin/python2.7 was installed by package python27-2.7.2_4
    
    boss.instageni.gpolab.bbn.com,[~],08:00(0)$ pkg_info -W /usr/local/libexec/pubsubd
    /usr/local/libexec/pubsubd was installed by package pubsub-0.95
    
    boss.instageni.gpolab.bbn.com,[~],08:01(0)$ pkg_info -W /usr/local/libexec/tftpd
    /usr/local/libexec/tftpd was installed by package emulab-tftp-hpa-0.48
    
    boss.instageni.gpolab.bbn.com,[~],08:01(0)$ pkg_info -W /usr/local/sbin/dhcpd
    /usr/local/sbin/dhcpd was installed by package isc-dhcp42-server-4.2.3_2
    
    boss.instageni.gpolab.bbn.com,[~],08:01(0)$ pkg_info -W /usr/local/sbin/httpd
    /usr/local/sbin/httpd was installed by package apache-2.2.22_5
    
    • So the summary of sourced packages here is:
      apache-2.2.22_5
      emulab-tftp-hpa-0.48
      isc-dhcp42-server-4.2.3_2
      perl-5.12.4_3
      pubsub-0.95
      python27-2.7.2_4
      
    • Not bothering to verify the distinfo from /usr/ports again, because i did it before, and the versions haven't changed
  • The following commands aren't part of packages (pkg_info -W reports nothing):
    /usr/libexec/sendmail/sendmail
    /usr/sbin/inetd
    /usr/sbin/lwresd
    /usr/sbin/mountd
    /usr/sbin/nfsd
    /usr/sbin/ntpd
    /usr/sbin/rpcbind
    /usr/sbin/sshd
    /usr/sbin/syslogd
    /usr/testbed/sbin/bootinfo
    /usr/testbed/sbin/mfrisbeed
    /usr/testbed/sbin/sdcollectd
    /usr/testbed/sbin/tmcd
    
  • The assumption is that these are either part of the FreeBSD base system, or are part of Emulab. How do we find out which?
    • I can look in the canonical source of the emulab compile, which should be /usr/testbed/obj/testbed. Here's a process for checking various things in /usr/testbed/sbin, on the suspicion that they are probably Emulab binaries:
      cd /usr/testbed/obj/testbed
      shortname=bootinfo    # or whatever
      shortmd5=$(md5 /usr/testbed/sbin/$shortname | awk '{print $4}')
      for path in $(find . -type f -name $shortname 2> /dev/null); do md5 $path; done | grep $shortmd5
      
    • Hmm, the first thing i tried that on, bootinfo, did not match, and i'm sure i'm on the right boss now:
      boss.instageni.gpolab.bbn.com,[~],08:07(0)$ md5 /usr/testbed/sbin/bootinfo /usr/testbed/obj/testbed/pxe/bootinfo
      MD5 (/usr/testbed/sbin/bootinfo) = 880f20f3320ccdeed81b685756cfb8cd
      MD5 (/usr/testbed/obj/testbed/pxe/bootinfo) = 1c651f32132f80888b5ef8896b042e12
      
      boss.instageni.gpolab.bbn.com,[~],08:08(0)$ ls -l /usr/testbed/sbin/bootinfo /usr/testbed/obj/testbed/pxe/bootinfo
      -rwxr-xr-x  1 root  wheel    140471 Dec 11 15:57 /usr/testbed/obj/testbed/pxe/bootinfo*
      -rwxr-xr-x  1 root  tbadmin  141046 Feb 26 11:10 /usr/testbed/sbin/bootinfo*
      
    • Ditto mfrisbeed:
      boss.instageni.gpolab.bbn.com,[~],08:09(0)$ md5 /usr/testbed/sbin/mfrisbeed /usr
      /testbed/obj/testbed/clientside/os/frisbee.redux/mfrisbeed
      MD5 (/usr/testbed/sbin/mfrisbeed) = 0d90a44d74e4c3520835de15a1743d01
      MD5 (/usr/testbed/obj/testbed/clientside/os/frisbee.redux/mfrisbeed) = 71125d6e351a78f8a930873cd203cf0b
      
      boss.instageni.gpolab.bbn.com,[~],08:09(0)$ ls -l /usr/testbed/sbin/mfrisbeed /usr/testbed/obj/testbed/clientside/os/frisbee.redux/mfrisbeed
      -rwxr-xr-x  1 root  wheel    80952 Dec 11 15:57 /usr/testbed/obj/testbed/clientside/os/frisbee.redux/mfrisbeed*
      -rwxr-xr-x  1 root  tbadmin  80952 Feb 15 18:01 /usr/testbed/sbin/mfrisbeed*
      
    • And the src files on this boss are old:
      boss.instageni.gpolab.bbn.com,[~],08:10(0)$ ls -l /usr/testbed/src/
      total 12
      drwxr-xr-x  54 root  wheel  2048 Dec 11 15:50 emulab-devel/
      drwxr-xr-x  53 root  wheel  2048 Dec 11 15:51 emulab-stable/
      drwxr-xr-x  54 282   6000   2560 Dec 11 15:54 testbed/
      

24 claimed that, on a non-development rack, source and object files will be found in /usr/testbed/{src,obj}/emulab-stable.

Picking this up via e-mail.

Results of testing step 1: 2013-03-01

Reusing the data from yesterday's testing, just to check Leigh's fix.

  • The following commands aren't part of packages (pkg_info -W reports nothing):
    /usr/libexec/sendmail/sendmail
    /usr/sbin/inetd
    /usr/sbin/lwresd
    /usr/sbin/mountd
    /usr/sbin/nfsd
    /usr/sbin/ntpd
    /usr/sbin/rpcbind
    /usr/sbin/sshd
    /usr/sbin/syslogd
    /usr/testbed/sbin/bootinfo
    /usr/testbed/sbin/mfrisbeed
    /usr/testbed/sbin/sdcollectd
    /usr/testbed/sbin/tmcd
    
  • The assumption is that these are either part of the FreeBSD base system, or are part of Emulab. How do we find out which?
    • I can look in the canonical source of the emulab compile, which should be /usr/testbed/obj/testbed. Here's a process for checking various things in /usr/testbed/sbin, on the suspicion that they are probably Emulab binaries:
      cd /usr/testbed/obj/testbed
      shortname=bootinfo    # or whatever
      shortmd5=$(md5 /usr/testbed/sbin/$shortname | awk '{print $4}')
      for path in $(find . -type f -name $shortname 2> /dev/null); do md5 $path; done | grep $shortmd5
      
    • Okay, so... the md5sums in obj still don't match:
      boss.instageni.gpolab.bbn.com,[/usr/testbed/src/testbed],14:25(0)$ md5 /usr/testbed/sbin/bootinfo /usr/testbed/obj/testbed/pxe/bootinfo
      MD5 (/usr/testbed/sbin/bootinfo) = 880f20f3320ccdeed81b685756cfb8cd
      MD5 (/usr/testbed/obj/testbed/pxe/bootinfo) = ce19b79f88e556a4ea643802f682274c
      
      boss.instageni.gpolab.bbn.com,[/usr/testbed/src/testbed],14:25(0)$ ls -l /usr/testbed/sbin/bootinfo /usr/testbed/obj/testbed/pxe/bootinfo
      -rwxrwxr-x  1 elabman  tbadmin  141246 Feb 12 18:01 /usr/testbed/obj/testbed/pxe/bootinfo*
      -rwxr-xr-x  1 root     tbadmin  141046 Feb 26 11:10 /usr/testbed/sbin/bootinfo*
      
      I asked about that via e-mail.
    • How about source:
      boss.instageni.gpolab.bbn.com,[/usr/testbed/src/testbed],15:01(0)$ git log | head -3
      commit 8f0a15bb370c42b64469a6d03e00c05945d64e03
      Author: Leigh B Stoller <stoller@flux.utah.edu>
      Date:   Tue Feb 26 07:43:29 2013 -0700
      
    • Compare to what the GENI AM API thinks the code tag is:
      omni -a https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am getversion
      INFO:omni:Loading config file /home/chaos/omni/omni_pgeni
      INFO:omni:Using control framework pg
      INFO:omni:AM URN: unspecified_AM_URN (url: https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am) has version:
      INFO:omni:{   'code': {   'am_code': 0,
                      'am_type': 'protogeni',
                      'geni_code': 0,
                      'protogeni_error_log': 'urn:publicid:IDN+instageni.gpolab.bbn.com+log+efb965bba3d6d8beb3d08b143895863b',
                      'protogeni_error_url': 'https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=efb965bba3d6d8beb3d08b143895863b'},
          'geni_api': 2,
          'output': '',
          'value': {   'code_tag': '8f0a15bb370c42b64469a6d03e00c05945d64e03',
      ...
      
    • And, as a sanity-check, the date looks good relative to the timestamps of live files:
      boss.instageni.gpolab.bbn.com,[/usr/testbed/src/testbed],15:02(0)$ ls -ld .git
      drwxrwxr-x  9 elabman  tbadmin  512 Feb 26 09:46 .git/
      
      boss.instageni.gpolab.bbn.com,[~],15:03(0)$ ls -lart /usr/testbed/sbin/ | tail
      lrwxr-xr-x   1 root  tbadmin      28 Feb 26 11:10 addmmuser@ -> /usr/testbed/libexec/runsuid
      lrwxr-xr-x   1 root  tbadmin      28 Feb 26 11:10 addmmlist@ -> /usr/testbed/libexec/runsuid
      lrwxr-xr-x   1 root  tbadmin      28 Feb 26 11:10 addjabberuser@ -> /usr/testbed/libexec/runsuid
      lrwxr-xr-x   1 root  tbadmin      28 Feb 26 11:10 addjabberchat@ -> /usr/testbed/libexec/runsuid
      drwxr-xr-x   2 root  tbadmin    1024 Feb 26 11:10 protogeni/
      lrwxr-xr-x   1 root  tbadmin      28 Feb 26 11:10 plabslicewrapper@ -> /usr/testbed/libexec/runsuid
      lrwxr-xr-x   1 root  tbadmin      28 Feb 26 11:10 plabnodewrapper@ -> /usr/testbed/libexec/runsuid
      lrwxr-xr-x   1 root  tbadmin      28 Feb 26 11:10 gencabundle@ -> /usr/testbed/libexec/runsuid
      lrwxr-xr-x   1 root  tbadmin      28 Feb 26 11:10 cleanupslice@ -> /usr/testbed/libexec/runsuid
      drwxrwxr-x   4 root  tbadmin    5120 Feb 26 11:10 ./
      

So, src definitely looks good, just asking Utah about obj as a sanity-check.

Results of testing step 1: 2013-03-12

Here goes again --- again, easiest to redo from scratch.

Testing on boss.instageni.gpolab.bbn.com:

  • sockstat -lL46 is my goto on FreeBSD. Get a full list of all binaries of processes which are listening on IPv4 or IPv6 sockets from non-localhost addresses:
    boss.instageni.gpolab.bbn.com,[~],14:24(0)$ for pid in $(sockstat -lL46 | awk '{print $3}' | grep -v PID | sort -u); do procstat -b $pid; done | awk '{print $3}' | sort -u
    usage: procstat [-h] [-C] [-M core] [-N system] [-w interval] 
                    [-b | -c | -e | -f | -i | -j | -k | -l | -s | -t | -v | -x] [-a | pid ...]
    /usr/libexec/sendmail/sendmail
    /usr/local/bin/perl
    /usr/local/bin/python2.7
    /usr/local/libexec/pubsubd
    /usr/local/libexec/tftpd
    /usr/local/sbin/dhcpd
    /usr/local/sbin/httpd
    /usr/sbin/inetd
    /usr/sbin/lwresd
    /usr/sbin/mountd
    /usr/sbin/nfsd
    /usr/sbin/ntpd
    /usr/sbin/rpcbind
    /usr/sbin/sshd
    /usr/sbin/syslogd
    /usr/testbed/sbin/bootinfo
    /usr/testbed/sbin/mfrisbeed
    /usr/testbed/sbin/sdcollectd
    /usr/testbed/sbin/tmcd
    PATH
    
  • The errors were a couple of pids which get reported as ?:
    boss.instageni.gpolab.bbn.com,[~],07:57(0)$ sockstat -lL46
    USER     COMMAND    PID   FD PROTO  LOCAL ADDRESS         FOREIGN ADDRESS      
    ...
    ?        ?          ?     ?  udp4   *:2049                *:*
    ?        ?          ?     ?  udp6   *:2049                *:*
    
    In the previous test, i concluded that this is normal nfsd weirdness and no cause for alarm.
  • So, the list of listeners is still:
    /usr/libexec/sendmail/sendmail
    /usr/local/bin/perl
    /usr/local/bin/python2.7
    /usr/local/libexec/pubsubd
    /usr/local/libexec/tftpd
    /usr/local/sbin/dhcpd
    /usr/local/sbin/httpd
    /usr/sbin/inetd
    /usr/sbin/lwresd
    /usr/sbin/mountd
    /usr/sbin/nfsd
    /usr/sbin/ntpd
    /usr/sbin/rpcbind
    /usr/sbin/sshd
    /usr/sbin/syslogd
    /usr/testbed/sbin/bootinfo
    /usr/testbed/sbin/mfrisbeed
    /usr/testbed/sbin/sdcollectd
    /usr/testbed/sbin/tmcd
    
  • The following commands are sourced from freebsd packages:
    boss.instageni.gpolab.bbn.com,[~],14:28(0)$ pkg_info -W /usr/local/bin/perl
    /usr/local/bin/perl was installed by package perl-5.12.4_3
    boss.instageni.gpolab.bbn.com,[~],14:29(0)$ pkg_info -W /usr/local/bin/python2.7/usr/local/bin/python2.7 was installed by package python27-2.7.2_4
    boss.instageni.gpolab.bbn.com,[~],14:29(0)$ pkg_info -W /usr/local/libexec/pubsubd
    /usr/local/libexec/pubsubd was installed by package pubsub-0.95
    boss.instageni.gpolab.bbn.com,[~],14:29(0)$ pkg_info -W /usr/local/libexec/tftpd/usr/local/libexec/tftpd was installed by package emulab-tftp-hpa-0.48
    boss.instageni.gpolab.bbn.com,[~],14:29(0)$ pkg_info -W /usr/local/sbin/dhcpd
    /usr/local/sbin/dhcpd was installed by package isc-dhcp42-server-4.2.3_2
    boss.instageni.gpolab.bbn.com,[~],14:29(0)$ pkg_info -W /usr/local/sbin/httpd
    /usr/local/sbin/httpd was installed by package apache-2.2.22_5
    
    • So the summary of sourced packages here is:
      apache-2.2.22_5
      emulab-tftp-hpa-0.48
      isc-dhcp42-server-4.2.3_2
      perl-5.12.4_3
      pubsub-0.95
      python27-2.7.2_4
      
    • Not bothering to verify the distinfo from /usr/ports again, because i did it before, and the versions haven't changed
  • The following commands aren't part of packages (pkg_info -W reports nothing):
    /usr/libexec/sendmail/sendmail
    /usr/sbin/inetd
    /usr/sbin/lwresd
    /usr/sbin/mountd
    /usr/sbin/nfsd
    /usr/sbin/ntpd
    /usr/sbin/rpcbind
    /usr/sbin/sshd
    /usr/sbin/syslogd
    /usr/testbed/sbin/bootinfo
    /usr/testbed/sbin/mfrisbeed
    /usr/testbed/sbin/sdcollectd
    /usr/testbed/sbin/tmcd
    
  • The assumption is that these are either part of the FreeBSD base system, or are part of Emulab. How do we find out which?
    • I can look in the canonical source of the emulab compile, which should be /usr/testbed/obj/testbed. Here's a process for checking various things in /usr/testbed/sbin, on the suspicion that they are probably Emulab binaries:
      cd /usr/testbed/obj/testbed
      shortname=bootinfo    # or whatever
      shortmd5=$(md5 /usr/testbed/sbin/$shortname | awk '{print $4}')
      for path in $(find . -type f -name $shortname 2> /dev/null); do md5 $path; done | grep $shortmd5
      
    • This time, i found matches for:
      MD5 (./clientside/os/frisbee.redux/mfrisbeed) = 0d90a44d74e4c3520835de15a1743d01
      MD5 (./clientside/sensors/slothd/sdcollectd) = 11891c5aeb0734a88ce6c153186daaf4
      MD5 (./tmcd/tmcd) = ba4eba54de73ea90a1ab20be2bcd1ad5
      
    • Still not bootinfo, but that's an unsurprising consequence of the previous snowflakiness --- bootinfo hasn't been recompiled since the source was synced up properly:
      -rwxr-xr-x  1 root  tbadmin  141046 Feb 26 11:10 /usr/testbed/sbin/bootinfo*
      
      So i don't expect this to be an issue for fresh rack installs.
    • The src files on this boss now date from the approximate time of the last install:
      boss.instageni.gpolab.bbn.com,[/usr/testbed/src/testbed],14:33(0)$ git log | head -3
      commit 7e584af01b968c18b7d87fe7e78a5dde3549c47f
      Author: Leigh B Stoller <stoller@flux.utah.edu>
      Date:   Fri Mar 8 12:22:38 2013 -0700
      
      boss.instageni.gpolab.bbn.com,[/usr/testbed/src/testbed],14:33(0)$ ls -lart /usr
      /testbed/sbin/ | tail
      -rwxr-xr-x   1 root  tbadmin    7658 Mar  8 19:39 load-descriptors*
      -rwxr-xr-x   1 root  tbadmin    4767 Mar  8 19:39 jabbersetup*
      lrwxr-xr-x   1 root  tbadmin      28 Mar  8 19:39 deljabberuser@ -> /usr/testbed/libexec/runsuid
      lrwxr-xr-x   1 root  tbadmin      28 Mar  8 19:39 deljabberchat@ -> /usr/testbed/libexec/runsuid
      lrwxr-xr-x   1 root  tbadmin      28 Mar  8 19:39 addjabberuser@ -> /usr/testbed/libexec/runsuid
      lrwxr-xr-x   1 root  tbadmin      28 Mar  8 19:39 addjabberchat@ -> /usr/testbed/libexec/runsuid
      drwxr-xr-x   2 root  tbadmin    1024 Mar  8 19:39 protogeni/
      lrwxr-xr-x   1 root  tbadmin      28 Mar  8 19:39 gencabundle@ -> /usr/testbed/libexec/runsuid
      lrwxr-xr-x   1 root  tbadmin      28 Mar  8 19:39 cleanupslice@ -> /usr/testbed/libexec/runsuid
      drwxrwxr-x   4 root  tbadmin    5120 Mar  8 19:39 ./
      

Step 2: identify network-listening software on the ops node

Using:

  • Using netstat, enumerate processes on ops which listen for network connections from outside the node
  • For each process found:
    • Use the command-line to determine what executable file is running
    • Use pkg_info to determine whether the executable file is part of a FreeBSD package/port
    • Otherwise, use documentation or iterate with the InstaGENI team to determine the origin of the software
  • For each FreeBSD package found, identify a location from which the port source can be obtained
  • For each non-FreeBSD software source found, identify a location from which the source code for that version can be obtained.

Verify:

  • The source of each network-listening file can be identified
  • FreeBSD ports can be identified for each FreeBSD-sourced package
  • The source code and identifiable version (e.g. a git tag) can be found for each non-FreeBSD software source

Results of testing step 2: 2012-05-27

  • In fact sockstat -lL46 is my goto on FreeBSD. Get a full list of all binaries of processes which are listening on IPv4 or IPv6 sockets from non-localhost addresses:
    ops,[~],09:33(0)$ for pid in $(sockstat -lL46 | awk '{print $3}' | grep -v PID | sort -u); do procstat -b $pid; done | awk '{print $3}' | sort -u
    /usr/libexec/sendmail/sendmail
    /usr/local/libexec/mysqld
    /usr/local/libexec/pubsubd
    /usr/local/sbin/httpd
    /usr/sbin/mountd
    /usr/sbin/nfsd
    /usr/sbin/ntpd
    /usr/sbin/rpcbind
    /usr/sbin/sshd
    /usr/sbin/syslogd
    PATH
    
  • The following commands are sourced from freebsd packages:
    $ pkg_info -W /usr/local/libexec/mysqld
    /usr/local/libexec/mysqld was installed by package mysql-server-5.0.92
    
    $ pkg_info -W /usr/local/libexec/pubsubd
    /usr/local/libexec/pubsubd was installed by package pubsub-0.95
    
    $ pkg_info -W /usr/local/sbin/httpd
    /usr/local/sbin/httpd was installed by package apache-2.2.21
    
    • So the summary of sourced packages here is:
      apache-2.2.21
      pubsub-0.95
      mysql-server-5.0.92
      
    • There is no /usr/ports on ops, so i will assume that all needed information can be found in /usr/ports on boss. Given that, we already checked apache and pubsub, so the only remaining item is mysql-server:
      $ cat /usr/ports/databases/mysql50-server/distinfo 
      SHA256 (mysql-5.0.92.tar.gz) = 223530c4cd6d45e2906e6c7d33615c5824816dfdb772d104f4165a483ea78a55
      SIZE (mysql-5.0.92.tar.gz) = 22440590
      
  • The following commands aren't part of packages (pkg_info -W reports nothing):
    /usr/libexec/sendmail/sendmail
    /usr/sbin/mountd
    /usr/sbin/nfsd
    /usr/sbin/ntpd
    /usr/sbin/rpcbind
    /usr/sbin/sshd
    /usr/sbin/syslogd
    
  • The assumption is that these are either part of the FreeBSD base system, or are part of Emulab. I'll assume at a guess that all of these are part of the base system, which is the same release and architecture as boss:
    ops,[~],09:35(0)$ uname -r
    8.3-RC1
    
    ops,[~],09:38(0)$ uname -m
    i386
    
  • Using the FreeBSD base distribution downloaded in step 1, i verified that those commands all appear in the FreeBSD 8.3-RC1 base distribution.

Step 3: identify network-listening software on the foam node

Using:

  • Using netstat, enumerate processes on foam which listen for network connections from outside the node
  • For each process found:
    • Use the command-line to determine what executable file is running
    • Use dpkg commands to determine whether the executable file is part of a Debian package
    • Otherwise, use documentation or iterate with the InstaGENI team to determine the origin of the software
  • For each Debian package found, identify a location from which the source package can be obtained
  • For each non-Debian package found, identify a location from which the source code for that version can be obtained.

Verify:

  • The source of each network-listening file can be identified
  • Source code or a source package can be identified for each Debian-sourced package
  • The source code and identifiable version (e.g. a git tag) can be found for each non-Debian software source

Results of testing step 3: 2012-11-21

  • Here's the netstat invocation to get all the IPv4/IPv6 listeners:
    foam,[~],07:56(0)$ sudo netstat -lnp46
    Active Internet connections (only servers)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
    tcp        0      0 0.0.0.0:3626            0.0.0.0:*               LISTEN      31388/nginx     
    tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      31388/nginx     
    tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      471/sshd        
    tcp6       0      0 :::22                   :::*                    LISTEN      471/sshd        
    udp        0      0 10.4.1.2:123            0.0.0.0:*                           17929/ntpd      
    udp        0      0 155.98.34.6:123         0.0.0.0:*                           17929/ntpd      
    udp        0      0 127.0.0.1:123           0.0.0.0:*                           17929/ntpd      
    udp        0      0 0.0.0.0:123             0.0.0.0:*                           17929/ntpd      
    udp6       0      0 fe80::216:3eff:fef0:123 :::*                                17929/ntpd      
    udp6       0      0 ::1:123                 :::*                                17929/ntpd      
    udp6       0      0 fe80::216:3eff:fef0:123 :::*                                17929/ntpd      
    udp6       0      0 :::123                  :::*                                17929/ntpd      
    
  • Looking up the full paths for those listeners:
    foam,[~],07:57(0)$ sudo ls -l /proc/{31388,471,17929}/exe
    lrwxrwxrwx 1 root root 0 Nov 21 07:59 /proc/17929/exe -> /usr/sbin/ntpd
    lrwxrwxrwx 1 root root 0 Nov 21 07:59 /proc/31388/exe -> /usr/sbin/nginx
    lrwxrwxrwx 1 root root 0 May 31 16:17 /proc/471/exe -> /usr/sbin/sshd
    
  • Find out what package each listener is from:
    foam,[~],07:59(0)$ dpkg -S /usr/sbin/sshd
    openssh-server: /usr/sbin/sshd
    foam,[~],08:00(0)$ dpkg -S /usr/sbin/ntpd
    ntp: /usr/sbin/ntpd
    foam,[~],08:01(0)$ dpkg -S /usr/sbin/nginx
    nginx-full: /usr/sbin/nginx
    
  • Get information about each package:
    foam,[~],08:03(0)$ dpkg -s openssh-server ntp nginx-full | egrep "^Package:|^Status:|^Maintainer:|^Version:|^$"
    Package: openssh-server
    Status: install ok installed
    Maintainer: Colin Watson <cjwatson@ubuntu.com>
    Version: 1:5.9p1-5ubuntu1
    
    Package: ntp
    Status: install ok installed
    Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
    Version: 1:4.2.6.p3+dfsg-1ubuntu3
    
    Package: nginx-full
    Status: install ok installed
    Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
    Version: 1.1.19-1
    
  • Download source packages and briefly verify that something was downloaded:
    foam,[~],08:04(0)$ mkdir tmp
    foam,[~],08:04(0)$ cd tmp/
    
    foam,[~/tmp],08:04(0)$ apt-get --download-only source openssh-server=1:5.9p1-5ubuntu1
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    Picking 'openssh' as source package instead of 'openssh-server'
    NOTICE: 'openssh' packaging is maintained in the 'Bzr' version control system at:
    http://anonscm.debian.org/bzr/pkg-ssh/openssh/trunk
    Please use:
    bzr branch http://anonscm.debian.org/bzr/pkg-ssh/openssh/trunk
    to retrieve the latest (possibly unreleased) updates to the package.
    Need to get 1,363 kB of source archives.
    Get:1 http://us.archive.ubuntu.com/ubuntu/ precise/main openssh 1:5.9p1-5ubuntu1 (dsc) [2,651 B]
    Get:2 http://us.archive.ubuntu.com/ubuntu/ precise/main openssh 1:5.9p1-5ubuntu1 (tar) [1,110 kB]
    Get:3 http://us.archive.ubuntu.com/ubuntu/ precise/main openssh 1:5.9p1-5ubuntu1 (diff) [251 kB]
    Fetched 1,363 kB in 0s (1,551 kB/s)
    Download complete and in download only mode
    
    foam,[~/tmp],08:04(0)$ apt-get --download-only source ntp=1:4.2.6.p3+dfsg-1ubuntu3
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    
    NOTICE: 'ntp' packaging is maintained in the 'Svn' version control system at:
    svn://svn.debian.org/pkg-ntp/ntp/trunk/
    Need to get 4,745 kB of source archives.
    Get:1 http://us.archive.ubuntu.com/ubuntu/ precise/main ntp 1:4.2.6.p3+dfsg-1ubuntu3 (dsc) [2,344 B]
    Get:2 http://us.archive.ubuntu.com/ubuntu/ precise/main ntp 1:4.2.6.p3+dfsg-1ubuntu3 (tar) [4,177 kB]
    Get:3 http://us.archive.ubuntu.com/ubuntu/ precise/main ntp 1:4.2.6.p3+dfsg-1ubuntu3 (diff) [565 kB]
    Fetched 4,745 kB in 1s (2,990 kB/s)
    Download complete and in download only mode
    
    foam,[~/tmp],08:05(0)$ apt-get --download-only source nginx-full=1.1.19-1
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    Picking 'nginx' as source package instead of 'nginx-full'
    NOTICE: 'nginx' packaging is maintained in the 'Svn' version control system at:
    svn://svn.debian.org/svn/collab-maint/deb-maint/nginx/trunk
    Need to get 1,353 kB of source archives.
    Get:1 http://us.archive.ubuntu.com/ubuntu/ precise/universe nginx 1.1.19-1 (dsc) [2,041 B]
    Get:2 http://us.archive.ubuntu.com/ubuntu/ precise/universe nginx 1.1.19-1 (tar) [716 kB]
    0% [2 nginx 0 B/716 kB 0%]
    Get:3 http://us.archive.ubuntu.com/ubuntu/ precise/universe nginx 1.1.19-1 (diff) [635 kB]
    Fetched 1,353 kB in 0s (1,602 kB/s)
    Download complete and in download only mode
    
    foam,[~/tmp],08:05(0)$ ls -l
    total 7304
    -rw-r--r-- 1 chaos chaos  634913 Apr 14  2012 nginx_1.1.19-1.debian.tar.gz
    -rw-r--r-- 1 chaos chaos    2041 Apr 14  2012 nginx_1.1.19-1.dsc
    -rw-r--r-- 1 chaos chaos  715741 Apr 14  2012 nginx_1.1.19.orig.tar.gz
    -rw-r--r-- 1 chaos chaos  565496 Mar  6  2012 ntp_4.2.6.p3+dfsg-1ubuntu3.debian.tar.gz
    -rw-r--r-- 1 chaos chaos    2344 Mar  6  2012 ntp_4.2.6.p3+dfsg-1ubuntu3.dsc
    -rw-r--r-- 1 chaos chaos 4176683 Oct 26  2011 ntp_4.2.6.p3+dfsg.orig.tar.gz
    -rw-r--r-- 1 chaos chaos  250819 Apr  2  2012 openssh_5.9p1-5ubuntu1.debian.tar.gz
    -rw-r--r-- 1 chaos chaos    2651 Apr  2  2012 openssh_5.9p1-5ubuntu1.dsc
    -rw-r--r-- 1 chaos chaos 1110014 Oct 17  2011 openssh_5.9p1.orig.tar.gz
    

Step 4: identify network-listening software on the FlowVisor node

Using:

  • Using netstat, enumerate processes on flowvisor which listen for network connections from outside the node
  • For each process found:
    • Use the command-line to determine what executable file is running
    • Use dpkg commands to determine whether the executable file is part of a Debian package
    • Otherwise, use documentation or iterate with the InstaGENI team to determine the origin of the software
  • For each Debian package found, identify a location from which the source package can be obtained
  • For each non-Debian package found, identify a location from which the source code for that version can be obtained.

Verify:

  • The source of each network-listening file can be identified
  • Source code or a source package can be identified for each Debian-sourced package
  • The source code and identifiable version (e.g. a git tag) can be found for each non-Debian software source

Results of testing step 4: 2012-11-21

  • Here's the netstat invocation to get all the IPv4/IPv6 listeners:
    flowvisor,[~],08:08(0)$ sudo netstat -lnp46
    Active Internet connections (only servers)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
    tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      27632/sshd      
    tcp6       0      0 :::6633                 :::*                    LISTEN      4527/java       
    tcp6       0      0 :::8080                 :::*                    LISTEN      4527/java       
    tcp6       0      0 :::8081                 :::*                    LISTEN      4527/java       
    tcp6       0      0 :::22                   :::*                    LISTEN      27632/sshd      
    udp6       0      0 :::59945                :::*                                4527/java       
    
  • Looking up the full paths for those listeners:
    flowvisor,[~],08:08(0)$ sudo ls -l /proc/{27632,4527}/exe
    lrwxrwxrwx 1 root      root      0 Nov  8 10:44 /proc/27632/exe -> /usr/sbin/sshd
    lrwxrwxrwx 1 flowvisor flowvisor 0 Oct 11 15:10 /proc/4527/exe -> /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java
    
  • Get package names:
    flowvisor,[~],08:15(0)$ dpkg -S /usr/sbin/sshd /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java
    openssh-server: /usr/sbin/sshd
    openjdk-6-jre-headless: /usr/lib/jvm/java-6-openjdk-amd64/jre/bin/java
    
  • We also want to know what program java is running:
    flowvisor,[~],08:12(0)$ ps -ef | grep 4527
    1000      4527  4526  0 Oct11 ?        03:48:01 java -server -Xms100M -Xmx2000M -XX:OnError=flowvisor-crash-logger -XX:+UseConcMarkSweepGC -Dorg.flowvisor.config_dir=/usr/local/etc/flowvisor -Dorg.flowvisor.install_dir=/usr/local/libexec/flowvisor -Dderby.system.home=/usr/local/share/db/flowvisor -Dfvlog.configuration=/usr/local/etc/flowvisor/fvlog.config -Dorg.flowvisor.config_dir=/usr/local/etc/flowvisor -Dorg.flowvisor.install_dir=/usr/local/libexec/flowvisor -Dderby.system.home=/usr/local/share/db/flowvisor -Dfvlog.configuration=/usr/local/etc/flowvisor/fvlog.config -Djavax.net.ssl.keyStore=/usr/local/etc/flowvisor/mySSLKeyStore -Djavax.net.ssl.keyStorePassword=CHANGEME_PASSWD -cp /usr/local/libexec/flowvisor/openflow.jar:/usr/local/libexec/flowvisor/xmlrpc-client-3.1.3.jar:/usr/local/libexec/flowvisor/xmlrpc-common-3.1.3.jar:/usr/local/libexec/flowvisor/xmlrpc-server-3.1.3.jar:/usr/local/libexec/flowvisor/commons-logging-1.1.jar:/usr/local/libexec/flowvisor/ws-commons-util-1.0.2.jar:/usr/local/libexec/flowvisor/jsse.jar:/usr/local/libexec/flowvisor/asm-3.0.jar:/usr/local/libexec/flowvisor/cglib-2.2.jar:/usr/local/libexec/flowvisor/commons-codec-1.4.jar:/usr/local/libexec/flowvisor/commons-collections-3.2.1.jar:/usr/local/libexec/flowvisor/commons-dbcp-1.4.jar:/usr/local/libexec/flowvisor/commons-pool-1.5.6.jar:/usr/local/libexec/flowvisor/gson-2.0.jar:/usr/local/libexec/flowvisor/jetty-continuation-7.0.2.v20100331.jar:/usr/local/libexec/flowvisor/jetty-http-7.0.2.v20100331.jar:/usr/local/libexec/flowvisor/jetty-io-7.0.2.v20100331.jar:/usr/local/libexec/flowvisor/jetty-security-7.0.2.v20100331.jar:/usr/local/libexec/flowvisor/jetty-server-7.0.2.v20100331.jar:/usr/local/libexec/flowvisor/jetty-util-7.0.2.v20100331.jar:/usr/local/libexec/flowvisor/servlet-api-2.5.jar:/usr/local/libexec/flowvisor/derby.jar:/usr/local/libexec/flowvisor/derbytools.jar:/usr/local/libexec/flowvisor/jna.jar:/usr/local/libexec/flowvisor/syslog4j-0.9.46-bin.jar:/usr/local/libexec/flowvisor/log4j-1.2.16.jar:/usr/local/libexec/flowvisor/flowvisor.jar org.flowvisor.FlowVisor
    
  • Unpacking that, here are all the paths mentioned in that invocation:
    /usr/local/etc/flowvisor
    /usr/local/libexec/flowvisor
    /usr/local/share/db/flowvisor
    /usr/local/etc/flowvisor/fvlog.config
    /usr/local/etc/flowvisor
    /usr/local/libexec/flowvisor
    /usr/local/share/db/flowvisor
    /usr/local/etc/flowvisor/fvlog.config
    /usr/local/etc/flowvisor/mySSLKeyStore
    /usr/local/libexec/flowvisor/openflow.jar
    /usr/local/libexec/flowvisor/xmlrpc-client-3.1.3.jar
    /usr/local/libexec/flowvisor/xmlrpc-common-3.1.3.jar
    /usr/local/libexec/flowvisor/xmlrpc-server-3.1.3.jar
    /usr/local/libexec/flowvisor/commons-logging-1.1.jar
    /usr/local/libexec/flowvisor/ws-commons-util-1.0.2.jar
    /usr/local/libexec/flowvisor/jsse.jar
    /usr/local/libexec/flowvisor/asm-3.0.jar
    /usr/local/libexec/flowvisor/cglib-2.2.jar
    /usr/local/libexec/flowvisor/commons-codec-1.4.jar
    /usr/local/libexec/flowvisor/commons-collections-3.2.1.jar
    /usr/local/libexec/flowvisor/commons-dbcp-1.4.jar
    /usr/local/libexec/flowvisor/commons-pool-1.5.6.jar
    /usr/local/libexec/flowvisor/gson-2.0.jar
    /usr/local/libexec/flowvisor/jetty-continuation-7.0.2.v20100331.jar
    /usr/local/libexec/flowvisor/jetty-http-7.0.2.v20100331.jar
    /usr/local/libexec/flowvisor/jetty-io-7.0.2.v20100331.jar
    /usr/local/libexec/flowvisor/jetty-security-7.0.2.v20100331.jar
    /usr/local/libexec/flowvisor/jetty-server-7.0.2.v20100331.jar
    /usr/local/libexec/flowvisor/jetty-util-7.0.2.v20100331.jar
    /usr/local/libexec/flowvisor/servlet-api-2.5.jar
    /usr/local/libexec/flowvisor/derby.jar
    /usr/local/libexec/flowvisor/derbytools.jar
    /usr/local/libexec/flowvisor/jna.jar
    /usr/local/libexec/flowvisor/syslog4j-0.9.46-bin.jar
    /usr/local/libexec/flowvisor/log4j-1.2.16.jar
    /usr/local/libexec/flowvisor/flowvisor.jar
    
  • Searching for packages for those:
    flowvisor,[~],08:15(0)$ dpkg -S /usr/local/etc/flowvisor /usr/local/libexec/flowvisor /usr/local/share/db/flowvisor /usr/local/etc/flowvisor/fvlog.config /usr/local/etc/flowvisor /usr/local/libexec/flowvisor /usr/local/share/db/flowvisor /usr/local/etc/flowvisor/fvlog.config /usr/local/etc/flowvisor/mySSLKeyStore /usr/local/libexec/flowvisor/openflow.jar /usr/local/libexec/flowvisor/xmlrpc-client-3.1.3.jar /usr/local/libexec/flowvisor/xmlrpc-common-3.1.3.jar /usr/local/libexec/flowvisor/xmlrpc-server-3.1.3.jar /usr/local/libexec/flowvisor/commons-logging-1.1.jar /usr/local/libexec/flowvisor/ws-commons-util-1.0.2.jar /usr/local/libexec/flowvisor/jsse.jar /usr/local/libexec/flowvisor/asm-3.0.jar /usr/local/libexec/flowvisor/cglib-2.2.jar /usr/local/libexec/flowvisor/commons-codec-1.4.jar /usr/local/libexec/flowvisor/commons-collections-3.2.1.jar /usr/local/libexec/flowvisor/commons-dbcp-1.4.jar /usr/local/libexec/flowvisor/commons-pool-1.5.6.jar /usr/local/libexec/flowvisor/gson-2.0.jar /usr/local/libexec/flowvisor/jetty-continuation-7.0.2.v20100331.jar /usr/local/libexec/flowvisor/jetty-http-7.0.2.v20100331.jar /usr/local/libexec/flowvisor/jetty-io-7.0.2.v20100331.jar /usr/local/libexec/flowvisor/jetty-security-7.0.2.v20100331.jar /usr/local/libexec/flowvisor/jetty-server-7.0.2.v20100331.jar /usr/local/libexec/flowvisor/jetty-util-7.0.2.v20100331.jar /usr/local/libexec/flowvisor/servlet-api-2.5.jar /usr/local/libexec/flowvisor/derby.jar /usr/local/libexec/flowvisor/derbytools.jar /usr/local/libexec/flowvisor/jna.jar /usr/local/libexec/flowvisor/syslog4j-0.9.46-bin.jar /usr/local/libexec/flowvisor/log4j-1.2.16.jar /usr/local/libexec/flowvisor/flowvisor.jar
    dpkg-query: no path found matching pattern /usr/local/etc/flowvisor.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor.
    dpkg-query: no path found matching pattern /usr/local/share/db/flowvisor.
    dpkg-query: no path found matching pattern /usr/local/etc/flowvisor/fvlog.config.
    dpkg-query: no path found matching pattern /usr/local/etc/flowvisor.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor.
    dpkg-query: no path found matching pattern /usr/local/share/db/flowvisor.
    dpkg-query: no path found matching pattern /usr/local/etc/flowvisor/fvlog.config.
    dpkg-query: no path found matching pattern /usr/local/etc/flowvisor/mySSLKeyStore.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/openflow.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/xmlrpc-client-3.1.3.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/xmlrpc-common-3.1.3.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/xmlrpc-server-3.1.3.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/commons-logging-1.1.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/ws-commons-util-1.0.2.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/jsse.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/asm-3.0.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/cglib-2.2.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/commons-codec-1.4.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/commons-collections-3.2.1.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/commons-dbcp-1.4.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/commons-pool-1.5.6.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/gson-2.0.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/jetty-continuation-7.0.2.v20100331.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/jetty-http-7.0.2.v20100331.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/jetty-io-7.0.2.v20100331.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/jetty-security-7.0.2.v20100331.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/jetty-server-7.0.2.v20100331.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/jetty-util-7.0.2.v20100331.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/servlet-api-2.5.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/derby.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/derbytools.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/jna.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/syslog4j-0.9.46-bin.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/log4j-1.2.16.jar.
    dpkg-query: no path found matching pattern /usr/local/libexec/flowvisor/flowvisor.jar.
    
  • Hmm, indeed, flowvisor isn't installed from a package at all:
    flowvisor,[~],08:17(0)$ dpkg -l | grep -i flow
    flowvisor,[~],08:17(1)$ 
    
  • Verifying that we can get versioning information for the packages:
    flowvisor,[~],08:19(0)$ dpkg -s openssh-server openjdk-6-jre-headless | egrep "^Package:|^Status:|^Maintainer:|^Version:|^$"
    Package: openssh-server
    Status: install ok installed
    Maintainer: Colin Watson <cjwatson@ubuntu.com>
    Version: 1:5.9p1-5ubuntu1
    
    Package: openjdk-6-jre-headless
    Status: install ok installed
    Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
    Version: 6b24-1.11.1-4ubuntu3
    
  • Verifying that we can download source for the packages:
    flowvisor,[~],08:20(0)$ mkdir tmp
    flowvisor,[~],08:20(0)$ cd tmp/
    
    flowvisor,[~/tmp],08:20(0)$ apt-get --download-only source openssh-server=1:5.9p1-5ubuntu1
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    Picking 'openssh' as source package instead of 'openssh-server'
    NOTICE: 'openssh' packaging is maintained in the 'Bzr' version control system at:
    http://anonscm.debian.org/bzr/pkg-ssh/openssh/trunk
    Please use:
    bzr branch http://anonscm.debian.org/bzr/pkg-ssh/openssh/trunk
    to retrieve the latest (possibly unreleased) updates to the package.
    Need to get 1,363 kB of source archives.
    Get:1 http://us.archive.ubuntu.com/ubuntu/ precise/main openssh 1:5.9p1-5ubuntu1 (dsc) [2,651 B]
    Get:2 http://us.archive.ubuntu.com/ubuntu/ precise/main openssh 1:5.9p1-5ubuntu1 (tar) [1,110 kB]
    Get:3 http://us.archive.ubuntu.com/ubuntu/ precise/main openssh 1:5.9p1-5ubuntu1 (diff) [251 kB]
    Fetched 1,363 kB in 1s (1,086 kB/s)
    Download complete and in download only mode
    
    flowvisor,[~/tmp],08:21(0)$ apt-get --download-only source openjdk-6-jre-headless=6b24-1.11.1-4ubuntu3
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    Picking 'openjdk-6' as source package instead of 'openjdk-6-jre-headless'
    NOTICE: 'openjdk-6' packaging is maintained in the 'Bzr' version control system at:
    http://bazaar.launchpad.net/~openjdk/openjdk/openjdk6
    Please use:
    bzr branch http://bazaar.launchpad.net/~openjdk/openjdk/openjdk6
    to retrieve the latest (possibly unreleased) updates to the package.
    Need to get 66.3 MB of source archives.
    Err http://us.archive.ubuntu.com/ubuntu/ precise-updates/main openjdk-6 6b24-1.11.1-4ubuntu3 (dsc)
      404  Not Found [IP: 91.189.91.15 80]
    Get:1 http://us.archive.ubuntu.com/ubuntu/ precise-updates/main openjdk-6 6b24-1.11.1-4ubuntu3 (tar) [66.1 MB]
    Err http://us.archive.ubuntu.com/ubuntu/ precise-updates/main openjdk-6 6b24-1.11.1-4ubuntu3 (diff)
      404  Not Found [IP: 91.189.91.15 80]
    Fetched 66.1 MB in 5s (12.9 MB/s)
    Failed to fetch http://us.archive.ubuntu.com/ubuntu/pool/main/o/openjdk-6/openjdk-6_6b24-1.11.1-4ubuntu3.dsc  404  Not Found [IP: 91.189.91.15 80]
    Failed to fetch http://us.archive.ubuntu.com/ubuntu/pool/main/o/openjdk-6/openjdk-6_6b24-1.11.1-4ubuntu3.diff.gz  404  Not Found [IP: 91.189.91.15 80]
    E: Failed to fetch some archives.
    
    flowvisor,[~/tmp],08:21(100)$ ls -l
    total 65888
    -rw-r--r-- 1 chaos chaos 66100230 Feb 16  2012 openjdk-6_6b24-1.11.1.orig.tar.gz
    -rw-r--r-- 1 chaos chaos   250819 Apr  2  2012 openssh_5.9p1-5ubuntu1.debian.tar.gz
    -rw-r--r-- 1 chaos chaos     2651 Apr  2  2012 openssh_5.9p1-5ubuntu1.dsc
    -rw-r--r-- 1 chaos chaos  1110014 Oct 17  2011 openssh_5.9p1.orig.tar.gz
    

So we can get openjdk itself, but not the Ubuntu-specific patches for it. That's somewhat surprising to me, but i think not a big deal. Looking at http://us.archive.ubuntu.com/ubuntu/pool/main/o/openjdk-6/, it seems that they do have some diffs there, just not the ones we are looking for.

I discussed the flowvisor question with Nick Bastin:

  • The version installed on flowvisor.utah.geniracks.net is 0.8.2, and is from a repository, not from a package:
    changeset:   7:49113872e1be
    branch:      0.8-MAINT
    user:        Nick Bastin <nick.bastin@gmail.com>
    date:        Mon Feb 06 17:00:00 2012 -0800
    summary:     Added tag FV-0.8.2 for changeset e3ef7c2a3463
    
  • The plan is to installed a packaged version of flowvisor on future racks if possible.
  • If not possible, we'll revisit whether there's another way to track the information about what is installed so that site admins can get access to it.

Results of testing step 4: 2013-02-28

All that remains here is to make sure flowvisor comes from a package on the standard rack:

flowvisor,[~],08:55(2)$ dpkg -S /usr/sbin/flowvisor 
flowvisor: /usr/sbin/flowvisor
flowvisor,[~],08:55(0)$ dpkg -l flowvisor
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version        Description
+++-==============-==============-============================================
ii  flowvisor      0.8.14         The OpenFlow FlowVisor
flowvisor,[~],08:55(0)$ hostname --long
flowvisor.instageni.gpolab.bbn.com

Done.

Step 5: identify network-listening software on the control host

Using:

  • Using netstat, enumerate processes on the control host which listen for network connections from outside the node
  • For each process found:
    • Use the command-line to determine what executable file is running
    • Use dpkg commands to determine whether the executable file is part of a Debian package
    • Otherwise, use documentation or iterate with the InstaGENI team to determine the origin of the software
  • For each Debian package found, identify a location from which the source package can be obtained
  • For each non-Debian package found, identify a location from which the source code for that version can be obtained.

Verify:

  • The source of each network-listening file can be identified
  • Source code or a source package can be identified for each Debian-sourced package
  • The source code and identifiable version (e.g. a git tag) can be found for each non-Debian software source

Results of testing step 5: 2012-05-17

  • Here's the netstat invocation to get all the IPv4/IPv6 listeners:
    control,[~],12:02(0)$ sudo netstat -lnp46
    Active Internet connections (only servers)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
    tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1007/sshd       
    tcp        0      0 127.0.0.1:6010          0.0.0.0:*               LISTEN      1650/0          
    tcp        0      0 127.0.0.1:6011          0.0.0.0:*               LISTEN      3936/3          
    tcp6       0      0 :::22                   :::*                    LISTEN      1007/sshd       
    tcp6       0      0 ::1:6010                :::*                    LISTEN      1650/0          
    tcp6       0      0 ::1:6011                :::*                    LISTEN      3936/3          
    
  • Looking up those binaries:
    control,[~],12:11(0)$ sudo ls -l /proc/{1007,1650,3936}/exe
    lrwxrwxrwx 1 root root 0 May 10 18:16 /proc/1007/exe -> /usr/sbin/sshd
    lrwxrwxrwx 1 root root 0 May 17 12:16 /proc/1650/exe -> /usr/sbin/sshd
    lrwxrwxrwx 1 root root 0 May 17 12:16 /proc/3936/exe -> /usr/sbin/sshd
    
  • So the only thing listening is sshd. Find out what package sshd is from:
    control,[~],12:19(1)$ dpkg -S /usr/sbin/sshd
    openssh-server: /usr/sbin/sshd
    
    control,[~],12:19(0)$ dpkg -s openssh-server
    Package: openssh-server
    Status: install ok installed
    Multi-Arch: foreign
    Priority: optional
    Section: net
    Installed-Size: 807
    Maintainer: Colin Watson <cjwatson@ubuntu.com>
    Architecture: amd64
    Source: openssh
    Version: 1:5.9p1-5ubuntu1
    ...
    
  • Testing download of deb source for this apt-provided package:
    control,[~],12:30(0)$ mkdir tmp
    control,[~],12:34(0)$ cd tmp/
    
    control,[~/tmp],12:34(0)$ apt-get --download-only source openssh-server=1:5.9p1-5ubuntu1
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    Picking 'openssh' as source package instead of 'openssh-server'
    NOTICE: 'openssh' packaging is maintained in the 'Bzr' version control system at:
    http://anonscm.debian.org/bzr/pkg-ssh/openssh/trunk
    Please use:
    bzr branch http://anonscm.debian.org/bzr/pkg-ssh/openssh/trunk
    to retrieve the latest (possibly unreleased) updates to the package.
    Need to get 1,363 kB of source archives.
    Get:1 http://us.archive.ubuntu.com/ubuntu/ precise/main openssh 1:5.9p1-5ubuntu1 (dsc) [2,651 B]
    Get:2 http://us.archive.ubuntu.com/ubuntu/ precise/main openssh 1:5.9p1-5ubuntu1 (tar) [1,110 kB]
    Get:3 http://us.archive.ubuntu.com/ubuntu/ precise/main openssh 1:5.9p1-5ubuntu1 (diff) [251 kB]
    Fetched 1,363 kB in 1s (827 kB/s)
    Download complete and in download only mode
    
    control,[~/tmp],12:34(0)$ ls
    openssh_5.9p1-5ubuntu1.debian.tar.gz  openssh_5.9p1.orig.tar.gz
    openssh_5.9p1-5ubuntu1.dsc
    
    control,[~/tmp],12:34(0)$ file *
    openssh_5.9p1-5ubuntu1.debian.tar.gz: gzip compressed data, from Unix, max compression
    openssh_5.9p1-5ubuntu1.dsc:           ASCII text
    openssh_5.9p1.orig.tar.gz:            gzip compressed data, extra field, from Unix
    
    The first tarball is the control files and sources patches. The second tarball is the original OpenSSH source code which was used.

Step 6: identify network-listening software on an OpenVZ experimental node

Using:

  • Using netstat, enumerate processes on an allocated node running the OpenVZ image which listen for network connections from outside the node
  • For each process found:
    • Use the command-line or /proc to determine what executable file is running
    • Use RPM tools to determine whether the executable file is part of an RPM
    • Otherwise, use documentation or iterate with the InstaGENI team to determine the origin of the software
  • For each RPM found, identify a location from which a source RPM for that package can be obtained
  • For each non-RPM software source found, identify a location from which the source code for that version can be obtained.

Verify:

  • The source of each network-listening file can be identified
  • RPM source packages can be found for each RPM-sourced package
  • The source code and identifiable version (e.g. a git tag) can be found for each non-RPM software source

Results of testing step 6: 2012-05-17

Testing on pc5.utah.geniracks.net, which is operating as an OpenVZ host right now.

  • List listening processes:
    vhost1,[~],12:47(4)$ sudo netstat -lnp -A inet > netstat.listeners
    vhost1,[~],12:47(0)$ sudo netstat -lnp -A inet6 >> netstat.listeners
    
    vhost1,[~],12:49(0)$ cut -b88- netstat.listeners | awk '{print $1}' | sort -u
    -
    1286/rpcbind
    1301/sshd
    1410/rpc.statd
    1551/emulab-syncd
    17497/sshd
    17506/pubsubd
    862/ntpd
    PID/Program
    
  • Find the binaries for the processes which are identified:
    vhost1,[~],12:59(0)$ sudo ls -l /proc/{862,1286,1301,1410,1551,17497,17506}/exe
    lrwxrwxrwx 1 root root 0 May 17 12:59 /proc/1286/exe -> /sbin/rpcbind
    lrwxrwxrwx 1 root root 0 May 17 12:59 /proc/1301/exe -> /usr/sbin/sshd
    lrwxrwxrwx 1 root root 0 May 17 12:59 /proc/1410/exe -> /sbin/rpc.statd
    lrwxrwxrwx 1 root root 0 May 17 12:59 /proc/1551/exe -> /usr/local/etc/emulab/emulab-syncd
    lrwxrwxrwx 1 root root 0 May 17 12:59 /proc/17497/exe -> /mnt/pcvm5-1/root/usr/sbin/sshd
    lrwxrwxrwx 1 root root 0 May 17 12:59 /proc/17506/exe -> /mnt/pcvm5-1/root/usr/local/libexec/pubsubd
    lrwxrwxrwx 1 root root 0 May 16 15:00 /proc/862/exe -> /usr/sbin/ntpd
    
  • Some of those processes are running inside an OpenVZ container, but are identical to binaries in the base system:
    $ md5sum /mnt/pcvm5-1/root/usr/sbin/sshd /usr/sbin/sshd 
    39aceab46fa9705600dc8b194649bf9c  /mnt/pcvm5-1/root/usr/sbin/sshd
    39aceab46fa9705600dc8b194649bf9c  /usr/sbin/sshd
    
    $ md5sum /mnt/pcvm5-1/root/usr/local/libexec/pubsubd /usr/local/libexec/pubsubd 
    823239d468e277b7c634d34d82c6049a  /mnt/pcvm5-1/root/usr/local/libexec/pubsubd
    823239d468e277b7c634d34d82c6049a  /usr/local/libexec/pubsubd
    
  • Some of those processes are from RPM packages:
    • Determine which RPM each package is from:
      $ rpm -qf /sbin/rpcbind 
      rpcbind-0.2.0-10.fc15.x86_64
      
      $ rpm -qf /usr/sbin/sshd 
      openssh-server-5.6p1-34.fc15.1.x86_64
      
      $ rpm -qf /sbin/rpc.statd 
      nfs-utils-1.2.4-1.fc15.x86_64
      
      $ rpm -qf /usr/sbin/ntpd
      ntp-4.2.6p3-4.fc15.x86_64
      
    • Determine which repo each of these packages came from:
      $ yum list installed rpcbind openssh-server nfs-utils ntp
      Installed Packages
      nfs-utils.x86_64                       1:1.2.4-1.fc15                   @updates
      ntp.x86_64                             4.2.6p3-4.fc15                   @fedora 
      openssh-server.x86_64                  5.6p1-34.fc15.1                  @updates
      rpcbind.x86_64                         0.2.0-10.fc15                    @fedora 
      
    • Look in /etc/yum.repos.d/*.repo for information about the locations of RPM files:
      $ cat /etc/yum.repos.d/fedora.repo
      [fedora]
      mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch
      ...
      
      $ cat /etc/yum.repos.d/fedora-updates.repo
      [updates]
      mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=updates-released-f$releasever&arch=$basearch
      
      $ cat /etc/redhat-release
      Fedora release 15 (Lovelock)
      
      $ uname -m
      x86_64
      
    • Browse to https://mirrors.fedoraproject.org/
    • Click to https://mirrors.fedoraproject.org/publiclist/Fedora/15/x86_64/
    • Browse to a reasonable-looking mirror
    • Click through to http://mirrors.kernel.org/fedora/releases/15/Everything/source/SRPMS/
    • Download http://mirrors.kernel.org/fedora/releases/15/Everything/source/SRPMS/ntp-4.2.6p3-4.fc15.src.rpm
    • Unpack the source file with rpm2cpio:
      rpm2cpio ntp-4.2.6p3-4.fc15.src.rpm | cpio -idmv
      
    • This yields a bunch of patches and a tarball of the source used.
    • The same procedure should work for other things from fedora, and for things from updates (for the latter s/releases/updates/ when browsing through the tree of available packages).
  • Some of those files are not from RPMs:
    /usr/local/etc/emulab/emulab-syncd
    /usr/local/libexec/pubsubd
    
  • Netstat also reports some listeners with no processes:
    Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
    tcp        0      0 0.0.0.0:58441               0.0.0.0:*                   LISTEN      -                   
    udp        0      0 0.0.0.0:990                 0.0.0.0:*                               -                   
    udp        0      0 0.0.0.0:45938               0.0.0.0:*                               -                   
    tcp        0      0 :::57373                    :::*                        LISTEN      -                   
    udp        0      0 :::47886                    :::*                                    -                   
    
    However, lsof -i does not report these ports, so i am inclined not to worry about them, not understanding what has caused netstat to report them.

Unresolved question:

  • How do i determine what version of the Emulab source was used to create a particular image?

Step 7: verify VLANs on the rack management switch

Using:

  • Establish a privileged login to the control plane switch
  • Obtain the list of all VLAN mappings for all interfaces
  • Determine which interfaces connect to experimental nodes
  • Obtain a list of the full MAC address table of the switch
  • Use interface listings on hosts and devices to determine the identities of all MAC addresses

Verify:

  • All experimental node control interfaces are on an expected VLAN
  • It is possible to identify and classify every MAC address visible on the control switch

Results of testing step 7: 2012-05-17

  • Telnet to procurve1 from boss
  • Look at the VLAN mappings for interfaces:
    ProCurve Switch 2610-24# show running-config 
    ...
    vlan 1 
       untagged 1-24,26-27 
       no untagged 25,28 
    
    vlan 257 
       untagged 25 
    
    vlan 260 
       untagged 28 
    
    
  • Then use show interfaces brief to find all up interfaces and their VLANs:
    ProCurve Switch 2610-24# show interfaces brief 
    
     Status and Counters - Port Status
    
                      | Intrusion                           MDI   Flow  Bcast 
      Port  Type      | Alert     Enabled Status Mode       Mode  Ctrl  Limit 
      ----- --------- + --------- ------- ------ ---------- ----- ----- ------
      1     10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
      2     10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
      3     10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
      4     10/100TX  | No        Yes     Up     100FDx     MDI   off   0     (vlan 1)
      5     10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
      6     10/100TX  | No        Yes     Up     100FDx     MDI   off   0     (vlan 1)
      7     10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
      8     10/100TX  | No        Yes     Up     100FDx     MDI   off   0     (vlan 1)
      9     10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
      10    10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
    ...
      21    10/100TX  | No        Yes     Up     100FDx     MDI   off   0     (vlan 1)
      22    10/100TX  | No        Yes     Up     100FDx     MDI   off   0     (vlan 1)
      23    10/100TX  | No        Yes     Up     100FDx     MDI   off   0     (vlan 1)
      24    10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
    ...
      26    100/1000T | No        Yes     Up     1000FDx    MDIX  off   0     (vlan 1)
    ...
    
  • As you'd expect, there are no mac addresses on vlans 257 and 260:
    ProCurve Switch 2610-24# show mac-address vlan 257
    
     Status and Counters - Address Table - VLAN 257
    
      MAC Address   Port 
      ------------- -----
     
    ProCurve Switch 2610-24# show mac-address vlan 260
    
     Status and Counters - Address Table - VLAN 260
    
      MAC Address   Port 
      ------------- -----
     
    
  • Here are some MACs on vlan 1, reordered by port:
    ProCurve Switch 2610-24# show mac-address vlan 1
    
     Status and Counters - Address Table - VLAN 1
    
      MAC Address   Port 
      ------------- -----
      e4115b-ed1cb4 1    (pc5[eth0]: per login to pc5)
      e83935-ae8586 2    (pc5[iLO]: per iLO information)
      e83935-ae35a6 4    (pc2[iLO]: per iLO information) 
      e83935-aec97c 6    (pc1[iLO]: per iLO information)
      e83935-b14e88 7    (pc3[eth0]: per login to pc3)
      e83935-ae8b2a 8    (pc3[iLO]: per iLO information)
      e83935-ae5566 10   (pc4[iLO]: per iLO information)
      00009b-6201df 21   
      00009b-6224df 23   
      00009b-6224e0 23   
      e4115b-eae224 23   (control[eth0]: per login to control)
      e4115b-e6580c 24   
      0024a8-547c5e 26   
      00d0bc-f414f8 26   (155.98.31.1: per control's arp table)
    
  • On reflection, i don't think tracking down the rest of this makes much sense until we have our own rack to look at. I will ask for a wiring diagram of the Utah rack, but it's probably better to wait for the BBN rack to assess what is plugged into the control network.