wiki:GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-MON-1

Version 7 (modified by chaos@bbn.com, 8 years ago) (diff)

--

Detailed test plan for IG-MON-1: Control Network Software and VLAN Inspection Test

This page is GPO's working page for performing IG-MON-1. It is public for informational purposes, but it is not an official status report. See GENIRacksHome/InstageniRacks/AcceptanceTestStatus for the current status of InstaGENI acceptance tests.

Last substantive edit of this page: 2012-05-17

Page format

  • The status chart summarizes the state of this test
  • The high-level description from test plan contains text copied exactly from the public test plan and acceptance criteria pages.
  • The steps contain things i will actually do/verify:
    • Steps may be composed of related substeps where i find this useful for clarity
    • Each step is either a preparatory step (identified by "(prep)") or a verification step (the default):
      • Preparatory steps are just things we have to do. They're not tests of the rack, but are prerequisites for subsequent verification steps
      • Verification steps are steps in which we will actually look at rack output and make sure it is as expected. They contain a Using: block, which lists the steps to run the verification, and an Expect: block which lists what outcome is expected for the test to pass.

Status of test

Step State Date completed Tickets Comments
1 Color(orange,Blocked)? instaticket:24 blocked on resolution of "how to get source" questions for Emulab FreeBSD systems
2 Color(orange,Blocked)? instaticket:24 blocked on resolution of "how to get source" questions for Emulab FreeBSD systems
3 Color(orange,Blocked)? blocked on sudo access to foam VM
4 Color(orange,Blocked)? blocked on sudo access to flowvisor VM
5 Color(green,Pass)?
6 Color(orange,Blocked)? blocked on resolution of "how to get Emulab version" question for Emulab images
7 Color(yellow,Blocked-site)? blocked on receipt of the BBN rack

High-level description from test plan

This test inspects the state of the rack control network, infrastructure nodes, and system software.

Procedure

  • A site administrator enumerates processes on each of the server host, the boss VM, the ops VM, the FOAM VM, and an experimental node configured for OpenVZ, which listen for network connections from other nodes, identifies what version of what software package is in use for each, and verifies that we know the source of each piece of software and could get access to its source code.
  • A site administrator reviews the configuration of the rack control plane switch and verifies that each experimental node's control and iLO interfaces are on the expected VLANs.
  • A site administrator reviews the MAC address table on the control plane switch, and verifies that all entries are identifiable and expected.

Criteria to verify as part of this test

  • VI.09. A public document explains how to identify the software versions and system file configurations running on the rack, and how to get information about recent changes to the rack software and configuration. (F.5)
  • VI.11. A public document describes the GENI software running on the rack, and explains how to get access to the source code of each piece of GENI software. (F.6)
  • VII.03. Site administrators can understand the expected control and dataplane network behavior of their rack. (F.2)
  • VII.04. Site administrators can view and investigate current system and network activity on their rack. (F.2)
  • VII.06. A site administrator can verify the control software and configurations on the rack at some point in time. (F.5)
  • VII.08. A site administrator can get access to source code for the version of each piece of GENI code installed on their site rack at some point in time. (F.6)
  • VII.09. A site administrator can determine the MAC addresses of all physical host interfaces, all network device interfaces, all active experimental VMs, and all recently-terminated experimental VMs. (C.3.f)
  • VII.10. A site administrator can locate current and recent CPU and memory utilization for each rack network device, and can find recent changes or errors in a log. (D.6.a)
  • VII.12. For each infrastructure and experimental host, a site administrator can locate current and recent uptime, CPU, disk, and memory utilization, interface traffic counters, process counts, and active user counts. (D.6.b)
  • VII.13. A site administrator can locate recent syslogs for all infrastructure and experimental hosts. (D.6.b)

Results of experiment setup on rack: 2012-05-17

Preparation: i wrote these tests on the assumption that there would be some active experiments on the rack while i was testing, and there aren't. So i wanted some running experiments to look at.

  • First attempt (didn't work):
    • Here is my rspec:
      jericho,[~],10:00(0)$ cat ~/IG-MON-nodes-A.rspec 
      <?xml version="1.0" encoding="UTF-8"?>
      <!-- This rspec will reserve one physical node and one openvz node, each
           with no OS specified, and create a single dataplane link between
           them.  It should work on any Emulab which has nodes available and
           supports OpenVZ.  -->
      <rspec xmlns="http://protogeni.net/resources/rspec/0.2">
        <node client_id="phys1" exclusive="true">
          <sliver_type name="raw" />
          <interface client_id="phys1:if0" />
        </node>
        <node client_id="virt1" exclusive="false">
          <sliver_type name="emulab-openvz" />
          <interface client_id="virt1:if0" />
        </node>
      
        <link client_id="phys1-virt1-0">
          <interface_ref client_id="phys1:if0"/>
          <interface_ref client_id="virt1:if0"/>
          <property source_id="phys1:if0" dest_id="virt1:if0"/>
          <property source_id="virt1:if0" dest_id="phys1:if0"/>
        </link>
      </rspec>
      
    • Make sure i have a long enough slice:
      omni renewslice ecgtest 2012-05-18  # Hmm, maybe not long enough if i work this evening?
      omni renewslice ecgtest 2012-05-19
      
    • Now try creating the sliver:
      jericho,[~],10:03(0)$ omni -a http://www.utah.geniracks.net/protogeni/xmlrpc/am createsliver ecgtest ~/IG-MON-nodes-A.rspec
      INFO:omni:Loading config file /home/chaos/omni/omni_pgeni
      INFO:omni:Using control framework pg
      INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest expires on 2012-05-19 00:00:00 UTC
      INFO:omni:Creating sliver(s) from rspec file /home/chaos/IG-MON-nodes-A.rspec for slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest
      ERROR:omni.protogeni:Call for Create Sliver urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest at http://www.utah.geniracks.net/protogeni/xmlrpc/am failed. Server says: <Fault 1: 'Must provide a virtualization_type'>
      INFO:omni:Asked http://www.utah.geniracks.net/protogeni/xmlrpc/am to reserve resources. Result:
      INFO:omni:<!-- Reserved resources for:
              Slice: ecgtest
              At AM:
              URL: http://www.utah.geniracks.net/protogeni/xmlrpc/am
       -->
      INFO:omni: ------------------------------------------------------------
      INFO:omni: Completed createsliver:
      
        Options as run:
                      aggregate: http://www.utah.geniracks.net/protogeni/xmlrpc/am
                      configfile: /home/chaos/omni/omni_pgeni
                      framework: pg
                      native: True
      
        Args: createsliver ecgtest /home/chaos/IG-MON-nodes-A.rspec
      
        Result Summary: Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest expires on 2012-05-19 00:00:00 UTC
      Asked http://www.utah.geniracks.net/protogeni/xmlrpc/am to reserve resources. No manifest Rspec returned. <Fault 1: 'Must provide a virtualization_type'> 
      INFO:omni: ============================================================
      
  • Is this just an rspec versioning issue?
    • Take 2 rspec:
      jericho,[~],10:11(0)$ cat IG-MON-nodes-B.rspec 
      <?xml version="1.0" encoding="UTF-8"?>
      <!-- This rspec will reserve one physical node and one openvz node, each
           with no OS specified, and create a single dataplane link between
           them.  It should work on any Emulab which has nodes available and
           supports OpenVZ.  -->
      <rspec xmlns="http://www.geni.net/resources/rspec/3"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
             xsi:schemaLocation="http://www.geni.net/resources/rspec/3
                                 http://www.geni.net/resources/rspec/3/request.xsd" 
             type="request">
      
        <node client_id="phys1" exclusive="true">
          <sliver_type name="raw" />
          <interface client_id="phys1:if0" />
        </node>
        <node client_id="virt1" exclusive="false">
          <sliver_type name="emulab-openvz" />
          <interface client_id="virt1:if0" />
        </node>
      
        <link client_id="phys1-virt1-0">
          <interface_ref client_id="phys1:if0"/>
          <interface_ref client_id="virt1:if0"/>
          <property source_id="phys1:if0" dest_id="virt1:if0"/>
          <property source_id="virt1:if0" dest_id="phys1:if0"/>
        </link>
      </rspec>
      
    • Try creating the sliver:
      jericho,[~],10:12(0)$ omni -a http://www.utah.geniracks.net/protogeni/xmlrpc/am createsliver ecgtest ~/IG-MON-nodes-B.rspec
      INFO:omni:Loading config file /home/chaos/omni/omni_pgeni
      INFO:omni:Using control framework pg
      INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest expires on 2012-05-19 00:00:00 UTC
      INFO:omni:Creating sliver(s) from rspec file /home/chaos/IG-MON-nodes-B.rspec for slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest
      INFO:omni:Asked http://www.utah.geniracks.net/protogeni/xmlrpc/am to reserve resources. Result:
      INFO:omni:<?xml version="1.0" ?>
      INFO:omni:<!-- Reserved resources for:
              Slice: ecgtest
              At AM:
              URL: http://www.utah.geniracks.net/protogeni/xmlrpc/am
       -->
      INFO:omni:<rspec type="manifest" xmlns="http://www.geni.net/resources/rspec/3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.geni.net/resources/rspec/3                            http://www.geni.net/resources/rspec/3/manifest.xsd">  
      
          <node client_id="phys1" component_id="urn:publicid:IDN+utah.geniracks.net+node+pc3" component_manager_id="urn:publicid:IDN+utah.geniracks.net+authority+cm" exclusive="true" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+366">    
              <sliver_type name="raw-pc"/>    
              <interface client_id="phys1:if0" component_id="urn:publicid:IDN+utah.geniracks.net+interface+pc3:eth1" mac_address="e83935b14e8a" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+369">      <ip address="10.10.1.1" type="ipv4"/>    </interface>    
            <rs:vnode name="pc3" xmlns:rs="http://www.protogeni.net/resources/rspec/ext/emulab/1"/>    <host name="phys1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net"/>    <services>      <login authentication="ssh-keys" hostname="pc3.utah.geniracks.net" port="22" username="chaos"/>    </services>  </node>  
          <node client_id="virt1" component_id="urn:publicid:IDN+utah.geniracks.net+node+pc5" component_manager_id="urn:publicid:IDN+utah.geniracks.net+authority+cm" exclusive="false" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+367">    
              <sliver_type name="emulab-openvz"/>    
              <interface client_id="virt1:if0" component_id="urn:publicid:IDN+utah.geniracks.net+interface+pc5:eth1" mac_address="00000a0a0102" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+370">      <ip address="10.10.1.2" type="ipv4"/>    </interface>    
            <rs:vnode name="pcvm5-1" xmlns:rs="http://www.protogeni.net/resources/rspec/ext/emulab/1"/>    <host name="virt1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net"/>    <services>      <login authentication="ssh-keys" hostname="pc5.utah.geniracks.net" port="30010" username="chaos"/>    </services>  </node>  
      
          <link client_id="phys1-virt1-0" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+368" vlantag="259">    
              <interface_ref client_id="phys1:if0" component_id="urn:publicid:IDN+utah.geniracks.net+interface+pc3:eth1" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+369"/>    
              <interface_ref client_id="virt1:if0" component_id="urn:publicid:IDN+utah.geniracks.net+interface+pc5:eth1" sliver_id="urn:publicid:IDN+utah.geniracks.net+sliver+370"/>    
              <property dest_id="virt1:if0" source_id="phys1:if0"/>    
              <property dest_id="phys1:if0" source_id="virt1:if0"/>    
          </link>  
      </rspec>
      INFO:omni: ------------------------------------------------------------
      INFO:omni: Completed createsliver:
      
        Options as run:
                      aggregate: http://www.utah.geniracks.net/protogeni/xmlrpc/am
                      configfile: /home/chaos/omni/omni_pgeni
                      framework: pg
                      native: True
      
        Args: createsliver ecgtest /home/chaos/IG-MON-nodes-B.rspec
      
        Result Summary: Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest expires on 2012-05-19 00:00:00 UTC
      Reserved resources on http://www.utah.geniracks.net/protogeni/xmlrpc/am.  
      INFO:omni: ============================================================
      
  • The sliverstatus output is very long, but the relevant information is:
    • pc3.utah.geniracks.net has been assigned to be phys1, and it is ready
    • virt1 is hosted on pc5.utah.geniracks.net, and my login port is 30010
    • When i login to phys1, it has active interfaces:
      eth0    E8:39:35:B1:4E:88  155.98.34.13/24
      eth1    E8:39:35:B1:4E:8A  10.10.1.1/24
      
    • When i login to virt1, it has active interfaces:
      eth999  00:00:AC:11:05:01  172.17.5.1/12
      mv1.1   82:01:0A:0A:01:02  10.10.1.2/24
      
    • I can ping from phys1 to 10.10.1.2
    • I can ping from virt1 to 10.10.1.1

Step 1: identify network-listening software on the boss node

Using:

  • Using netstat, enumerate processes on boss which listen for network connections from outside the node
  • For each process found:
    • Use the command-line to determine what executable file is running
    • Use pkg_info to determine whether the executable file is part of a FreeBSD package/port
    • Otherwise, use documentation or iterate with the InstaGENI team to determine the origin of the software
  • For each FreeBSD package found, identify a location from which the port source can be obtained
  • For each non-FreeBSD software source found, identify a location from which the source code for that version can be obtained.

Verify:

  • The source of each network-listening file can be identified
  • FreeBSD ports can be identified for each FreeBSD-sourced package
  • The source code and identifiable version (e.g. a git tag) can be found for each non-FreeBSD software source

Results of testing: 2012-05-17

  • I said netstat a bunch in these test definitions, but in fact sockstat -lL46 is my goto on FreeBSD. Get a full list of all binaries of processes which are listening on IPv4 or IPv6 sockets from non-localhost addresses:
    $ for pid in $(sockstat -lL46 | awk '{print $3}' | grep -v PID | sort -u); do procstat -b $pid; done | awk '{print $3}' | sort -u
    /usr/libexec/sendmail/sendmail
    /usr/local/bin/python2.6     # running /usr/testbed/sbin/sslxmlrpc_server.py
    /usr/local/libexec/pubsubd
    /usr/local/libexec/tftpd
    /usr/local/sbin/dhcpd
    /usr/local/sbin/httpd
    /usr/sbin/inetd
    /usr/sbin/lwresd             # hard link from /usr/sbin/named
    /usr/sbin/mountd
    /usr/sbin/nfsd
    /usr/sbin/ntpd
    /usr/sbin/rpcbind
    /usr/sbin/sshd
    /usr/sbin/syslogd
    /usr/testbed/sbin/bootinfo
    /usr/testbed/sbin/capserver
    /usr/testbed/sbin/mfrisbeed
    /usr/testbed/sbin/sdcollectd
    /usr/testbed/sbin/tmcd
    PATH
    
  • The following commands are sourced from freebsd packages:
    $ pkg_info -W /usr/local/bin/python2.6
    /usr/local/bin/python2.6 was installed by package python26-2.6.6
    
    $ pkg_info -W /usr/local/libexec/pubsubd
    /usr/local/libexec/pubsubd was installed by package pubsub-0.95
    
    $ pkg_info -W /usr/local/libexec/tftpd
    /usr/local/libexec/tftpd was installed by package emulab-tftp-hpa-0.48
    
    $ pkg_info -W /usr/local/libexec/tftpd
    /usr/local/libexec/tftpd was installed by package emulab-tftp-hpa-0.48
    
    $ pkg_info -W /usr/local/sbin/dhcpd
    /usr/local/sbin/dhcpd was installed by package isc-dhcp42-server-4.2.3_1
    
    $ pkg_info -W /usr/local/sbin/httpd
    /usr/local/sbin/httpd was installed by package apache-2.2.21
    
    • So the summary of sourced packages here is:
      apache-2.2.21
      emulab-tftp-hpa-0.48
      isc-dhcp42-server-4.2.3_1
      pubsub-0.95
      python26-2.6.6
      
  • The following commands aren't part of packages (pkg_info -W reports nothing):
    /usr/libexec/sendmail/sendmail
    /usr/sbin/inetd
    /usr/sbin/lwresd             # hard link from /usr/sbin/named
    /usr/sbin/mountd
    /usr/sbin/nfsd
    /usr/sbin/ntpd
    /usr/sbin/rpcbind
    /usr/sbin/sshd
    /usr/sbin/syslogd
    /usr/testbed/sbin/bootinfo
    /usr/testbed/sbin/capserver
    /usr/testbed/sbin/mfrisbeed
    /usr/testbed/sbin/sdcollectd
    /usr/testbed/sbin/sslxmlrpc_server.py
    /usr/testbed/sbin/tmcd
    
  • The assumption is that these are either part of the FreeBSD base system, or are part of Emulab. How do we find out which?
    • If the OS had been compiled recently, i could look in /usr/obj for binaries which were identical to things on the system. However, it's currently running the base install (afaict), so that won't work.
    • Since Emulab has been compiled recently, i can look in the canonical source of that compile, which i believe is /users/stoller/testbed/obj. Here's a process for checking various things in /usr/testbed/sbin, on the suspicion that they are probably Emulab binaries:
      shortname=bootinfo    # or whatever
      shortmd5=$(md5 /usr/testbed/sbin/$shortname | awk '{print $4}')
      for path in $(find . -type f -name $shortname 2> /dev/null); do md5 $path; done | grep $shortmd5
      
  • That finds that the following items are from Emulab:
    /usr/testbed/sbin/bootinfo:            ./pxe/bootinfo
    /usr/testbed/sbin/capserver:           ./capture/capserver
    /usr/testbed/sbin/mfrisbeed:           ./clientside/os/frisbee.redux/mfrisbeed
    /usr/testbed/sbin/sdcollectd:          ./clientside/sensors/slothd/sdcollectd
    /usr/testbed/sbin/sslxmlrpc_server.py: ./xmlrpc/sslxmlrpc_server.py
    /usr/testbed/sbin/tmcd:                ./tmcd/tmcd
    
  • So the following items are not from Emulab, and we assume they would be part of the base install:
    /usr/libexec/sendmail/sendmail
    /usr/sbin/inetd
    /usr/sbin/lwresd             # hard link from /usr/sbin/named
    /usr/sbin/mountd
    /usr/sbin/nfsd
    /usr/sbin/ntpd
    /usr/sbin/rpcbind
    /usr/sbin/sshd
    /usr/sbin/syslogd
    

So, what's needed to be able to finalize this?

  1. Ask InstaGENI to provide a reliable way for site admins to find out the .../src and .../obj directories which correspond to the installed software.
  2. Ask someone to come up with a suggestion for how to reverse engineer from installed software which is assumed to be part of the FreeBSD base, to the version/source code which was used to generate it.
  3. Ask someone where the source for the FreeBSD packages installed on the system, some of which are Emulab-specific, come from.

Step 2: identify network-listening software on the ops node

Using:

  • Using netstat, enumerate processes on ops which listen for network connections from outside the node
  • For each process found:
    • Use the command-line to determine what executable file is running
    • Use pkg_info to determine whether the executable file is part of a FreeBSD package/port
    • Otherwise, use documentation or iterate with the InstaGENI team to determine the origin of the software
  • For each FreeBSD package found, identify a location from which the port source can be obtained
  • For each non-FreeBSD software source found, identify a location from which the source code for that version can be obtained.

Verify:

  • The source of each network-listening file can be identified
  • FreeBSD ports can be identified for each FreeBSD-sourced package
  • The source code and identifiable version (e.g. a git tag) can be found for each non-FreeBSD software source

Step 3: identify network-listening software on the foam node

Using:

  • Using netstat, enumerate processes on foam which listen for network connections from outside the node
  • For each process found:
    • Use the command-line to determine what executable file is running
    • Use dpkg commands to determine whether the executable file is part of a Debian package
    • Otherwise, use documentation or iterate with the InstaGENI team to determine the origin of the software
  • For each Debian package found, identify a location from which the source package can be obtained
  • For each non-Debian package found, identify a location from which the source code for that version can be obtained.

Verify:

  • The source of each network-listening file can be identified
  • Source code or a source package can be identified for each Debian-sourced package
  • The source code and identifiable version (e.g. a git tag) can be found for each non-Debian software source

Step 4: identify network-listening software on the FlowVisor node

Using:

  • Using netstat, enumerate processes on flowvisor which listen for network connections from outside the node
  • For each process found:
    • Use the command-line to determine what executable file is running
    • Use dpkg commands to determine whether the executable file is part of a Debian package
    • Otherwise, use documentation or iterate with the InstaGENI team to determine the origin of the software
  • For each Debian package found, identify a location from which the source package can be obtained
  • For each non-Debian package found, identify a location from which the source code for that version can be obtained.

Verify:

  • The source of each network-listening file can be identified
  • Source code or a source package can be identified for each Debian-sourced package
  • The source code and identifiable version (e.g. a git tag) can be found for each non-Debian software source

Step 5: identify network-listening software on the control host

Using:

  • Using netstat, enumerate processes on the control host which listen for network connections from outside the node
  • For each process found:
    • Use the command-line to determine what executable file is running
    • Use dpkg commands to determine whether the executable file is part of a Debian package
    • Otherwise, use documentation or iterate with the InstaGENI team to determine the origin of the software
  • For each Debian package found, identify a location from which the source package can be obtained
  • For each non-Debian package found, identify a location from which the source code for that version can be obtained.

Verify:

  • The source of each network-listening file can be identified
  • Source code or a source package can be identified for each Debian-sourced package
  • The source code and identifiable version (e.g. a git tag) can be found for each non-Debian software source

Results of testing: 2012-05-17

  • Here's the netstat invocation to get all the IPv4/IPv6 listeners:
    control,[~],12:02(0)$ sudo netstat -lnp46
    Active Internet connections (only servers)
    Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
    tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1007/sshd       
    tcp        0      0 127.0.0.1:6010          0.0.0.0:*               LISTEN      1650/0          
    tcp        0      0 127.0.0.1:6011          0.0.0.0:*               LISTEN      3936/3          
    tcp6       0      0 :::22                   :::*                    LISTEN      1007/sshd       
    tcp6       0      0 ::1:6010                :::*                    LISTEN      1650/0          
    tcp6       0      0 ::1:6011                :::*                    LISTEN      3936/3          
    
  • Looking up those binaries:
    control,[~],12:11(0)$ sudo ls -l /proc/{1007,1650,3936}/exe
    lrwxrwxrwx 1 root root 0 May 10 18:16 /proc/1007/exe -> /usr/sbin/sshd
    lrwxrwxrwx 1 root root 0 May 17 12:16 /proc/1650/exe -> /usr/sbin/sshd
    lrwxrwxrwx 1 root root 0 May 17 12:16 /proc/3936/exe -> /usr/sbin/sshd
    
  • So the only thing listening is sshd. Find out what package sshd is from:
    control,[~],12:19(1)$ dpkg -S /usr/sbin/sshd
    openssh-server: /usr/sbin/sshd
    
    control,[~],12:19(0)$ dpkg -s openssh-server
    Package: openssh-server
    Status: install ok installed
    Multi-Arch: foreign
    Priority: optional
    Section: net
    Installed-Size: 807
    Maintainer: Colin Watson <cjwatson@ubuntu.com>
    Architecture: amd64
    Source: openssh
    Version: 1:5.9p1-5ubuntu1
    ...
    
  • Testing download of deb source for this apt-provided package:
    control,[~],12:30(0)$ mkdir tmp
    control,[~],12:34(0)$ cd tmp/
    
    control,[~/tmp],12:34(0)$ apt-get --download-only source openssh-server=1:5.9p1-5ubuntu1
    Reading package lists... Done
    Building dependency tree       
    Reading state information... Done
    Picking 'openssh' as source package instead of 'openssh-server'
    NOTICE: 'openssh' packaging is maintained in the 'Bzr' version control system at:
    http://anonscm.debian.org/bzr/pkg-ssh/openssh/trunk
    Please use:
    bzr branch http://anonscm.debian.org/bzr/pkg-ssh/openssh/trunk
    to retrieve the latest (possibly unreleased) updates to the package.
    Need to get 1,363 kB of source archives.
    Get:1 http://us.archive.ubuntu.com/ubuntu/ precise/main openssh 1:5.9p1-5ubuntu1 (dsc) [2,651 B]
    Get:2 http://us.archive.ubuntu.com/ubuntu/ precise/main openssh 1:5.9p1-5ubuntu1 (tar) [1,110 kB]
    Get:3 http://us.archive.ubuntu.com/ubuntu/ precise/main openssh 1:5.9p1-5ubuntu1 (diff) [251 kB]
    Fetched 1,363 kB in 1s (827 kB/s)
    Download complete and in download only mode
    
    control,[~/tmp],12:34(0)$ ls
    openssh_5.9p1-5ubuntu1.debian.tar.gz  openssh_5.9p1.orig.tar.gz
    openssh_5.9p1-5ubuntu1.dsc
    
    control,[~/tmp],12:34(0)$ file *
    openssh_5.9p1-5ubuntu1.debian.tar.gz: gzip compressed data, from Unix, max compression
    openssh_5.9p1-5ubuntu1.dsc:           ASCII text
    openssh_5.9p1.orig.tar.gz:            gzip compressed data, extra field, from Unix
    
    The first tarball is the control files and sources patches. The second tarball is the original OpenSSH source code which was used.

Step 6: identify network-listening software on an OpenVZ experimental node

Using:

  • Using netstat, enumerate processes on an allocated node running the OpenVZ image which listen for network connections from outside the node
  • For each process found:
    • Use the command-line or /proc to determine what executable file is running
    • Use RPM tools to determine whether the executable file is part of an RPM
    • Otherwise, use documentation or iterate with the InstaGENI team to determine the origin of the software
  • For each RPM found, identify a location from which a source RPM for that package can be obtained
  • For each non-RPM software source found, identify a location from which the source code for that version can be obtained.

Verify:

  • The source of each network-listening file can be identified
  • RPM source packages can be found for each RPM-sourced package
  • The source code and identifiable version (e.g. a git tag) can be found for each non-RPM software source

Results of testing: 2012-05-17

Testing on pc5.utah.geniracks.net, which is operating as an OpenVZ host right now.

  • List listening processes:
    vhost1,[~],12:47(4)$ sudo netstat -lnp -A inet > netstat.listeners
    vhost1,[~],12:47(0)$ sudo netstat -lnp -A inet6 >> netstat.listeners
    
    vhost1,[~],12:49(0)$ cut -b88- netstat.listeners | awk '{print $1}' | sort -u
    -
    1286/rpcbind
    1301/sshd
    1410/rpc.statd
    1551/emulab-syncd
    17497/sshd
    17506/pubsubd
    862/ntpd
    PID/Program
    
  • Find the binaries for the processes which are identified:
    vhost1,[~],12:59(0)$ sudo ls -l /proc/{862,1286,1301,1410,1551,17497,17506}/exe
    lrwxrwxrwx 1 root root 0 May 17 12:59 /proc/1286/exe -> /sbin/rpcbind
    lrwxrwxrwx 1 root root 0 May 17 12:59 /proc/1301/exe -> /usr/sbin/sshd
    lrwxrwxrwx 1 root root 0 May 17 12:59 /proc/1410/exe -> /sbin/rpc.statd
    lrwxrwxrwx 1 root root 0 May 17 12:59 /proc/1551/exe -> /usr/local/etc/emulab/emulab-syncd
    lrwxrwxrwx 1 root root 0 May 17 12:59 /proc/17497/exe -> /mnt/pcvm5-1/root/usr/sbin/sshd
    lrwxrwxrwx 1 root root 0 May 17 12:59 /proc/17506/exe -> /mnt/pcvm5-1/root/usr/local/libexec/pubsubd
    lrwxrwxrwx 1 root root 0 May 16 15:00 /proc/862/exe -> /usr/sbin/ntpd
    
  • Some of those processes are running inside an OpenVZ container, but are identical to binaries in the base system:
    $ md5sum /mnt/pcvm5-1/root/usr/sbin/sshd /usr/sbin/sshd 
    39aceab46fa9705600dc8b194649bf9c  /mnt/pcvm5-1/root/usr/sbin/sshd
    39aceab46fa9705600dc8b194649bf9c  /usr/sbin/sshd
    
    $ md5sum /mnt/pcvm5-1/root/usr/local/libexec/pubsubd /usr/local/libexec/pubsubd 
    823239d468e277b7c634d34d82c6049a  /mnt/pcvm5-1/root/usr/local/libexec/pubsubd
    823239d468e277b7c634d34d82c6049a  /usr/local/libexec/pubsubd
    
  • Some of those processes are from RPM packages:
    • Determine which RPM each package is from:
      $ rpm -qf /sbin/rpcbind 
      rpcbind-0.2.0-10.fc15.x86_64
      
      $ rpm -qf /usr/sbin/sshd 
      openssh-server-5.6p1-34.fc15.1.x86_64
      
      $ rpm -qf /sbin/rpc.statd 
      nfs-utils-1.2.4-1.fc15.x86_64
      
      $ rpm -qf /usr/sbin/ntpd
      ntp-4.2.6p3-4.fc15.x86_64
      
    • Determine which repo each of these packages came from:
      $ yum list installed rpcbind openssh-server nfs-utils ntp
      Installed Packages
      nfs-utils.x86_64                       1:1.2.4-1.fc15                   @updates
      ntp.x86_64                             4.2.6p3-4.fc15                   @fedora 
      openssh-server.x86_64                  5.6p1-34.fc15.1                  @updates
      rpcbind.x86_64                         0.2.0-10.fc15                    @fedora 
      
    • Look in /etc/yum.repos.d/*.repo for information about the locations of RPM files:
      $ cat /etc/yum.repos.d/fedora.repo
      [fedora]
      mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch
      ...
      
      $ cat /etc/yum.repos.d/fedora-updates.repo
      [updates]
      mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=updates-released-f$releasever&arch=$basearch
      
      $ cat /etc/redhat-release
      Fedora release 15 (Lovelock)
      
      $ uname -m
      x86_64
      
    • Browse to https://mirrors.fedoraproject.org/
    • Click to https://mirrors.fedoraproject.org/publiclist/Fedora/15/x86_64/
    • Browse to a reasonable-looking mirror
    • Click through to http://mirrors.kernel.org/fedora/releases/15/Everything/source/SRPMS/
    • Download http://mirrors.kernel.org/fedora/releases/15/Everything/source/SRPMS/ntp-4.2.6p3-4.fc15.src.rpm
    • Unpack the source file with rpm2cpio:
      rpm2cpio ntp-4.2.6p3-4.fc15.src.rpm | cpio -idmv
      
    • This yields a bunch of patches and a tarball of the source used.
    • The same procedure should work for other things from fedora, and for things from updates (for the latter s/releases/updates/ when browsing through the tree of available packages).
  • Some of those files are not from RPMs:
    /usr/local/etc/emulab/emulab-syncd
    /usr/local/libexec/pubsubd
    
  • Netstat also reports some listeners with no processes:
    Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name   
    tcp        0      0 0.0.0.0:58441               0.0.0.0:*                   LISTEN      -                   
    udp        0      0 0.0.0.0:990                 0.0.0.0:*                               -                   
    udp        0      0 0.0.0.0:45938               0.0.0.0:*                               -                   
    tcp        0      0 :::57373                    :::*                        LISTEN      -                   
    udp        0      0 :::47886                    :::*                                    -                   
    
    However, lsof -i does not report these ports, so i am inclined not to worry about them, not understanding what has caused netstat to report them.

Unresolved question:

  • How do i determine what version of the Emulab source was used to create a particular image?

Step 7: verify VLANs on the rack management switch

Using:

  • Establish a privileged login to the control plane switch
  • Obtain the list of all VLAN mappings for all interfaces
  • Determine which interfaces connect to experimental nodes
  • Obtain a list of the full MAC address table of the switch
  • Use interface listings on hosts and devices to determine the identities of all MAC addresses

Verify:

  • All experimental node control interfaces are on an expected VLAN
  • It is possible to identify and classify every MAC address visible on the control switch

Results of testing: 2012-05-17

  • Telnet to procurve1 from boss
  • Look at the VLAN mappings for interfaces:
    ProCurve Switch 2610-24# show running-config 
    ...
    vlan 1 
       untagged 1-24,26-27 
       no untagged 25,28 
    
    vlan 257 
       untagged 25 
    
    vlan 260 
       untagged 28 
    
    
  • Then use show interfaces brief to find all up interfaces and their VLANs:
    ProCurve Switch 2610-24# show interfaces brief 
    
     Status and Counters - Port Status
    
                      | Intrusion                           MDI   Flow  Bcast 
      Port  Type      | Alert     Enabled Status Mode       Mode  Ctrl  Limit 
      ----- --------- + --------- ------- ------ ---------- ----- ----- ------
      1     10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
      2     10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
      3     10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
      4     10/100TX  | No        Yes     Up     100FDx     MDI   off   0     (vlan 1)
      5     10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
      6     10/100TX  | No        Yes     Up     100FDx     MDI   off   0     (vlan 1)
      7     10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
      8     10/100TX  | No        Yes     Up     100FDx     MDI   off   0     (vlan 1)
      9     10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
      10    10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
    ...
      21    10/100TX  | No        Yes     Up     100FDx     MDI   off   0     (vlan 1)
      22    10/100TX  | No        Yes     Up     100FDx     MDI   off   0     (vlan 1)
      23    10/100TX  | No        Yes     Up     100FDx     MDI   off   0     (vlan 1)
      24    10/100TX  | No        Yes     Up     100FDx     MDIX  off   0     (vlan 1)
    ...
      26    100/1000T | No        Yes     Up     1000FDx    MDIX  off   0     (vlan 1)
    ...
    
  • As you'd expect, there are no mac addresses on vlans 257 and 260:
    ProCurve Switch 2610-24# show mac-address vlan 257
    
     Status and Counters - Address Table - VLAN 257
    
      MAC Address   Port 
      ------------- -----
     
    ProCurve Switch 2610-24# show mac-address vlan 260
    
     Status and Counters - Address Table - VLAN 260
    
      MAC Address   Port 
      ------------- -----
     
    
  • Here are some MACs on vlan 1, reordered by port:
    ProCurve Switch 2610-24# show mac-address vlan 1
    
     Status and Counters - Address Table - VLAN 1
    
      MAC Address   Port 
      ------------- -----
      e4115b-ed1cb4 1    (pc5[eth0]: per login to pc5)
      e83935-ae8586 2    (pc5[iLO]: per iLO information)
      e83935-ae35a6 4    (pc2[iLO]: per iLO information) 
      e83935-aec97c 6    (pc1[iLO]: per iLO information)
      e83935-b14e88 7    (pc3[eth0]: per login to pc3)
      e83935-ae8b2a 8    (pc3[iLO]: per iLO information)
      e83935-ae5566 10   (pc4[iLO]: per iLO information)
      00009b-6201df 21   
      00009b-6224df 23   
      00009b-6224e0 23   
      e4115b-eae224 23   (control[eth0]: per login to control)
      e4115b-e6580c 24   
      0024a8-547c5e 26   
      00d0bc-f414f8 26   (155.98.31.1: per control's arp table)
    
  • On reflection, i don't think tracking down the rest of this makes much sense until we have our own rack to look at. I will ask for a wiring diagram of the Utah rack, but it's probably better to wait for the BBN rack to assess what is plugged into the control network.