[[PageOutline]] = Detailed test plan for EG-MON-1: Control Network Software and VLAN Inspection Test = ''This page is GPO's working page for performing EG-MON-1. It is public for informational purposes, but it is not an official status report. See [wiki:GENIRacksHome/ExogeniRacks/AcceptanceTestStatus] for the current status of ExoGENI acceptance tests.'' ''Last update: 2012-10-10'' == Page format == * The status chart summarizes the state of this test * The high-level description from test plan contains text copied exactly from the public test plan and acceptance criteria pages. * The steps contain things i will actually do/verify: * Steps may be composed of related substeps where i find this useful for clarity * Each step is either a preparatory step (identified by "(prep)") or a verification step (the default): * Preparatory steps are just things we have to do. They're not tests of the rack, but are prerequisites for subsequent verification steps * Verification steps are steps in which we will actually look at rack output and make sure it is as expected. They contain a '''Using:''' block, which lists the steps to run the verification, and an '''Expect:''' block which lists what outcome is expected for the test to pass. == Status of test == Meaning of states: * [[Color(lightgreen,Pass)]]: Step is completed and passed (for a verification step), or is completed (for a prep step) * [[Color(red,Fail)]]: Step is completed and failed, and is not being revisited * [[Color(#63B8FF,In Progress)]]: We are currently testing or iterating on this step * [[Color(orange,Blocked)]]: Step is blocked by some other step or activity || '''Step''' || '''State''' || '''Date completed''' || '''Open Tickets''' || '''Closed Tickets/Comments''' || || 1 || [[Color(orange,Blocked)]] || || exoticket:93 [[BR]] exoticket:96 [[BR]] exoticket:98 || complete except for issues in open tickets [[BR]] (exoticket:95) || || 2 || [[Color(lightgreen,Pass)]] || 2012-08-15 || || || || 3 ||[[Color(#63B8FF,In Progress)]]|| || || (exoticket:10) || || 4 ||[[Color(lightgreen,Pass)]]|| 2012-10-10 || || (exoticket:10) || == High-level description from test plan == This test inspects the state of the rack control network, infrastructure nodes, and system software. === Procedure === * A site administrator enumerates processes on each of the head node and an !OpenStack worker node which listen for network connections from other nodes, identifies what version of what software package is in use for each, and verifies that we know the source of each piece of software and could get access to its source code. * A site administrator reviews the configuration of the rack management switch and verifies that each worker node's control interfaces are on the expected VLANs for that worker node's function (!OpenStack or bare metal). * A site administrator reviews the MAC address table on the management switch, and verifies that all entries are identifiable and expected. === Criteria to verify as part of this test === * VI.09. A public document explains how to identify the software versions and system file configurations running on the rack, and how to get information about recent changes to the rack software and configuration. (F.5) * VI.11. A public document describes the GENI software running on the rack, and explains how to get access to the source code of each piece of GENI software. (F.6) * VII.03. Site administrators can understand the expected control and dataplane network behavior of their rack. (F.2) * VII.04. Site administrators can view and investigate current system and network activity on their rack. (F.2) * VII.06. A site administrator can verify the control software and configurations on the rack at some point in time. (F.5) * VII.08. A site administrator can get access to source code for the version of each piece of GENI code installed on their site rack at some point in time. (F.6) * VII.09. A site administrator can determine the MAC addresses of all physical host interfaces, all network device interfaces, all active experimental VMs, and all recently-terminated experimental VMs. (C.3.f) * VII.10. A site administrator can locate current and recent CPU and memory utilization for each rack network device, and can find recent changes or errors in a log. (D.6.a) * VII.12. For each infrastructure and experimental host, a site administrator can locate current and recent uptime, CPU, disk, and memory utilization, interface traffic counters, process counts, and active user counts. (D.6.b) * VII.13. A site administrator can locate recent syslogs for all infrastructure and experimental hosts. (D.6.b) == Step 1: identify network-listening software on the head node == === Overview of Step 1 === '''Using:''' * Using netstat, enumerate processes on bbn-hn which listen for network connections from outside the node * For each process found: * Use the command-line or `/proc` to determine what executable file is running * Use RPM tools to determine whether the executable file is part of an RPM * Otherwise, use documentation or iterate with the ExoGENI team to determine the origin of the software * For each RPM found, identify a location from which a source RPM for that package can be obtained * For each non-RPM software source found, identify a location from which the source code for that version can be obtained. '''Verify:''' * The source of each network-listening file can be identified * RPM source packages can be found for each RPM-sourced package * The source code and identifiable version (e.g. a git tag) can be found for each non-RPM software source === Results of Step 1 from 2012-08-03 === A list of processes that are listening: {{{ [11:11:51] jbs@bbn-hn:/home/jbs +$ sudo netstat -anp | egrep ':\*.+LISTEN' | egrep -v '(127.0.0.1|192.168.103.10|10.100.0.1|10.103.0.1)' > netstat-raw.txt [11:17:13] jbs@bbn-hn:/home/jbs +$ cat netstat-raw.txt tcp 0 0 0.0.0.0:8811 0.0.0.0:* LISTEN 29140/python tcp 0 0 192.1.242.3:8139 0.0.0.0:* LISTEN 724/ruby tcp 0 0 0.0.0.0:36843 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:875 0.0.0.0:* LISTEN 3258/rpc.rquotad tcp 0 0 0.0.0.0:8812 0.0.0.0:* LISTEN 29162/python tcp 0 0 0.0.0.0:9292 0.0.0.0:* LISTEN 28780/python tcp 0 0 0.0.0.0:47308 0.0.0.0:* LISTEN 2806/rpc.statd tcp 0 0 0.0.0.0:8813 0.0.0.0:* LISTEN 29185/python tcp 0 0 0.0.0.0:8781 0.0.0.0:* LISTEN 28606/python tcp 0 0 0.0.0.0:8814 0.0.0.0:* LISTEN 29205/python tcp 0 0 0.0.0.0:8782 0.0.0.0:* LISTEN 28701/python tcp 0 0 0.0.0.0:8815 0.0.0.0:* LISTEN 29205/python tcp 0 0 0.0.0.0:8783 0.0.0.0:* LISTEN 28814/python tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 2780/rpcbind tcp 0 0 0.0.0.0:8816 0.0.0.0:* LISTEN 29185/python tcp 0 0 0.0.0.0:8784 0.0.0.0:* LISTEN 28878/python tcp 0 0 0.0.0.0:14000 0.0.0.0:* LISTEN 19259/java tcp 0 0 0.0.0.0:8817 0.0.0.0:* LISTEN 29162/python tcp 0 0 0.0.0.0:8785 0.0.0.0:* LISTEN 28878/python tcp 0 0 0.0.0.0:4369 0.0.0.0:* LISTEN 4123/epmd tcp 0 0 0.0.0.0:10514 0.0.0.0:* LISTEN 2717/rsyslogd tcp 0 0 0.0.0.0:8818 0.0.0.0:* LISTEN 29140/python tcp 0 0 0.0.0.0:8786 0.0.0.0:* LISTEN 28814/python tcp 0 0 0.0.0.0:8787 0.0.0.0:* LISTEN 28701/python tcp 0 0 0.0.0.0:8788 0.0.0.0:* LISTEN 28606/python tcp 0 0 0.0.0.0:54004 0.0.0.0:* LISTEN 3274/rpc.mountd tcp 0 0 0.0.0.0:51413 0.0.0.0:* LISTEN 2378/java tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN 23122/vsftpd tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 3634/sshd tcp 0 0 0.0.0.0:8791 0.0.0.0:* LISTEN 28903/python tcp 0 0 0.0.0.0:11000 0.0.0.0:* LISTEN 29224/java tcp 0 0 0.0.0.0:8792 0.0.0.0:* LISTEN 28923/python tcp 0 0 0.0.0.0:49784 0.0.0.0:* LISTEN 3274/rpc.mountd tcp 0 0 0.0.0.0:8793 0.0.0.0:* LISTEN 28943/python tcp 0 0 0.0.0.0:3001 0.0.0.0:* LISTEN 23124/xcatd tcp 0 0 0.0.0.0:8794 0.0.0.0:* LISTEN 28965/python tcp 0 0 0.0.0.0:3002 0.0.0.0:* LISTEN 23127/xcatd tcp 0 0 0.0.0.0:8795 0.0.0.0:* LISTEN 28965/python tcp 0 0 0.0.0.0:8796 0.0.0.0:* LISTEN 28943/python tcp 0 0 192.1.242.3:636 0.0.0.0:* LISTEN 3199/slapd tcp 0 0 0.0.0.0:8797 0.0.0.0:* LISTEN 28923/python tcp 0 0 0.0.0.0:8798 0.0.0.0:* LISTEN 28903/python tcp 0 0 0.0.0.0:3551 0.0.0.0:* LISTEN 3660/apcupsd tcp 0 0 0.0.0.0:12000 0.0.0.0:* LISTEN 19470/java tcp 0 0 0.0.0.0:53504 0.0.0.0:* LISTEN 3274/rpc.mountd tcp 0 0 0.0.0.0:8801 0.0.0.0:* LISTEN 28987/python tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:8802 0.0.0.0:* LISTEN 29016/python tcp 0 0 0.0.0.0:8803 0.0.0.0:* LISTEN 29042/python tcp 0 0 0.0.0.0:8804 0.0.0.0:* LISTEN 29084/python tcp 0 0 0.0.0.0:43972 0.0.0.0:* LISTEN 4163/beam.smp tcp 0 0 0.0.0.0:8805 0.0.0.0:* LISTEN 29084/python tcp 0 0 192.1.242.3:3333 0.0.0.0:* LISTEN 28639/python tcp 0 0 0.0.0.0:8773 0.0.0.0:* LISTEN 27286/nginx tcp 0 0 192.1.242.3:389 0.0.0.0:* LISTEN 3199/slapd tcp 0 0 0.0.0.0:8806 0.0.0.0:* LISTEN 29042/python tcp 0 0 0.0.0.0:8807 0.0.0.0:* LISTEN 29016/python tcp 0 0 0.0.0.0:9191 0.0.0.0:* LISTEN 28826/python tcp 0 0 0.0.0.0:7911 0.0.0.0:* LISTEN 27903/dhcpd tcp 0 0 0.0.0.0:8808 0.0.0.0:* LISTEN 28987/python tcp 0 0 0.0.0.0:3626 0.0.0.0:* LISTEN 27286/nginx tcp 0 0 0.0.0.0:3306 0.0.0.0:* LISTEN 3948/mysqld tcp 0 0 :::14443 :::* LISTEN 19259/java tcp 0 0 :::51339 :::* LISTEN 3274/rpc.mountd tcp 0 0 :::46254 :::* LISTEN - tcp 0 0 :::111 :::* LISTEN 2780/rpcbind tcp 0 0 :::50000 :::* LISTEN 24491/java tcp 0 0 :::12080 :::* LISTEN 19470/java tcp 0 0 :::8080 :::* LISTEN 9114/java tcp 0 0 :::80 :::* LISTEN 8477/httpd tcp 0 0 :::50001 :::* LISTEN 1380/java tcp 0 0 :::13009 :::* LISTEN 19625/java tcp 0 0 :::10514 :::* LISTEN 2717/rsyslogd tcp 0 0 :::11443 :::* LISTEN 29224/java tcp 0 0 :::50004 :::* LISTEN 9566/java tcp 0 0 :::51413 :::* LISTEN 2378/java tcp 0 0 ::1:53 :::* LISTEN 13783/named tcp 0 0 :::22 :::* LISTEN 3634/sshd tcp 0 0 :::45430 :::* LISTEN 2806/rpc.statd tcp 0 0 :::55000 :::* LISTEN 24491/java tcp 0 0 :::13080 :::* LISTEN 19625/java tcp 0 0 :::55001 :::* LISTEN 1380/java tcp 0 0 :::14009 :::* LISTEN 19259/java tcp 0 0 ::1:25 :::* LISTEN 4095/master tcp 0 0 ::1:6010 :::* LISTEN 16835/sshd tcp 0 0 ::1:6011 :::* LISTEN 32641/sshd tcp 0 0 :::12443 :::* LISTEN 19470/java tcp 0 0 :::443 :::* LISTEN 8477/httpd tcp 0 0 :::33595 :::* LISTEN 3274/rpc.mountd tcp 0 0 :::55004 :::* LISTEN 9566/java tcp 0 0 :::55740 :::* LISTEN 9114/java tcp 0 0 :::6556 :::* LISTEN 3642/xinetd tcp 0 0 :::6557 :::* LISTEN 22989/xinetd tcp 0 0 :::14080 :::* LISTEN 19259/java tcp 0 0 :::11009 :::* LISTEN 29224/java tcp 0 0 :::2049 :::* LISTEN - tcp 0 0 :::13443 :::* LISTEN 19625/java tcp 0 0 :::58949 :::* LISTEN 3274/rpc.mountd tcp 0 0 :::11080 :::* LISTEN 29224/java tcp 0 0 :::5672 :::* LISTEN 4163/beam.smp tcp 0 0 :::12009 :::* LISTEN 19470/java tcp 0 0 :::11081 :::* LISTEN 2378/java tcp 0 0 :::6633 :::* LISTEN 9114/java }}} That's 103 things, which seems like a lot, but I gather that this is normal. Ah, and there's some duplication in there. Stripping that into a list of PIDs: {{{ [11:22:25] jbs@bbn-hn:/home/jbs +$ awk '{print $NF}' netstat-raw.txt | sed -r -e 's|/.*||' | sort -n | uniq > netstat-pids.txt [11:23:04] jbs@bbn-hn:/home/jbs +$ cat netstat-pids.txt - 724 1380 2378 2717 2780 2806 3199 3258 3274 3634 3642 3660 3948 4095 4123 4163 8477 9114 9566 13783 16835 19259 19470 19625 22989 23122 23124 23127 24491 27286 27903 28606 28639 28701 28780 28814 28826 28878 28903 28923 28943 28965 28987 29016 29042 29084 29140 29162 29185 29205 29224 32641 }}} For some of those, the PID is '-': {{{ [11:25:15] jbs@bbn-hn:/home/jbs +$ egrep -- - netstat-raw.txt tcp 0 0 0.0.0.0:36843 0.0.0.0:* LISTEN - tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN - tcp 0 0 :::46254 :::* LISTEN - tcp 0 0 :::2049 :::* LISTEN - }}} lsof doesn't know what they are either: {{{ [11:26:32] jbs@bbn-hn:/home/jbs +$ sudo lsof -i | grep 2049 [11:27:15] jbs@bbn-hn:/home/jbs +$ sudo lsof -i | grep 46254 [11:27:26] jbs@bbn-hn:/home/jbs +$ sudo lsof -i | grep 36843 }}} We think those are probably NFS-related things, which perhaps are listening from a kernel module rather than a process. Leaving those aside, here are the others: {{{ [11:40:08] jbs@bbn-hn:/home/jbs +$ for pid in $(cat netstat-pids.txt) ; do ps -efwww | egrep "^[^ ]+ +$pid" ; done >& netstat-commands.txt [11:40:11] jbs@bbn-hn:/home/jbs +$ cat netstat-commands.txt root 724 1 0 Jul03 ? 02:44:09 /usr/bin/ruby /usr/sbin/puppetd 2006 1380 1 0 Jul26 ? 00:06:48 java -server -d64 -Xmx2g -Xms2g -Xmn800m -XX:+UseParallelGC -XX:+AggressiveOpts -XX:+UseFastAccessorMethods -XX:InlineSmallCode=8192 -XX:MaxInlineSize=8192 -XX:FreqInlineSize=8192 -XX:CompileThreshold=1500 -XX:PreBlockSpin=8 -Dnet.orca.vlan=vlan-2601- -Dnet.floodlightcontroller.restserver.RestApiServer.port=55001 -Dnet.floodlightcontroller.core.FloodlightProvider.openflowport=50001 -Dfloodlight.modules=net.floodlightcontroller.learningswitch.LearningSwitch,net.floodlightcontroller.counter.NullCounterStore,net.floodlightcontroller.perfmon.NullPktInProcessingTime -jar /opt/floodlight/floodlight.jar 2006 2378 2372 0 Jul20 ? 00:08:06 java -classpath /opt/imageproxy/axis2:/opt/imageproxy/axis2/conf:/usr/java/latest/lib/tools.jar::/opt/imageproxy/axis2/lib/activation-1.1.jar:/opt/imageproxy/axis2/lib/antlr-2.7.7.jar:/opt/imageproxy/axis2/lib/apache-mime4j-core-0.7.2.jar:/opt/imageproxy/axis2/lib/axiom-api-1.2.13.jar:/opt/imageproxy/axis2/lib/axiom-dom-1.2.13.jar:/opt/imageproxy/axis2/lib/axiom-impl-1.2.13.jar:/opt/imageproxy/axis2/lib/axis2-adb-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-adb-codegen-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-ant-plugin-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-clustering-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-codegen-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-corba-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-fastinfoset-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-java2wsdl-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-jaxbri-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-jaxws-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-jibx-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-json-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-kernel-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-metadata-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-mtompolicy-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-saaj-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-soapmonitor-servlet-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-spring-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-transport-http-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-transport-local-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-xmlbeans-1.6.2.jar:/opt/imageproxy/axis2/lib/bcel-5.1.jar:/opt/imageproxy/axis2/lib/commons-cli-1.2.jar:/opt/imageproxy/axis2/lib/commons-codec-1.3.jar:/opt/imageproxy/axis2/lib/commons-fileupload-1.2.jar:/opt/imageproxy/axis2/lib/commons-httpclient-3.1.jar:/opt/imageproxy/axis2/lib/commons-io-1.4.jar:/opt/imageproxy/axis2/lib/commons-logging-1.1.1.jar:/opt/imageproxy/axis2/lib/geronimo-annotation_1.0_spec-1.1.jar:/opt/imageproxy/axis2/lib/geronimo-jaxws_2.2_spec-1.0.jar:/opt/imageproxy/axis2/lib/geronimo-jta_1.1_spec-1.1.jar:/opt/imageproxy/axis2/lib/geronimo-saaj_1.3_spec-1.0.1.jar:/opt/imageproxy/axis2/lib/geronimo-stax-api_1.0_spec-1.0.1.jar:/opt/imageproxy/axis2/lib/geronimo-ws-metadata_2.0_spec-1.1.2.jar:/opt/imageproxy/axis2/lib/httpcore-4.0.jar:/opt/imageproxy/axis2/lib/jalopy-1.5rc3.jar:/opt/imageproxy/axis2/lib/jaxb-api-2.1.jar:/opt/imageproxy/axis2/lib/jaxb-impl-2.1.7.jar:/opt/imageproxy/axis2/lib/jaxb-xjc-2.1.7.jar:/opt/imageproxy/axis2/lib/jaxen-1.1.1.jar:/opt/imageproxy/axis2/lib/jaxws-tools-2.1.3.jar:/opt/imageproxy/axis2/lib/jettison-1.0-RC2.jar:/opt/imageproxy/axis2/lib/jibx-bind-1.2.jar:/opt/imageproxy/axis2/lib/jibx-run-1.2.jar:/opt/imageproxy/axis2/lib/jsr311-api-1.0.jar:/opt/imageproxy/axis2/lib/juli-6.0.16.jar:/opt/imageproxy/axis2/lib/log4j-1.2.15.jar:/opt/imageproxy/axis2/lib/mail-1.4.jar:/opt/imageproxy/axis2/lib/mex-1.6.2-impl.jar:/opt/imageproxy/axis2/lib/neethi-3.0.2.jar:/opt/imageproxy/axis2/lib/regexp-1.2.jar:/opt/imageproxy/axis2/lib/tribes-6.0.16.jar:/opt/imageproxy/axis2/lib/woden-api-1.0M9.jar:/opt/imageproxy/axis2/lib/woden-impl-commons-1.0M9.jar:/opt/imageproxy/axis2/lib/woden-impl-dom-1.0M9.jar:/opt/imageproxy/axis2/lib/wsdl4j-1.6.2.jar:/opt/imageproxy/axis2/lib/wstx-asl-3.2.9.jar:/opt/imageproxy/axis2/lib/xalan-2.7.0.jar:/opt/imageproxy/axis2/lib/xmlbeans-2.3.0.jar:/opt/imageproxy/axis2/lib/xml-resolver-1.2.jar:/opt/imageproxy/axis2/lib/XmlSchema-1.4.7.jar: -Djava.endorsed.dirs=/opt/imageproxy/axis2/lib/endorsed:/usr/java/latest/jre/lib/endorsed:/usr/java/latest/lib/endorsed org.apache.axis2.transport.SimpleAxis2Server -repo /opt/imageproxy/axis2/repository -conf /opt/imageproxy/axis2/conf/axis2.xml root 2717 1 0 Jun13 ? 00:09:00 /sbin/rsyslogd -i /var/run/syslogd.pid -c 4 rpc 2780 1 0 Jun13 ? 00:00:02 rpcbind rpcuser 2806 1 0 Jun13 ? 00:00:00 rpc.statd ldap 3199 1 0 Jun13 ? 00:10:20 /usr/sbin/slapd -h ldap://192.1.242.3 ldaps://192.1.242.3 ldap://192.168.103.10 ldaps://192.168.103.10 ldap://10.100.0.1 ldaps://10.100.0.1 -u ldap -f /etc/openldap/slapd.conf root 3258 1 0 Jun13 ? 00:00:00 rpc.rquotad root 3274 1 0 Jun13 ? 00:00:00 rpc.mountd root 3634 1 0 Jun13 ? 00:01:34 /usr/sbin/sshd root 3642 1 0 Jun13 ? 00:00:18 xinetd -stayalive -pidfile /var/run/xinetd.pid root 3660 1 0 Jun13 ? 00:04:13 /sbin/apcupsd -f /etc/apcupsd/ups06.conf -P /var/run/apcupsd-ups06.pid mysql 3948 3706 0 Jun13 ? 02:05:09 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --open-files-limit=8192 --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock --port=3306 root 4095 1 0 Jun13 ? 00:00:28 /usr/libexec/postfix/master rabbitmq 4123 1 0 Jun13 ? 00:00:16 /usr/lib64/erlang/erts-5.8.5/bin/epmd -daemon rabbitmq 4163 4162 0 Jun13 ? 00:18:00 /usr/lib64/erlang/erts-5.8.5/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -noshell -noinput -sname rabbit@bbn-hn -boot /var/lib/rabbitmq/mnesia/rabbit@bbn-hn-plugins-expand/rabbit -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -kernel error_logger {file,"/var/log/rabbitmq/rabbit@bbn-hn.log"} -sasl sasl_error_logger {file,"/var/log/rabbitmq/rabbit@bbn-hn-sasl.log"} -os_mon start_cpu_sup true -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@bbn-hn" root 8477 1 0 Jun22 ? 00:00:42 /usr/sbin/httpd 491 9114 1 3 Jun27 ? 1-10:41:10 java -server -Xms256M -Xmx1500M -XX:OnError=flowvisor-crash-logger -XX:+UseConcMarkSweepGC -Dorg.flowvisor.config_dir=/opt/flowvisor/etc/flowvisor -Dorg.flowvisor.install_dir=/opt/flowvisor/libexec/flowvisor -Djavax.net.ssl.keyStore=/opt/flowvisor/etc/flowvisor/mySSLKeyStore -Djavax.net.ssl.keyStorePassword=CHANGEME_PASSWD -cp /opt/flowvisor/libexec/flowvisor/openflow.jar:/opt/flowvisor/libexec/flowvisor/xmlrpc-client-3.1.3.jar:/opt/flowvisor/libexec/flowvisor/xmlrpc-common-3.1.3.jar:/opt/flowvisor/libexec/flowvisor/xmlrpc-server-3.1.3.jar:/opt/flowvisor/libexec/flowvisor/commons-logging-1.1.jar:/opt/flowvisor/libexec/flowvisor/ws-commons-util-1.0.2.jar:/opt/flowvisor/libexec/flowvisor/jsse.jar:/opt/flowvisor/libexec/flowvisor/asm-3.0.jar:/opt/flowvisor/libexec/flowvisor/cglib-2.2.jar:/opt/flowvisor/libexec/flowvisor/commons-codec-1.4.jar:/opt/flowvisor/libexec/flowvisor/gson-1.7.1.jar:/opt/flowvisor/libexec/flowvisor/jetty-continuation-7.0.2.v20100331.jar:/opt/flowvisor/libexec/flowvisor/jetty-http-7.0.2.v20100331.jar:/opt/flowvisor/libexec/flowvisor/jetty-io-7.0.2.v20100331.jar:/opt/flowvisor/libexec/flowvisor/jetty-security-7.0.2.v20100331.jar:/opt/flowvisor/libexec/flowvisor/jetty-server-7.0.2.v20100331.jar:/opt/flowvisor/libexec/flowvisor/jetty-util-7.0.2.v20100331.jar:/opt/flowvisor/libexec/flowvisor/servlet-api-2.5.jar:/opt/flowvisor/libexec/flowvisor/flowvisor.jar org.flowvisor.FlowVisor /opt/flowvisor/etc/flowvisor/config.xml 2006 9566 1 0 Aug01 ? 00:01:35 java -server -d64 -Xmx2g -Xms2g -Xmn800m -XX:+UseParallelGC -XX:+AggressiveOpts -XX:+UseFastAccessorMethods -XX:InlineSmallCode=8192 -XX:MaxInlineSize=8192 -XX:FreqInlineSize=8192 -XX:CompileThreshold=1500 -XX:PreBlockSpin=8 -Dnet.orca.vlan=vlan-5- -Dnet.floodlightcontroller.restserver.RestApiServer.port=55004 -Dnet.floodlightcontroller.core.FloodlightProvider.openflowport=50004 -Dfloodlight.modules=net.floodlightcontroller.learningswitch.LearningSwitch,net.floodlightcontroller.counter.NullCounterStore,net.floodlightcontroller.perfmon.NullPktInProcessingTime -jar /opt/floodlight/floodlight.jar named 13783 1 0 Jun22 ? 00:01:04 /usr/sbin/named -u named jonmills 16835 16833 0 00:09 ? 00:00:00 sshd: jonmills@pts/2 2006 19259 1 0 Jul05 ? 00:24:49 /usr/java/latest/bin/java -Djava.util.logging.config.file=/opt/orca-aux/tomcat-7/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:MaxPermSize=128m -Xmx1024m -Djavax.net.ssl.trustStore=/opt/orca-aux/tomcat-7/ssl/trusted.jks -Djavax.net.ssl.trustStorePassword=orcaorca -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=14000 -Djava.endorsed.dirs=/opt/orca-aux/tomcat-7/endorsed -classpath /opt/orca-aux/tomcat-7/bin/bootstrap.jar:/opt/orca-aux/tomcat-7/bin/tomcat-juli.jar -Dcatalina.base=/opt/orca-aux/tomcat-7 -Dcatalina.home=/opt/orca-aux/tomcat-7 -Djava.io.tmpdir=/opt/orca-aux/tomcat-7/temp org.apache.catalina.startup.Bootstrap start 2006 19470 1 0 Jul20 ? 01:02:40 /usr/java/latest/bin/java -Djava.util.logging.config.file=/opt/orca-12080/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xmx1024m -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=12000 -Djava.endorsed.dirs=/opt/orca-12080/tomcat/endorsed -classpath /opt/orca-12080/tomcat/bin/bootstrap.jar:/opt/orca-12080/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/orca-12080/tomcat -Dcatalina.home=/opt/orca-12080/tomcat -Djava.io.tmpdir=/opt/orca-12080/tomcat/temp org.apache.catalina.startup.Bootstrap start 2006 19625 1 0 Jul20 ? 00:29:16 /usr/java/latest/bin/java -Djava.util.logging.config.file=/opt/orca-13080/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/opt/orca-13080/tomcat/endorsed -classpath /opt/orca-13080/tomcat/bin/bootstrap.jar:/opt/orca-13080/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/orca-13080/tomcat -Dcatalina.home=/opt/orca-13080/tomcat -Djava.io.tmpdir=/opt/orca-13080/tomcat/temp org.apache.catalina.startup.Bootstrap start rack_bbn 22989 1 0 Jun18 ? 00:00:24 /omd/sites/rack_bbn/var/tmp/xinetd -pidfile /omd/sites/rack_bbn/tmp/run/xinetd.pid -filelog /omd/sites/rack_bbn/var/log/xinetd.log -f /omd/sites/rack_bbn/etc/xinetd.conf root 23122 1 0 Jun22 ? 00:00:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf root 23124 1 0 Jun22 ? 00:00:06 xcatd: SSL listener root 23127 23124 0 Jun22 ? 00:00:01 xcatd: install monitor 2006 24491 1 0 Aug01 ? 00:01:25 java -server -d64 -Xmx2g -Xms2g -Xmn800m -XX:+UseParallelGC -XX:+AggressiveOpts -XX:+UseFastAccessorMethods -XX:InlineSmallCode=8192 -XX:MaxInlineSize=8192 -XX:FreqInlineSize=8192 -XX:CompileThreshold=1500 -XX:PreBlockSpin=8 -Dnet.orca.vlan=vlan-2- -Dnet.floodlightcontroller.restserver.RestApiServer.port=55000 -Dnet.floodlightcontroller.core.FloodlightProvider.openflowport=50000 -Dfloodlight.modules=net.floodlightcontroller.learningswitch.LearningSwitch,net.floodlightcontroller.counter.NullCounterStore,net.floodlightcontroller.perfmon.NullPktInProcessingTime -jar /opt/floodlight/floodlight.jar root 27286 1 0 Jun18 ? 00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf dhcpd 27903 1 0 Jun22 ? 00:02:12 /usr/sbin/dhcpd -user dhcpd -group dhcpd bond2.1007 bond2.1006 nova 28606 28603 0 Jul06 ? 00:06:51 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8781.log --ec2_listen_port=8781 --osapi_listen_port=8788 nova 28639 1 0 Jul06 ? 00:00:07 /usr/bin/python /usr/bin/nova-objectstore --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-objectstore.log --pidfile=/var/run/nova/nova-objectstore.pid nova 28701 28699 0 Jul06 ? 00:06:46 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8782.log --ec2_listen_port=8782 --osapi_listen_port=8787 glance 28780 1 0 Jul06 ? 00:00:54 /usr/bin/python /usr/bin/glance-api --config-file=/etc/glance/glance-api.conf --log-config=/etc/glance/logging-api.conf nova 28814 28812 0 Jul06 ? 00:06:32 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8783.log --ec2_listen_port=8783 --osapi_listen_port=8786 glance 28826 1 0 Jul06 ? 00:00:50 /usr/bin/python /usr/bin/glance-registry --config-file=/etc/glance/glance-registry.conf -v --log-config=/etc/glance/logging-registry.conf nova 28878 28876 0 Jul06 ? 00:06:09 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8784.log --ec2_listen_port=8784 --osapi_listen_port=8785 nova 28903 28901 0 Jul06 ? 00:06:12 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8791.log --ec2_listen_port=8791 --osapi_listen_port=8798 nova 28923 28921 0 Jul06 ? 00:06:49 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8792.log --ec2_listen_port=8792 --osapi_listen_port=8797 nova 28943 28941 0 Jul06 ? 00:06:28 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8793.log --ec2_listen_port=8793 --osapi_listen_port=8796 nova 28965 28963 0 Jul06 ? 00:07:28 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8794.log --ec2_listen_port=8794 --osapi_listen_port=8795 nova 28987 28985 0 Jul06 ? 00:06:01 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8801.log --ec2_listen_port=8801 --osapi_listen_port=8808 nova 29016 29014 0 Jul06 ? 00:06:33 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8802.log --ec2_listen_port=8802 --osapi_listen_port=8807 nova 29042 29040 0 Jul06 ? 00:09:05 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8803.log --ec2_listen_port=8803 --osapi_listen_port=8806 nova 29084 29080 0 Jul06 ? 00:06:34 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8804.log --ec2_listen_port=8804 --osapi_listen_port=8805 nova 29140 29138 0 Jul06 ? 00:06:11 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8811.log --ec2_listen_port=8811 --osapi_listen_port=8818 nova 29162 29160 0 Jul06 ? 00:06:29 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8812.log --ec2_listen_port=8812 --osapi_listen_port=8817 nova 29185 29183 0 Jul06 ? 00:06:42 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8813.log --ec2_listen_port=8813 --osapi_listen_port=8816 nova 29205 29203 0 Jul06 ? 00:06:47 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8814.log --ec2_listen_port=8814 --osapi_listen_port=8815 2006 29224 1 0 Jul20 ? 00:29:38 /usr/java/latest/bin/java -Djava.util.logging.config.file=/opt/orca-11080/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xmx1024m -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=11000 -Djava.endorsed.dirs=/opt/orca-11080/tomcat/endorsed -classpath /opt/orca-11080/tomcat/bin/bootstrap.jar:/opt/orca-11080/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/orca-11080/tomcat -Dcatalina.home=/opt/orca-11080/tomcat -Djava.io.tmpdir=/opt/orca-11080/tomcat/temp org.apache.catalina.startup.Bootstrap start jonmills 32641 32639 0 10:09 ? 00:00:00 sshd: jonmills@pts/1 }}} Just to sanity check that we haven't missed anything: {{{ [11:40:14] jbs@bbn-hn:/home/jbs +$ wc -l netstat-* 52 netstat-commands.txt 53 netstat-pids.txt 103 netstat-raw.txt 208 total }}} The change from raw to pids is because of duplicates, and from pids to commands is because of the '-' PID dropping out. So, let's start plowing through those. This is from an RPM: {{{ root 724 1 0 Jul03 ? 02:44:09 /usr/bin/ruby /usr/sbin/puppetd [11:41:39] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/sbin/puppetd puppet-2.7.18-1.el6.noarch }}} Lumping these Floodlight things together, Java is from an RPM, but floodlight.jar isn't: {{{ 2006 1380 1 0 Jul26 ? 00:06:48 java -server -d64 -Xmx2g -Xms2g -Xmn800m -XX:+UseParallelGC -XX:+AggressiveOpts -XX:+UseFastAccessorMethods -XX:InlineSmallCode=8192 -XX:MaxInlineSize=8192 -XX:FreqInlineSize=8192 -XX:CompileThreshold=1500 -XX:PreBlockSpin=8 -Dnet.orca.vlan=vlan-2601- -Dnet.floodlightcontroller.restserver.RestApiServer.port=55001 -Dnet.floodlightcontroller.core.FloodlightProvider.openflowport=50001 -Dfloodlight.modules=net.floodlightcontroller.learningswitch.LearningSwitch,net.floodlightcontroller.counter.NullCounterStore,net.floodlightcontroller.perfmon.NullPktInProcessingTime -jar /opt/floodlight/floodlight.jar 2006 9566 1 0 Aug01 ? 00:01:35 java -server -d64 -Xmx2g -Xms2g -Xmn800m -XX:+UseParallelGC -XX:+AggressiveOpts -XX:+UseFastAccessorMethods -XX:InlineSmallCode=8192 -XX:MaxInlineSize=8192 -XX:FreqInlineSize=8192 -XX:CompileThreshold=1500 -XX:PreBlockSpin=8 -Dnet.orca.vlan=vlan-5- -Dnet.floodlightcontroller.restserver.RestApiServer.port=55004 -Dnet.floodlightcontroller.core.FloodlightProvider.openflowport=50004 -Dfloodlight.modules=net.floodlightcontroller.learningswitch.LearningSwitch,net.floodlightcontroller.counter.NullCounterStore,net.floodlightcontroller.perfmon.NullPktInProcessingTime -jar /opt/floodlight/floodlight.jar 2006 24491 1 0 Aug01 ? 00:01:25 java -server -d64 -Xmx2g -Xms2g -Xmn800m -XX:+UseParallelGC -XX:+AggressiveOpts -XX:+UseFastAccessorMethods -XX:InlineSmallCode=8192 -XX:MaxInlineSize=8192 -XX:FreqInlineSize=8192 -XX:CompileThreshold=1500 -XX:PreBlockSpin=8 -Dnet.orca.vlan=vlan-2- -Dnet.floodlightcontroller.restserver.RestApiServer.port=55000 -Dnet.floodlightcontroller.core.FloodlightProvider.openflowport=50000 -Dfloodlight.modules=net.floodlightcontroller.learningswitch.LearningSwitch,net.floodlightcontroller.counter.NullCounterStore,net.floodlightcontroller.perfmon.NullPktInProcessingTime -jar /opt/floodlight/floodlight.jar [15:56:51] jbs@bbn-hn:/home/jbs +$ which java java is /usr/java/default/bin/java [15:56:52] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/java/default/bin/java jdk-1.6.0_30-fcs.x86_64 [15:56:56] jbs@bbn-hn:/home/jbs +$ rpm -qf /opt/floodlight/floodlight.jar file /opt/floodlight/floodlight.jar is not owned by any package }}} Java is from an RPM (see above), imageproxy isn't: {{{ 2006 2378 2372 0 Jul20 ? 00:08:06 java -classpath /opt/imageproxy/axis2:/opt/imageproxy/axis2/conf:/usr/java/latest/lib/tools.jar::/opt/imageproxy/axis2/lib/activation-1.1.jar:/opt/imageproxy/axis2/lib/antlr-2.7.7.jar:/opt/imageproxy/axis2/lib/apache-mime4j-core-0.7.2.jar:/opt/imageproxy/axis2/lib/axiom-api-1.2.13.jar:/opt/imageproxy/axis2/lib/axiom-dom-1.2.13.jar:/opt/imageproxy/axis2/lib/axiom-impl-1.2.13.jar:/opt/imageproxy/axis2/lib/axis2-adb-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-adb-codegen-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-ant-plugin-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-clustering-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-codegen-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-corba-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-fastinfoset-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-java2wsdl-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-jaxbri-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-jaxws-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-jibx-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-json-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-kernel-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-metadata-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-mtompolicy-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-saaj-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-soapmonitor-servlet-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-spring-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-transport-http-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-transport-local-1.6.2.jar:/opt/imageproxy/axis2/lib/axis2-xmlbeans-1.6.2.jar:/opt/imageproxy/axis2/lib/bcel-5.1.jar:/opt/imageproxy/axis2/lib/commons-cli-1.2.jar:/opt/imageproxy/axis2/lib/commons-codec-1.3.jar:/opt/imageproxy/axis2/lib/commons-fileupload-1.2.jar:/opt/imageproxy/axis2/lib/commons-httpclient-3.1.jar:/opt/imageproxy/axis2/lib/commons-io-1.4.jar:/opt/imageproxy/axis2/lib/commons-logging-1.1.1.jar:/opt/imageproxy/axis2/lib/geronimo-annotation_1.0_spec-1.1.jar:/opt/imageproxy/axis2/lib/geronimo-jaxws_2.2_spec-1.0.jar:/opt/imageproxy/axis2/lib/geronimo-jta_1.1_spec-1.1.jar:/opt/imageproxy/axis2/lib/geronimo-saaj_1.3_spec-1.0.1.jar:/opt/imageproxy/axis2/lib/geronimo-stax-api_1.0_spec-1.0.1.jar:/opt/imageproxy/axis2/lib/geronimo-ws-metadata_2.0_spec-1.1.2.jar:/opt/imageproxy/axis2/lib/httpcore-4.0.jar:/opt/imageproxy/axis2/lib/jalopy-1.5rc3.jar:/opt/imageproxy/axis2/lib/jaxb-api-2.1.jar:/opt/imageproxy/axis2/lib/jaxb-impl-2.1.7.jar:/opt/imageproxy/axis2/lib/jaxb-xjc-2.1.7.jar:/opt/imageproxy/axis2/lib/jaxen-1.1.1.jar:/opt/imageproxy/axis2/lib/jaxws-tools-2.1.3.jar:/opt/imageproxy/axis2/lib/jettison-1.0-RC2.jar:/opt/imageproxy/axis2/lib/jibx-bind-1.2.jar:/opt/imageproxy/axis2/lib/jibx-run-1.2.jar:/opt/imageproxy/axis2/lib/jsr311-api-1.0.jar:/opt/imageproxy/axis2/lib/juli-6.0.16.jar:/opt/imageproxy/axis2/lib/log4j-1.2.15.jar:/opt/imageproxy/axis2/lib/mail-1.4.jar:/opt/imageproxy/axis2/lib/mex-1.6.2-impl.jar:/opt/imageproxy/axis2/lib/neethi-3.0.2.jar:/opt/imageproxy/axis2/lib/regexp-1.2.jar:/opt/imageproxy/axis2/lib/tribes-6.0.16.jar:/opt/imageproxy/axis2/lib/woden-api-1.0M9.jar:/opt/imageproxy/axis2/lib/woden-impl-commons-1.0M9.jar:/opt/imageproxy/axis2/lib/woden-impl-dom-1.0M9.jar:/opt/imageproxy/axis2/lib/wsdl4j-1.6.2.jar:/opt/imageproxy/axis2/lib/wstx-asl-3.2.9.jar:/opt/imageproxy/axis2/lib/xalan-2.7.0.jar:/opt/imageproxy/axis2/lib/xmlbeans-2.3.0.jar:/opt/imageproxy/axis2/lib/xml-resolver-1.2.jar:/opt/imageproxy/axis2/lib/XmlSchema-1.4.7.jar: -Djava.endorsed.dirs=/opt/imageproxy/axis2/lib/endorsed:/usr/java/latest/jre/lib/endorsed:/usr/java/latest/lib/endorsed org.apache.axis2.transport.SimpleAxis2Server -repo /opt/imageproxy/axis2/repository -conf /opt/imageproxy/axis2/conf/axis2.xml [15:57:00] jbs@bbn-hn:/home/jbs +$ rpm -qf /opt/imageproxy file /opt/imageproxy is not owned by any package }}} This is from an RPM: {{{ root 2717 1 0 Jun13 ? 00:09:00 /sbin/rsyslogd -i /var/run/syslogd.pid -c 4 [15:59:03] jbs@bbn-hn:/home/jbs +$ rpm -qf /sbin/rsyslogd rsyslog-4.6.2-12.el6.x86_64 }}} This is from an RPM: {{{ rpc 2780 1 0 Jun13 ? 00:00:02 rpcbind [16:00:13] jbs@bbn-hn:/home/jbs +$ which rpcbind rpcbind is /sbin/rpcbind [16:00:28] jbs@bbn-hn:/home/jbs +$ rpm -qf /sbin/rpcbind rpcbind-0.2.0-8.el6.x86_64 }}} This is from an RPM: {{{ rpcuser 2806 1 0 Jun13 ? 00:00:00 rpc.statd [16:00:32] jbs@bbn-hn:/home/jbs +$ which rpc.statd rpc.statd is /sbin/rpc.statd [16:00:52] jbs@bbn-hn:/home/jbs +$ rpm -qf /sbin/rpc.statd nfs-utils-1.2.3-15.el6.x86_64 }}} This is from an RPM: {{{ ldap 3199 1 0 Jun13 ? 00:10:20 /usr/sbin/slapd -h ldap://192.1.242.3 ldaps://192.1.242.3 ldap://192.168.103.10 ldaps://192.168.103.10 ldap://10.100.0.1 ldaps://10.100.0.1 -u ldap -f /etc/openldap/slapd.conf [16:00:58] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/sbin/slapd openldap-servers-2.4.23-20.el6.x86_64 }}} This is from an RPM: {{{ root 3258 1 0 Jun13 ? 00:00:00 rpc.rquotad [16:01:28] jbs@bbn-hn:/home/jbs +$ which rpc.rquotad rpc.rquotad is /usr/sbin/rpc.rquotad [16:01:34] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/sbin/rpc.rquotad quota-3.17-16.el6.x86_64 }}} This is from an RPM: {{{ root 3274 1 0 Jun13 ? 00:00:00 rpc.mountd [16:01:40] jbs@bbn-hn:/home/jbs +$ which rpc.mountd rpc.mountd is /usr/sbin/rpc.mountd [16:02:24] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/sbin/rpc.mountd nfs-utils-1.2.3-15.el6.x86_64 }}} This is from an RPM: {{{ root 3634 1 0 Jun13 ? 00:01:34 /usr/sbin/sshd [16:06:46] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/sbin/sshd openssh-server-5.3p1-70.el6_2.2.x86_64 }}} This is from an RPM: {{{ root 3642 1 0 Jun13 ? 00:00:18 xinetd -stayalive -pidfile /var/run/xinetd.pid [16:07:15] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/sbin/xinetd xinetd-2.3.14-33.el6.x86_64 }}} This is from an RPM: {{{ root 3660 1 0 Jun13 ? 00:04:13 /sbin/apcupsd -f /etc/apcupsd/ups06.conf -P /var/run/apcupsd-ups06.pid [16:07:45] jbs@bbn-hn:/home/jbs +$ rpm -qf /sbin/apcupsd apcupsd-3.14.8-4.el6.x86_64 }}} This is from an RPM: {{{ mysql 3948 3706 0 Jun13 ? 02:05:09 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --open-files-limit=8192 --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock --port=3306 [16:08:22] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/libexec/mysqld mysql-server-5.1.61-1.el6_2.1.x86_64 }}} This is from an RPM: {{{ root 4095 1 0 Jun13 ? 00:00:28 /usr/libexec/postfix/master [16:08:28] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/libexec/postfix/master postfix-2.6.6-2.2.el6_1.x86_64 }}} This is from an RPM: {{{ rabbitmq 4123 1 0 Jun13 ? 00:00:16 /usr/lib64/erlang/erts-5.8.5/bin/epmd -daemon [16:09:09] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/lib64/erlang/erts-5.8.5/bin/epmd erlang-erts-R14B-04.1.el6.x86_64 }}} This is from an RPM: {{{ rabbitmq 4163 4162 0 Jun13 ? 00:18:00 /usr/lib64/erlang/erts-5.8.5/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -noshell -noinput -sname rabbit@bbn-hn -boot /var/lib/rabbitmq/mnesia/rabbit@bbn-hn-plugins-expand/rabbit -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -kernel error_logger {file,"/var/log/rabbitmq/rabbit@bbn-hn.log"} -sasl sasl_error_logger {file,"/var/log/rabbitmq/rabbit@bbn-hn-sasl.log"} -os_mon start_cpu_sup true -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@bbn-hn" [16:09:46] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/lib64/erlang/erts-5.8.5/bin/beam.smp erlang-erts-R14B-04.1.el6.x86_64 [16:10:42] jbs@bbn-hn:/home/jbs +$ rpm -qf /var/lib/rabbitmq rabbitmq-server-2.6.1-1.el6.noarch }}} (I don't really know what that is; I didn't check every path or file on that commandline, but I'm optimistic that they're all from RPMs.) This is from an RPM: {{{ root 8477 1 0 Jun22 ? 00:00:42 /usr/sbin/httpd [16:13:19] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/sbin/httpd httpd-2.2.15-15.el6.centos.1.x86_64 }}} Java is from an RPM (see above), and so is FlowVisor: {{{ 491 9114 1 3 Jun27 ? 1-10:41:10 java -server -Xms256M -Xmx1500M -XX:OnError=flowvisor-crash-logger -XX:+UseConcMarkSweepGC -Dorg.flowvisor.config_dir=/opt/flowvisor/etc/flowvisor -Dorg.flowvisor.install_dir=/opt/flowvisor/libexec/flowvisor -Djavax.net.ssl.keyStore=/opt/flowvisor/etc/flowvisor/mySSLKeyStore -Djavax.net.ssl.keyStorePassword=CHANGEME_PASSWD -cp /opt/flowvisor/libexec/flowvisor/openflow.jar:/opt/flowvisor/libexec/flowvisor/xmlrpc-client-3.1.3.jar:/opt/flowvisor/libexec/flowvisor/xmlrpc-common-3.1.3.jar:/opt/flowvisor/libexec/flowvisor/xmlrpc-server-3.1.3.jar:/opt/flowvisor/libexec/flowvisor/commons-logging-1.1.jar:/opt/flowvisor/libexec/flowvisor/ws-commons-util-1.0.2.jar:/opt/flowvisor/libexec/flowvisor/jsse.jar:/opt/flowvisor/libexec/flowvisor/asm-3.0.jar:/opt/flowvisor/libexec/flowvisor/cglib-2.2.jar:/opt/flowvisor/libexec/flowvisor/commons-codec-1.4.jar:/opt/flowvisor/libexec/flowvisor/gson-1.7.1.jar:/opt/flowvisor/libexec/flowvisor/jetty-continuation-7.0.2.v20100331.jar:/opt/flowvisor/libexec/flowvisor/jetty-http-7.0.2.v20100331.jar:/opt/flowvisor/libexec/flowvisor/jetty-io-7.0.2.v20100331.jar:/opt/flowvisor/libexec/flowvisor/jetty-security-7.0.2.v20100331.jar:/opt/flowvisor/libexec/flowvisor/jetty-server-7.0.2.v20100331.jar:/opt/flowvisor/libexec/flowvisor/jetty-util-7.0.2.v20100331.jar:/opt/flowvisor/libexec/flowvisor/servlet-api-2.5.jar:/opt/flowvisor/libexec/flowvisor/flowvisor.jar org.flowvisor.FlowVisor /opt/flowvisor/etc/flowvisor/config.xml [16:14:02] jbs@bbn-hn:/home/jbs +$ rpm -qf /opt/flowvisor/libexec/flowvisor/flowvisor.jar flowvisor-0.8.2-1.x86_64 }}} This is from an RPM: {{{ named 13783 1 0 Jun22 ? 00:01:04 /usr/sbin/named -u named [16:15:45] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/sbin/named bind-9.7.3-8.P3.el6_2.2.x86_64 }}} This is from an RPM: {{{ jonmills 16835 16833 0 00:09 ? 00:00:00 sshd: jonmills@pts/2 [16:16:07] jbs@bbn-hn:/home/jbs +$ which sshd sshd is /usr/sbin/sshd [16:16:12] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/sbin/sshd openssh-server-5.3p1-70.el6_2.2.x86_64 }}} Java is from an RPM (see above), I don't think the other stuff here is: {{{ 2006 19259 1 0 Jul05 ? 00:24:49 /usr/java/latest/bin/java -Djava.util.logging.config.file=/opt/orca-aux/tomcat-7/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:MaxPermSize=128m -Xmx1024m -Djavax.net.ssl.trustStore=/opt/orca-aux/tomcat-7/ssl/trusted.jks -Djavax.net.ssl.trustStorePassword=orcaorca -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=14000 -Djava.endorsed.dirs=/opt/orca-aux/tomcat-7/endorsed -classpath /opt/orca-aux/tomcat-7/bin/bootstrap.jar:/opt/orca-aux/tomcat-7/bin/tomcat-juli.jar -Dcatalina.base=/opt/orca-aux/tomcat-7 -Dcatalina.home=/opt/orca-aux/tomcat-7 -Djava.io.tmpdir=/opt/orca-aux/tomcat-7/temp org.apache.catalina.startup.Bootstrap start [16:16:15] jbs@bbn-hn:/home/jbs +$ rpm -qf /opt/orca-aux/tomcat-7/bin/tomcat-juli.jar file /opt/orca-aux/tomcat-7/bin/tomcat-juli.jar is not owned by any package [16:17:07] jbs@bbn-hn:/home/jbs +$ rpm -qf /opt/orca-aux/tomcat-7/bin file /opt/orca-aux/tomcat-7/bin is not owned by any package [16:17:10] jbs@bbn-hn:/home/jbs +$ rpm -qf /opt/orca-aux/tomcat-7 file /opt/orca-aux/tomcat-7 is not owned by any package [16:17:11] jbs@bbn-hn:/home/jbs +$ rpm -qf /opt/orca-aux file /opt/orca-aux is not owned by any package [16:18:40] jbs@bbn-hn:/home/jbs +$ rpm -qa | grep -i orca || echo no match no match }}} Java is from an RPM (see above), I don't think the other stuff here is: {{{ 2006 19470 1 0 Jul20 ? 01:02:40 /usr/java/latest/bin/java -Djava.util.logging.config.file=/opt/orca-12080/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xmx1024m -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=12000 -Djava.endorsed.dirs=/opt/orca-12080/tomcat/endorsed -classpath /opt/orca-12080/tomcat/bin/bootstrap.jar:/opt/orca-12080/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/orca-12080/tomcat -Dcatalina.home=/opt/orca-12080/tomcat -Djava.io.tmpdir=/opt/orca-12080/tomcat/temp org.apache.catalina.startup.Bootstrap start [16:19:43] jbs@bbn-hn:/home/jbs +$ rpm -qf /opt/orca-12080/tomcat/bin/tomcat-juli.jar file /opt/orca-12080/tomcat/bin/tomcat-juli.jar is not owned by any package [16:19:47] jbs@bbn-hn:/home/jbs +$ rpm -qf /opt/orca-12080/tomcat/bin file /opt/orca-12080/tomcat/bin is not owned by any package [16:19:49] jbs@bbn-hn:/home/jbs +$ rpm -qf /opt/orca-12080 file /opt/orca-12080 is not owned by any package [16:19:51] jbs@bbn-hn:/home/jbs +$ rpm -qa | grep -i orca || echo no match no match }}} Java is from an RPM (see above), I don't think the other stuff here is: {{{ 2006 19625 1 0 Jul20 ? 00:29:16 /usr/java/latest/bin/java -Djava.util.logging.config.file=/opt/orca-13080/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/opt/orca-13080/tomcat/endorsed -classpath /opt/orca-13080/tomcat/bin/bootstrap.jar:/opt/orca-13080/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/orca-13080/tomcat -Dcatalina.home=/opt/orca-13080/tomcat -Djava.io.tmpdir=/opt/orca-13080/tomcat/temp org.apache.catalina.startup.Bootstrap start [16:21:25] jbs@bbn-hn:/home/jbs +$ rpm -qf /opt/orca-13080/tomcat/bin/tomcat-juli.jar file /opt/orca-13080/tomcat/bin/tomcat-juli.jar is not owned by any package [16:21:38] jbs@bbn-hn:/home/jbs +$ rpm -qf /opt/orca-13080/tomcat/bin file /opt/orca-13080/tomcat/bin is not owned by any package [16:21:40] jbs@bbn-hn:/home/jbs +$ rpm -qf /opt/orca-13080/tomcat file /opt/orca-13080/tomcat is not owned by any package [16:21:42] jbs@bbn-hn:/home/jbs +$ rpm -qf /opt/orca-13080 file /opt/orca-13080 is not owned by any package }}} This is not from an RPM: {{{ rack_bbn 22989 1 0 Jun18 ? 00:00:24 /omd/sites/rack_bbn/var/tmp/xinetd -pidfile /omd/sites/rack_bbn/tmp/run/xinetd.pid -filelog /omd/sites/rack_bbn/var/log/xinetd.log -f /omd/sites/rack_bbn/etc/xinetd.conf [16:21:43] jbs@bbn-hn:/home/jbs +$ rpm -qf /omd/sites/rack_bbn/var/tmp/xinetd file /omd/sites/rack_bbn/var/tmp/xinetd is not owned by any package }}} This is from an RPM: {{{ root 23122 1 0 Jun22 ? 00:00:00 /usr/sbin/vsftpd /etc/vsftpd/vsftpd.conf [16:23:13] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/sbin/vsftpd vsftpd-2.2.2-6.el6_2.1.x86_64 }}} This is from an RPM: {{{ root 23124 1 0 Jun22 ? 00:00:06 xcatd: SSL listener [16:23:32] jbs@bbn-hn:/home/jbs +$ which xcatd xcatd is /opt/xcat/sbin/xcatd [16:23:35] jbs@bbn-hn:/home/jbs +$ rpm -qf /opt/xcat/sbin/xcatd xCAT-server-2.6.9-snap201111090305.noarch }}} This is from an RPM: {{{ root 23127 23124 0 Jun22 ? 00:00:01 xcatd: install monitor [16:23:32] jbs@bbn-hn:/home/jbs +$ which xcatd xcatd is /opt/xcat/sbin/xcatd [16:23:35] jbs@bbn-hn:/home/jbs +$ rpm -qf /opt/xcat/sbin/xcatd xCAT-server-2.6.9-snap201111090305.noarch }}} This is from an RPM: {{{ root 27286 1 0 Jun18 ? 00:00:00 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf [16:23:40] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/sbin/nginx nginx-0.8.54-1.el6.x86_64 }}} This is from an RPM: {{{ dhcpd 27903 1 0 Jun22 ? 00:02:12 /usr/sbin/dhcpd -user dhcpd -group dhcpd bond2.1007 bond2.1006 [16:24:50] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/sbin/dhcpd dhcp-4.1.1-25.P1.el6_2.1.x86_64 }}} Lumping these nova things together, Python is from an RPM, as are nova-api and nova-objectstore: {{{ nova 28606 28603 0 Jul06 ? 00:06:51 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8781.log --ec2_listen_port=8781 --osapi_listen_port=8788 nova 28639 1 0 Jul06 ? 00:00:07 /usr/bin/python /usr/bin/nova-objectstore --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-objectstore.log --pidfile=/var/run/nova/nova-objectstore.pid nova 28701 28699 0 Jul06 ? 00:06:46 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8782.log --ec2_listen_port=8782 --osapi_listen_port=8787 nova 28814 28812 0 Jul06 ? 00:06:32 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8783.log --ec2_listen_port=8783 --osapi_listen_port=8786 nova 28878 28876 0 Jul06 ? 00:06:09 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8784.log --ec2_listen_port=8784 --osapi_listen_port=8785 nova 28903 28901 0 Jul06 ? 00:06:12 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8791.log --ec2_listen_port=8791 --osapi_listen_port=8798 nova 28923 28921 0 Jul06 ? 00:06:49 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8792.log --ec2_listen_port=8792 --osapi_listen_port=8797 nova 28943 28941 0 Jul06 ? 00:06:28 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8793.log --ec2_listen_port=8793 --osapi_listen_port=8796 nova 28965 28963 0 Jul06 ? 00:07:28 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8794.log --ec2_listen_port=8794 --osapi_listen_port=8795 nova 28987 28985 0 Jul06 ? 00:06:01 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8801.log --ec2_listen_port=8801 --osapi_listen_port=8808 nova 29016 29014 0 Jul06 ? 00:06:33 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8802.log --ec2_listen_port=8802 --osapi_listen_port=8807 nova 29042 29040 0 Jul06 ? 00:09:05 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8803.log --ec2_listen_port=8803 --osapi_listen_port=8806 nova 29084 29080 0 Jul06 ? 00:06:34 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8804.log --ec2_listen_port=8804 --osapi_listen_port=8805 nova 29140 29138 0 Jul06 ? 00:06:11 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8811.log --ec2_listen_port=8811 --osapi_listen_port=8818 nova 29162 29160 0 Jul06 ? 00:06:29 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8812.log --ec2_listen_port=8812 --osapi_listen_port=8817 nova 29185 29183 0 Jul06 ? 00:06:42 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8813.log --ec2_listen_port=8813 --osapi_listen_port=8816 nova 29205 29203 0 Jul06 ? 00:06:47 /usr/bin/python /usr/bin/nova-api --flagfile=/etc/nova/nova.conf --logfile=/var/log/nova/nova-api-8814.log --ec2_listen_port=8814 --osapi_listen_port=8815 [16:27:28] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/bin/python python-2.6.6-29.el6.x86_64 [16:27:31] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/bin/nova-api openstack-nova-api-2011.3-b1281neuca2.noarch [16:27:34] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/bin/nova-objectstore openstack-nova-objectstore-2011.3-b1281neuca2.noarch }}} Lumping these glance things together, all of these are from RPMs: {{{ glance 28780 1 0 Jul06 ? 00:00:54 /usr/bin/python /usr/bin/glance-api --config-file=/etc/glance/glance-api.conf --log-config=/etc/glance/logging-api.conf glance 28826 1 0 Jul06 ? 00:00:50 /usr/bin/python /usr/bin/glance-registry --config-file=/etc/glance/glance-registry.conf -v --log-config=/etc/glance/logging-registry.conf [16:28:17] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/bin/python python-2.6.6-29.el6.x86_64 [16:28:18] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/bin/glance-api openstack-glance-2011.3-b1277.noarch [16:28:20] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/bin/glance-registry openstack-glance-2011.3-b1277.noarch }}} Java is from an RPM (see above), I don't think the other stuff here is: {{{ 2006 29224 1 0 Jul20 ? 00:29:38 /usr/java/latest/bin/java -Djava.util.logging.config.file=/opt/orca-11080/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xmx1024m -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=11000 -Djava.endorsed.dirs=/opt/orca-11080/tomcat/endorsed -classpath /opt/orca-11080/tomcat/bin/bootstrap.jar:/opt/orca-11080/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/orca-11080/tomcat -Dcatalina.home=/opt/orca-11080/tomcat -Djava.io.tmpdir=/opt/orca-11080/tomcat/temp org.apache.catalina.startup.Bootstrap start [16:29:14] jbs@bbn-hn:/home/jbs +$ rpm -qf /opt/orca-11080/tomcat/bin/tomcat-juli.jar file /opt/orca-11080/tomcat/bin/tomcat-juli.jar is not owned by any package [16:29:15] jbs@bbn-hn:/home/jbs +$ rpm -qf /opt/orca-11080/tomcat/bin file /opt/orca-11080/tomcat/bin is not owned by any package [16:29:17] jbs@bbn-hn:/home/jbs +$ rpm -qf /opt/orca-11080/tomcat file /opt/orca-11080/tomcat is not owned by any package [16:29:18] jbs@bbn-hn:/home/jbs +$ rpm -qf /opt/orca-11080 file /opt/orca-11080 is not owned by any package [16:29:19] jbs@bbn-hn:/home/jbs +$ rpm -qa | grep -i orca || echo no match no match }}} This is from an RPM: {{{ jonmills 32641 32639 0 10:09 ? 00:00:00 sshd: jonmills@pts/1 [16:29:51] jbs@bbn-hn:/home/jbs +$ which sshd sshd is /usr/sbin/sshd [16:29:52] jbs@bbn-hn:/home/jbs +$ rpm -qf /usr/sbin/sshd openssh-server-5.3p1-70.el6_2.2.x86_64 }}} So, that's these RPMs: {{{ puppet-2.7.18-1.el6.noarch jdk-1.6.0_30-fcs.x86_64 rsyslog-4.6.2-12.el6.x86_64 rpcbind-0.2.0-8.el6.x86_64 nfs-utils-1.2.3-15.el6.x86_64 openldap-servers-2.4.23-20.el6.x86_64 quota-3.17-16.el6.x86_64 nfs-utils-1.2.3-15.el6.x86_64 openssh-server-5.3p1-70.el6_2.2.x86_64 xinetd-2.3.14-33.el6.x86_64 apcupsd-3.14.8-4.el6.x86_64 mysql-server-5.1.61-1.el6_2.1.x86_64 postfix-2.6.6-2.2.el6_1.x86_64 erlang-erts-R14B-04.1.el6.x86_64 erlang-erts-R14B-04.1.el6.x86_64 rabbitmq-server-2.6.1-1.el6.noarch httpd-2.2.15-15.el6.centos.1.x86_64 flowvisor-0.8.2-1.x86_64 bind-9.7.3-8.P3.el6_2.2.x86_64 openssh-server-5.3p1-70.el6_2.2.x86_64 vsftpd-2.2.2-6.el6_2.1.x86_64 xCAT-server-2.6.9-snap201111090305.noarch xCAT-server-2.6.9-snap201111090305.noarch nginx-0.8.54-1.el6.x86_64 dhcp-4.1.1-25.P1.el6_2.1.x86_64 python-2.6.6-29.el6.x86_64 openstack-nova-api-2011.3-b1281neuca2.noarch openstack-nova-objectstore-2011.3-b1281neuca2.noarch python-2.6.6-29.el6.x86_64 openstack-glance-2011.3-b1277.noarch openstack-glance-2011.3-b1277.noarch openssh-server-5.3p1-70.el6_2.2.x86_64 }}} Sorted and uniq-ed: {{{ apcupsd-3.14.8-4.el6.x86_64 bind-9.7.3-8.P3.el6_2.2.x86_64 dhcp-4.1.1-25.P1.el6_2.1.x86_64 erlang-erts-R14B-04.1.el6.x86_64 flowvisor-0.8.2-1.x86_64 httpd-2.2.15-15.el6.centos.1.x86_64 jdk-1.6.0_30-fcs.x86_64 mysql-server-5.1.61-1.el6_2.1.x86_64 nfs-utils-1.2.3-15.el6.x86_64 nginx-0.8.54-1.el6.x86_64 openldap-servers-2.4.23-20.el6.x86_64 openssh-server-5.3p1-70.el6_2.2.x86_64 openstack-glance-2011.3-b1277.noarch openstack-nova-api-2011.3-b1281neuca2.noarch openstack-nova-objectstore-2011.3-b1281neuca2.noarch postfix-2.6.6-2.2.el6_1.x86_64 puppet-2.7.18-1.el6.noarch python-2.6.6-29.el6.x86_64 quota-3.17-16.el6.x86_64 rabbitmq-server-2.6.1-1.el6.noarch rpcbind-0.2.0-8.el6.x86_64 rsyslog-4.6.2-12.el6.x86_64 vsftpd-2.2.2-6.el6_2.1.x86_64 xCAT-server-2.6.9-snap201111090305.noarch xinetd-2.3.14-33.el6.x86_64 }}} And these things which aren't from RPMs: {{{ /opt/floodlight/floodlight.jar is not owned by any package /opt/imageproxy is not owned by any package /opt/orca-aux/tomcat-7/bin/tomcat-juli.jar is not owned by any package /opt/orca-aux/tomcat-7/bin is not owned by any package /opt/orca-aux/tomcat-7 is not owned by any package /opt/orca-aux is not owned by any package /opt/orca-12080/tomcat/bin/tomcat-juli.jar is not owned by any package /opt/orca-12080/tomcat/bin is not owned by any package /opt/orca-12080 is not owned by any package /opt/orca-13080/tomcat/bin/tomcat-juli.jar is not owned by any package /opt/orca-13080/tomcat/bin is not owned by any package /opt/orca-13080/tomcat is not owned by any package /opt/orca-13080 is not owned by any package /omd/sites/rack_bbn/var/tmp/xinetd is not owned by any package /opt/orca-11080/tomcat/bin/tomcat-juli.jar is not owned by any package /opt/orca-11080/tomcat/bin is not owned by any package /opt/orca-11080/tomcat is not owned by any package /opt/orca-11080 is not owned by any package }}} The xinetd in /omd is a copy of the stock xinetd: {{{ [12:27:40] jbs@bbn-hn:/home/jbs +$ md5sum /omd/sites/rack_bbn/var/tmp/xinetd /usr/sbin/xinetd 1c0ef402cd3ad81e01cda637a6c02104 /omd/sites/rack_bbn/var/tmp/xinetd 1c0ef402cd3ad81e01cda637a6c02104 /usr/sbin/xinetd }}} So that essentialy is from an RPM. ExoGENI ticket [exoticket:93] is tracking the task of identifying the versions and sources of the others. As to the RPMs, these look like standard CentOS: {{{ [11:58:34] jbs@bbn-hn:/home/jbs +$ yum list installed apcupsd-3.14.8-4.el6.x86_64 bind-9.7.3-8.P3.el6_2.2.x86_64 dhcp-4.1.1-25.P1.el6_2.1.x86_64 erlang-erts-R14B-04.1.el6.x86_64 httpd-2.2.15-15.el6.centos.1.x86_64 mysql-server-5.1.61-1.el6_2.1.x86_64 nfs-utils-1.2.3-15.el6.x86_64 nginx-0.8.54-1.el6.x86_64 openldap-servers-2.4.23-20.el6.x86_64 openssh-server-5.3p1-70.el6_2.2.x86_64 postfix-2.6.6-2.2.el6_1.x86_64 python-2.6.6-29.el6.x86_64 quota-3.17-16.el6.x86_64 rabbitmq-server-2.6.1-1.el6.noarch rpcbind-0.2.0-8.el6.x86_64 rsyslog-4.6.2-12.el6.x86_64 vsftpd-2.2.2-6.el6_2.1.x86_64 xinetd-2.3.14-33.el6.x86_64 Loaded plugins: fastestmirror, refresh-packagekit, security Loading mirror speeds from cached hostfile * base: mirror.dattobackup.com * epel: mirror.metrocast.net * extras: mirror.dattobackup.com * updates: mirror.metrocast.net Installed Packages apcupsd.x86_64 3.14.8-4.el6 @epel dhcp.x86_64 12:4.1.1-25.P1.el6_2.1 @CentOS Updates/6.2 erlang-erts.x86_64 R14B-04.1.el6 @UIedited_1/6.2 httpd.x86_64 2.2.15-15.el6.centos.1 @CentOS Updates/6.2 mysql-server.x86_64 5.1.61-1.el6_2.1 @CentOS Updates/6.2 nfs-utils.x86_64 1:1.2.3-15.el6 @anaconda-CentOS-201112091719.x86_64/6.2 nginx.x86_64 0.8.54-1.el6 @UIedited_1/6.2 openldap-servers.x86_64 2.4.23-20.el6 @base openssh-server.x86_64 5.3p1-70.el6_2.2 @CentOS Updates/6.2 postfix.x86_64 2:2.6.6-2.2.el6_1 @anaconda-CentOS-201112091719.x86_64/6.2 python.x86_64 2.6.6-29.el6 @anaconda-CentOS-201112091719.x86_64/6.2 quota.x86_64 1:3.17-16.el6 @anaconda-CentOS-201112091719.x86_64/6.2 rabbitmq-server.noarch 2.6.1-1.el6 @UIedited_1/6.2 rpcbind.x86_64 0.2.0-8.el6 @anaconda-CentOS-201112091719.x86_64/6.2 rsyslog.x86_64 4.6.2-12.el6 @anaconda-CentOS-201112091719.x86_64/6.2 vsftpd.x86_64 2.2.2-6.el6_2.1 @CentOS Updates/6.2 xinetd.x86_64 2:2.3.14-33.el6 @anaconda-CentOS-201112091719.x86_64/6.2 }}} These look like not: {{{ [11:58:38] jbs@bbn-hn:/home/jbs +$ yum list installed flowvisor-0.8.2-1.x86_64 jdk-1.6.0_30-fcs.x86_64 openstack-glance-2011.3-b1277.noarch openstack-nova-api-2011.3-b1281neuca2.noarch openstack-nova-objectstore-2011.3-b1281neuca2.noarch puppet-2.7.18-1.el6.noarch xCAT-server-2.6.9-snap201111090305.noarch Loaded plugins: fastestmirror, refresh-packagekit, security Loading mirror speeds from cached hostfile * base: mirror.dattobackup.com * epel: mirror.metrocast.net * extras: mirror.dattobackup.com * updates: mirror.metrocast.net Installed Packages flowvisor.x86_64 0.8.2-1 @exogeni jdk.x86_64 2000:1.6.0_30-fcs @ExoGENI/6.2 openstack-glance.noarch 1:2011.3-b1277 @ExoGENI-noarch/6.2 openstack-nova-api.noarch 1:2011.3-b1281neuca2 @exogeni-noarch openstack-nova-objectstore.noarch 1:2011.3-b1281neuca2 @exogeni-noarch puppet.noarch 2.7.18-1.el6 @puppetlabs-products xCAT-server.noarch 4:2.6.9-snap201111090305 @xCAT/6.2 }}} I can find the name of a source RPM for each of those: {{{ [12:12:44] jbs@bbn-hn:/home/jbs +$ for package in flowvisor-0.8.2-1.x86_64 jdk-1.6.0_30-fcs.x86_64 openstack-glance-2011.3-b1277.noarch openstack-nova-api-2011.3-b1281neuca2.noarch openstack-nova-objectstore-2011.3-b1281neuca2.noarch puppet-2.7.18-1.el6.noarch xCAT-server-2.6.9-snap201111090305.noarch ; do rpm -q $package --qf 'Name : %{NAME}\nSource RPM : %{SOURCERPM}\n\n' ; done Name : flowvisor Source RPM : flowvisor-0.8.2-1.src.rpm Name : jdk Source RPM : jdk-1.6.0_30-fcs.src.rpm Name : openstack-glance Source RPM : openstack-glance-2011.3-b1277.src.rpm Name : openstack-nova-api Source RPM : openstack-nova-2011.3-b1281neuca2.src.rpm Name : openstack-nova-objectstore Source RPM : openstack-nova-2011.3-b1281neuca2.src.rpm Name : puppet Source RPM : puppet-2.7.18-1.el6.src.rpm Name : xCAT-server Source RPM : xCAT-server-2.6.9-snap201111090305.src.rpm }}} So, now to connect that with the repo URLs... That's a somewhat manual process, as you need to fish the repo URL out of the relevant file in /etc/yum.repos.d, but those seem to be: {{{ [14:19:44] jbs@bbn-hn:/home/jbs +$ grep baseurl /etc/yum.repos.d/exogeni.repo baseurl=http://software.exogeni.net/repo/exogeni/6/current/$basearch baseurl=http://software.exogeni.net/repo/exogeni/6/current/noarch [14:19:55] jbs@bbn-hn:/home/jbs +$ grep baseurl /etc/yum.repos.d/puppetlabs.repo baseurl=http://yum.puppetlabs.com/el/6/products/$basearch baseurl=http://yum.puppetlabs.com/el/6/dependencies/$basearch baseurl=http://yum.puppetlabs.com/el/6/products/SRPMS baseurl=http://yum.puppetlabs.com/el/6/dependencies/SRPMS }}} Hmm, I don't see a repo for xCAT anywhere: {{{ [14:20:23] jbs@bbn-hn:/home/jbs +$ grep -i xcat /etc/yum.repos.d/* || echo no match no match }}} ExoGENI ticket [exoticket:98] is tracking the task of making sure that repo exists. Meanwhile, http://software.exogeni.net/repo/exogeni doesn't seem to have an obvious SRPM location, so I've created ExoGENI ticket [exoticket:96] for that. Puppet does have SRPMs, as the baseurl listings above imply, and http://yum.puppetlabs.com/el/6/products/SRPMS/puppet-2.7.18-1.el6.src.rpm has one for Puppet 2.7.18, so that seems good. Aside from the open tickets (and the one question to exogeni-design), I think this step is now complete. == Step 2: identify network-listening software on an !OpenStack worker node == === Overview of Step 2 === '''Using:''' * Using netstat, enumerate processes on bbn-w1 which listen for network connections from outside the node * For each process found: * Use the command-line or `/proc` to determine what executable file is running * Use RPM tools to determine whether the executable file is part of an RPM * Otherwise, use documentation or iterate with the ExoGENI team to determine the origin of the software * For each RPM found, identify a location from which a source RPM for that package can be obtained * For each non-RPM software source found, identify a location from which the source code for that version can be obtained. '''Verify:''' * The source of each network-listening file can be identified * RPM source packages can be found for each RPM-sourced package * The source code and identifiable version (e.g. a git tag) can be found for each non-RPM software source === Results of Step 2 from 2012-08-15 === A list of processes that are listening: {{{ [11:20:57] jbs@bbn-w1:/home/jbs +$ sudo netstat -anp | egrep ':\*.+LISTEN' | egrep -v '(127.0.0.1|192.168.103.10|10.100.0.1|10.103.0.1)' > netstat-raw.txt [11:21:22] jbs@bbn-w1:/home/jbs +$ cat netstat-raw.txt tcp 0 0 0.0.0.0:8139 0.0.0.0:* LISTEN 2572/ruby tcp 0 0 0.0.0.0:5900 0.0.0.0:* LISTEN 18815/qemu-kvm tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1983/rpcbind tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 2272/sshd tcp 0 0 0.0.0.0:33762 0.0.0.0:* LISTEN 2001/rpc.statd tcp 0 0 :::111 :::* LISTEN 1983/rpcbind tcp 0 0 :::22 :::* LISTEN 2272/sshd tcp 0 0 ::1:25 :::* LISTEN 2374/master tcp 0 0 :::6556 :::* LISTEN 2280/xinetd tcp 0 0 :::48004 :::* LISTEN 2001/rpc.statd }}} Stripping that into a list of PIDs: {{{ [11:21:36] jbs@bbn-w1:/home/jbs +$ awk '{print $NF}' netstat-raw.txt | sed -r -e 's|/.*||' | sort -n | uniq > netstat-pids.txt [11:22:03] jbs@bbn-w1:/home/jbs +$ cat netstat-pids.txt 1983 2001 2272 2280 2374 2572 18815 }}} Here are the commands: {{{ [11:22:38] jbs@bbn-w1:/home/jbs +$ for pid in $(cat netstat-pids.txt) ; do ps -efwww | egrep "^[^ ]+ +$pid" ; done >& netstat-commands.txt [11:23:47] jbs@bbn-w1:/home/jbs +$ cat netstat-commands.txt rpc 1983 1 0 Jul17 ? 00:00:01 rpcbind rpcuser 2001 1 0 Jul17 ? 00:00:00 rpc.statd root 2272 1 0 Jul17 ? 00:00:00 /usr/sbin/sshd root 2280 1 0 Jul17 ? 00:00:05 xinetd -stayalive -pidfile /var/run/xinetd.pid root 2374 1 0 Jul17 ? 00:00:10 /usr/libexec/postfix/master root 2572 1 0 Jul17 ? 01:55:23 /usr/bin/ruby /usr/sbin/puppetd qemu 18815 1 0 Aug14 ? 00:01:20 /usr/libexec/qemu-kvm -S -M pc-0.14 -enable-kvm -m 128 -smp 1,sockets=1,cores=1,threads=1 -name instance-00000d78 -uuid efb7e543-0ed8-9dac-76dd-12345b4be3a9 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000d78.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -kernel /var/lib/nova/instances/instance-00000d78/kernel -initrd /var/lib/nova/instances/instance-00000d78/ramdisk -append root=/dev/vda console=ttyS0 selinux=0 -drive file=/var/lib/nova/instances/instance-00000d78/disk,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/var/lib/nova/instances/instance-00000d78/disk.local,if=none,id=drive-virtio-disk1,format=qcow2 -device virtio-blk-pci,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=22,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=02:16:3e:69:b0:db,bus=pci.0,addr=0x3 -netdev tap,fd=23,id=hostnet1 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:e6:5e:1a,bus=pci.0,addr=0x4 -chardev file,id=charserial0,path=/var/lib/nova/instances/instance-00000d78/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -usb -vnc 0.0.0.0:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 }}} Just to sanity check that we haven't missed anything: {{{ [11:23:51] jbs@bbn-w1:/home/jbs +$ wc -l netstat-* 7 netstat-commands.txt 7 netstat-pids.txt 10 netstat-raw.txt 24 total }}} The change from raw to pids is because of duplicates: Two for 1983, two for 2001, and two for 2272. Let's plow through each of those. This is from an RPM: {{{ rpc 1983 1 0 Jul17 ? 00:00:01 rpcbind [11:27:39] jbs@bbn-w1:/home/jbs +$ which rpcbind rpcbind is /sbin/rpcbind [11:27:41] jbs@bbn-w1:/home/jbs +$ rpm -qf /sbin/rpcbind rpcbind-0.2.0-8.el6.x86_64 }}} This is from an RPM: {{{ rpcuser 2001 1 0 Jul17 ? 00:00:00 rpc.statd [11:27:45] jbs@bbn-w1:/home/jbs +$ which rpc.statd rpc.statd is /sbin/rpc.statd [11:28:02] jbs@bbn-w1:/home/jbs +$ rpm -qf /sbin/rpc.statd nfs-utils-1.2.3-15.el6.x86_64 }}} This is from an RPM: {{{ root 2272 1 0 Jul17 ? 00:00:00 /usr/sbin/sshd [11:28:35] jbs@bbn-w1:/home/jbs +$ rpm -qf /usr/sbin/sshd openssh-server-5.3p1-70.el6_2.2.x86_64 }}} This is from an RPM: {{{ root 2280 1 0 Jul17 ? 00:00:05 xinetd -stayalive -pidfile /var/run/xinetd.pid [11:28:50] jbs@bbn-w1:/home/jbs +$ which xinetd xinetd is /usr/sbin/xinetd [11:28:52] jbs@bbn-w1:/home/jbs +$ rpm -qf /usr/sbin/xinetd xinetd-2.3.14-33.el6.x86_64 }}} This is from an RPM: {{{ root 2374 1 0 Jul17 ? 00:00:10 /usr/libexec/postfix/master [11:28:56] jbs@bbn-w1:/home/jbs +$ rpm -qf /usr/libexec/postfix/master postfix-2.6.6-2.2.el6_1.x86_64 }}} These are from RPMs: {{{ root 2572 1 0 Jul17 ? 01:55:23 /usr/bin/ruby /usr/sbin/puppetd [11:29:12] jbs@bbn-w1:/home/jbs +$ rpm -qf /usr/bin/ruby ruby-1.8.7.352-4.el6_2.x86_64 [11:29:29] jbs@bbn-w1:/home/jbs +$ rpm -qf /usr/sbin/puppetd puppet-2.6.14-1.el6.noarch }}} This is not from an RPM; but there seems to be a qemu-kvm RPM; but it doesn't contain any files: {{{ qemu 18815 1 0 Aug14 ? 00:01:20 /usr/libexec/qemu-kvm -S -M pc-0.14 -enable-kvm -m 128 -smp 1,sockets=1,cores=1,threads=1 -name instance-00000d78 -uuid efb7e543-0ed8-9dac-76dd-12345b4be3a9 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-00000d78.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -kernel /var/lib/nova/instances/instance-00000d78/kernel -initrd /var/lib/nova/instances/instance-00000d78/ramdisk -append root=/dev/vda console=ttyS0 selinux=0 -drive file=/var/lib/nova/instances/instance-00000d78/disk,if=none,id=drive-virtio-disk0,format=qcow2 -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/var/lib/nova/instances/instance-00000d78/disk.local,if=none,id=drive-virtio-disk1,format=qcow2 -device virtio-blk-pci,bus=pci.0,addr=0x6,drive=drive-virtio-disk1,id=virtio-disk1 -netdev tap,fd=22,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=02:16:3e:69:b0:db,bus=pci.0,addr=0x3 -netdev tap,fd=23,id=hostnet1 -device virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:e6:5e:1a,bus=pci.0,addr=0x4 -chardev file,id=charserial0,path=/var/lib/nova/instances/instance-00000d78/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -usb -vnc 0.0.0.0:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 [11:29:34] jbs@bbn-w1:/home/jbs +$ rpm -qf /usr/libexec/qemu-kvm file /usr/libexec/qemu-kvm is not owned by any package [11:30:58] jbs@bbn-w1:/home/jbs +$ rpm -q qemu-kvm qemu-kvm-0.15.0-3.el6.x86_64 [11:31:15] jbs@bbn-w1:/home/jbs +$ rpm -ql qemu-kvm (contains no files) }}} Ah: {{{ [11:31:17] jbs@bbn-w1:/home/jbs +$ rpm -qi qemu-kvm Name : qemu-kvm Relocations: (not relocatable) Version : 0.15.0 Vendor: (none) Release : 3.el6 Build Date: Thu Nov 24 12:24:20 2011 Install Date: Fri Feb 24 08:56:13 2012 Build Host: osc-build.vm.griddynamics.net Group : Development/Tools Source RPM: qemu-0.15.0-3.el6.src.rpm Size : 0 License: GPLv2+ and LGPLv2+ and BSD Signature : RSA/SHA1, Thu Nov 24 12:25:34 2011, Key ID 499590cc596f1e88 URL : http://www.qemu.org/ Summary : QEMU metapackage for KVM support Description : This is a meta-package that provides a qemu-system- package for native architectures where kvm can be enabled. For example, in an x86 system, this will install qemu-system-x86 }}} And ah, /usr/libexec/qemu-kvm is just a symlink, to an executable which *is* from an RPM: {{{ [11:31:47] jbs@bbn-w1:/home/jbs +$ file /usr/libexec/qemu-kvm /usr/libexec/qemu-kvm: symbolic link to `/usr/bin/qemu-kvm' [11:32:17] jbs@bbn-w1:/home/jbs +$ rpm -qf /usr/bin/qemu-kvm qemu-system-x86-0.15.0-3.el6.x86_64 }}} Ok then. So, that's these RPMs: {{{ rpcbind-0.2.0-8.el6.x86_64 nfs-utils-1.2.3-15.el6.x86_64 openssh-server-5.3p1-70.el6_2.2.x86_64 xinetd-2.3.14-33.el6.x86_64 postfix-2.6.6-2.2.el6_1.x86_64 ruby-1.8.7.352-4.el6_2.x86_64 puppet-2.6.14-1.el6.noarch qemu-system-x86-0.15.0-3.el6.x86_64 }}} Those all look like standard CentOS things: {{{ [11:39:19] jbs@bbn-w1:/home/jbs +$ yum -C list installed rpcbind-0.2.0-8.el6.x86_64 nfs-utils-1.2.3-15.el6.x86_64 openssh-server-5.3p1-70.el6_2.2.x86_64 xinetd-2.3.14-33.el6.x86_64 postfix-2.6.6-2.2.el6_1.x86_64 ruby-1.8.7.352-4.el6_2.x86_64 puppet-2.6.14-1.el6.noarch qemu-system-x86-0.15.0-3.el6.x86_64 Loaded plugins: fastestmirror, priorities, refresh-packagekit, security 190 packages excluded due to repository priority protections Installed Packages nfs-utils.x86_64 1:1.2.3-15.el6 @anaconda-CentOS-201112091719.x86_64/6.2 openssh-server.x86_64 5.3p1-70.el6_2.2 @CentOS Updates/6.2 postfix.x86_64 2:2.6.6-2.2.el6_1 @anaconda-CentOS-201112091719.x86_64/6.2 puppet.noarch 2.6.14-1.el6 @epel qemu-system-x86.x86_64 2:0.15.0-3.el6 @ExoGENI/6.2 rpcbind.x86_64 0.2.0-8.el6 @anaconda-CentOS-201112091719.x86_64/6.2 ruby.x86_64 1.8.7.352-4.el6_2 @updates xinetd.x86_64 2:2.3.14-33.el6 @anaconda-CentOS-201112091719.x86_64/6.2 }}} ('yum -C' to use cached info only, because these bbn-w1 can't do DNS right now.) So, that identifies the source of each network-listening file, and since they're all standard CentOS RPMs, source RPMs for them can be found in standard CentOS places. I've created ExoGENI ticket [exoticket:104] for the DNS issue, but it's not a blocker for this test. == Step 3: verify VLANs on the rack management switch == === Overview of Step 3 === '''Using:''' * Establish a privileged login to the 8052 (management) switch * Obtain the list of all VLAN mappings for all interfaces * Determine which interfaces connect to worker nodes * Create a listing of worker node control interfaces to VLANs by interface type '''Verify:''' * All worker node IMM interfaces are access ports on VLAN 1006 * All worker node iSCSI interfaces are access ports on VLAN 1009 * Control interfaces for all workers in !OpenStack mode are access ports on VLAN 1007 * Control interfaces for all workers in bare metal mode are access ports on VLAN 1008 === Results of Step 3 from 2012-10-10 === Established a privileged login to the 8052 management switch: {{{ [lnevers@bbn-hn]$ ssh bbn-8052 Enter radius password: IBM Networking Operating System RackSwitch G8052. bbn-8052.bbn.xo> }}} Determined all VLAN mappings for all interfaces: {{{ bbn-8052.bbn.xo>show vlan VLAN Name Status Ports ---- -------------------------------- ------ ------------------------- 1 Default VLAN ena XGE3 20 VLAN 20 ena empty 829 VLAN 829 ena empty 1006 ExoGENI Management Private ena 1-20 XGE4 1007 OpenStack Worker Nodes ena 17-20 37-44 1008 xCAT (Bare Metal) ena 17-20 45 46 1009 iSCSI Worker ena 25-36 1010 Public Internet Connectivity ena 47-XGE2 1011 OpenStack VMs ena 17-20 37-44 1012 iSCSI Head Node ena 21-24 bbn-8052.bbn.xo> }}} Determined which interfaces connect to worker nodes, by displaying the ports associated with the OpenStack Worker nodes,and xCAT VLANs: {{{ bbn-8052.bbn.xo>show vlan 1006 VLAN Name Status Ports ---- -------------------------------- ------ ------------------------- 1006 ExoGENI Management Private ena 1-20 XGE4 Private-VLAN Type Mapped-To Status Ports ------------ --------- ------------------ ---------- -------------------- 1006 - - dis 1-20 XGE4 }}} Note: Worker node IMM interfaces are accessed ports on VLAN 1006 {{{ bbn-8052.bbn.xo>show vlan 1007 VLAN Name Status Ports ---- -------------------------------- ------ ------------------------- 1007 OpenStack Worker Nodes ena 17-20 37-44 Private-VLAN Type Mapped-To Status Ports ------------ --------- ------------------ ---------- -------------------- 1007 - - dis 17-20 37-44 }}} Note: Control interfaces for all workers in !OpenStack mode are access ports on VLAN 1007 {{{ bbn-8052.bbn.xo>show vlan 1008 VLAN Name Status Ports ---- -------------------------------- ------ ------------------------- 1008 xCAT (Bare Metal) ena 17-20 45 46 Private-VLAN Type Mapped-To Status Ports ------------ --------- ------------------ ---------- -------------------- 1008 - - dis 17-20 45 46 }}} Note: Worker node iSCSI interfaces are access ports on VLAN 1009 {{{ VLAN Name Status Ports ---- -------------------------------- ------ ------------------------- 1009 iSCSI Worker ena 25-36 Private-VLAN Type Mapped-To Status Ports ------------ --------- ------------------ ---------- -------------------- 1009 - - dis 25-36 }}} Note: Control interfaces for all workers in bare metal mode are access ports on VLAN 1008bbn-8052.bbn.xo>show vlan 1009 Need to Create a listing of worker node control interfaces to VLANs by interface type == Step 4: verify MAC addresses on the rack management switch == === Overview of Step 4 === '''Using:''' * Establish a privileged login to the 8052 (management) switch * Obtain a list of the full MAC address table of the switch * Use interface listings on hosts and devices to determine the identities of all MAC addresses '''Verify:''' * It is possible to identify and classify every MAC address visible on the switch === Results of Step 4 from 2012-10-10 === Established a privileged login to the 8052 management switch: {{{ [lnevers@bbn-hn]$ ssh bbn-8052 Enter radius password: IBM Networking Operating System RackSwitch G8052. bbn-8052.bbn.xo> }}} Determined full list of MAC address table for switch: {{{ bbn-8052.bbn.xo>show mac-address-table Mac address Aging Time: 300 Total number of FDB entries : 61 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 00:10:18:a8:73:d2 1012 53 TRK 00:10:18:a8:73:d4 1007 54 TRK 00:10:18:a8:73:d6 1007 54 TRK 00:80:e5:1d:f7:13 1012 23 FWD 00:80:e5:25:2e:a3 1012 24 FWD 00:80:e5:2d:01:64 1006 3 FWD 00:80:e5:2d:03:ac 1006 4 FWD 02:16:3e:02:f8:be 1011 44 FWD 02:16:3e:06:03:c8 1011 44 FWD 02:16:3e:07:2f:58 1011 41 FWD 02:16:3e:18:f3:fd 1011 37 FWD 02:16:3e:1a:e8:1e 1011 38 FWD 02:16:3e:1c:2b:7f 1011 38 FWD 02:16:3e:20:b9:e7 1011 40 FWD 02:16:3e:2c:38:33 1011 54 TRK 02:16:3e:2e:d9:ec 1011 43 FWD 02:16:3e:35:2d:c6 1011 40 FWD 02:16:3e:38:81:d4 1011 39 FWD 02:16:3e:3c:29:90 1011 42 FWD 02:16:3e:45:49:69 1011 41 FWD 02:16:3e:4d:67:aa 1011 43 FWD 02:16:3e:57:cc:be 1011 40 FWD 02:16:3e:63:a3:2e 1011 37 FWD 02:16:3e:64:87:e4 1011 42 FWD 02:16:3e:64:a3:40 1011 39 FWD 02:16:3e:66:40:a9 1011 41 FWD 02:16:3e:74:37:c6 1011 39 FWD 02:16:3e:7e:11:13 1011 37 FWD 02:16:3e:7f:36:22 1011 43 FWD 08:17:f4:b5:2a:00 1006 XGE4 FWD 08:17:f4:b5:2a:3f 1006 XGE4 FWD 5c:f3:fc:6b:10:a8 1006 54 TRK 5c:f3:fc:6b:10:a8 1007 54 TRK 5c:f3:fc:6b:10:a8 1008 54 TRK 5c:f3:fc:6b:10:aa 1007 54 TRK 5c:f3:fc:ba:13:34 1010 48 FWD 5c:f3:fc:ba:13:36 1012 53 TRK 5c:f3:fc:ba:4e:78 1007 38 FWD 5c:f3:fc:ba:51:d4 1007 40 FWD 5c:f3:fc:ba:56:68 1007 37 FWD 5c:f3:fc:ba:56:68 1011 37 FWD 5c:f3:fc:ba:65:c0 1007 39 FWD 5c:f3:fc:c0:14:c5 1006 1 FWD 5c:f3:fc:c0:50:09 1006 6 FWD 5c:f3:fc:c0:53:65 1006 2 FWD 5c:f3:fc:c0:57:f9 1006 5 FWD 5c:f3:fc:c0:67:51 1006 7 FWD 6c:ae:8b:1b:e7:8a 1008 46 FWD 6c:ae:8b:1b:e7:8e 1006 15 FWD 6c:ae:8b:1b:e7:f2 1007 42 FWD 6c:ae:8b:1b:e7:f6 1006 11 FWD 6c:ae:8b:1b:e9:d2 1008 45 FWD 6c:ae:8b:1b:e9:d6 1006 14 FWD 6c:ae:8b:1b:ee:fa 1007 41 FWD 6c:ae:8b:1b:ee:fe 1006 10 FWD 6c:ae:8b:1b:ef:52 1007 43 FWD 6c:ae:8b:1b:ef:56 1006 12 FWD 6c:ae:8b:1b:f0:aa 1007 44 FWD 6c:ae:8b:1b:f0:ae 1006 13 FWD cc:ef:48:7a:7a:a9 1010 48 FWD f8:c0:01:bb:f8:cb 1006 9 FWD }}} Used interface listings on hosts and devices to determine the identities of all MAC addresses on the switch. {{{ bbn-8052.bbn.xo>show mac-address-table address 00:10:18:a8:73:d2 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 00:10:18:a8:73:d2 1012 53 TRK bbn-8052.bbn.xo>show mac-address-table address 00:10:18:a8:73:d4 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 00:10:18:a8:73:d4 1007 54 TRK bbn-8052.bbn.xo>show mac-address-table address 00:10:18:a8:73:d6 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 00:10:18:a8:73:d6 1007 54 TRK bbn-8052.bbn.xo>show mac-address-table address 00:80:e5:1d:f7:13 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 00:80:e5:1d:f7:13 1012 23 FWD bbn-8052.bbn.xo>show mac-address-table address 00:80:e5:25:2e:a3 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 00:80:e5:25:2e:a3 1012 24 FWD bbn-8052.bbn.xo>show mac-address-table address 00:80:e5:2d:01:64 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 00:80:e5:2d:01:64 1006 3 FWD bbn-8052.bbn.xo>show mac-address-table address 00:80:e5:2d:03:ac MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 00:80:e5:2d:03:ac 1006 4 FWD bbn-8052.bbn.xo>show mac-address-table address 02:16:3e:02:f8:be MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 02:16:3e:02:f8:be 1011 44 FWD bbn-8052.bbn.xo>show mac-address-table address 02:16:3e:06:03:c8 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 02:16:3e:06:03:c8 1011 44 FWD bbn-8052.bbn.xo>show mac-address-table address 02:16:3e:07:2f:58 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 02:16:3e:07:2f:58 1011 41 FWD bbn-8052.bbn.xo>show mac-address-table address 02:16:3e:18:f3:fd MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 02:16:3e:18:f3:fd 1011 37 FWD bbn-8052.bbn.xo>show mac-address-table address 02:16:3e:1a:e8:1e MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 02:16:3e:1a:e8:1e 1011 38 FWD bbn-8052.bbn.xo>show mac-address-table address 02:16:3e:1c:2b:7f MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 02:16:3e:1c:2b:7f 1011 38 FWD bbn-8052.bbn.xo>show mac-address-table address 02:16:3e:20:b9:e7 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 02:16:3e:20:b9:e7 1011 40 FWD bbn-8052.bbn.xo>show mac-address-table address 02:16:3e:2c:38:33 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 02:16:3e:2c:38:33 1011 54 TRK bbn-8052.bbn.xo>show mac-address-table address 02:16:3e:2e:d9:ec MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 02:16:3e:2e:d9:ec 1011 43 FWD bbn-8052.bbn.xo>show mac-address-table address 02:16:3e:35:2d:c6 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 02:16:3e:35:2d:c6 1011 40 FWD bbn-8052.bbn.xo>show mac-address-table address 02:16:3e:38:81:d4 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 02:16:3e:38:81:d4 1011 39 FWD bbn-8052.bbn.xo>show mac-address-table address 02:16:3e:3c:29:90 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 02:16:3e:3c:29:90 1011 42 FWD bbn-8052.bbn.xo>show mac-address-table address 02:16:3e:45:49:69 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 02:16:3e:45:49:69 1011 41 FWD bbn-8052.bbn.xo>show mac-address-table address 02:16:3e:4d:67:aa MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 02:16:3e:4d:67:aa 1011 43 FWD bbn-8052.bbn.xo>show mac-address-table address 02:16:3e:57:cc:be MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 02:16:3e:57:cc:be 1011 40 FWD bbn-8052.bbn.xo>show mac-address-table address 02:16:3e:63:a3:2e MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 02:16:3e:63:a3:2e 1011 37 FWD bbn-8052.bbn.xo>show mac-address-table address 02:16:3e:64:87:e4 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 02:16:3e:64:87:e4 1011 42 FWD bbn-8052.bbn.xo>show mac-address-table address 02:16:3e:64:a3:40 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 02:16:3e:64:a3:40 1011 39 FWD bbn-8052.bbn.xo>show mac-address-table address 02:16:3e:66:40:a9 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 02:16:3e:66:40:a9 1011 41 FWD bbn-8052.bbn.xo>show mac-address-table address 02:16:3e:74:37:c6 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 02:16:3e:74:37:c6 1011 39 FWD bbn-8052.bbn.xo>show mac-address-table address 02:16:3e:7e:11:13 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 02:16:3e:7e:11:13 1011 37 FWD bbn-8052.bbn.xo>show mac-address-table address 02:16:3e:7f:36:22 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 02:16:3e:7f:36:22 1011 43 FWD bbn-8052.bbn.xo>show mac-address-table address 08:17:f4:b5:2a:00 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 08:17:f4:b5:2a:00 1006 XGE4 FWD bbn-8052.bbn.xo>show mac-address-table address 08:17:f4:b5:2a:3f MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 08:17:f4:b5:2a:3f 1006 XGE4 FWD bbn-8052.bbn.xo>show mac-address-table address 5c:f3:fc:6b:10:a8 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 5c:f3:fc:6b:10:a8 1007 54 TRK 5c:f3:fc:6b:10:a8 1006 54 TRK 5c:f3:fc:6b:10:a8 1008 54 TRK bbn-8052.bbn.xo>show mac-address-table address 5c:f3:fc:6b:10:a8 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 5c:f3:fc:6b:10:a8 1007 54 TRK 5c:f3:fc:6b:10:a8 1006 54 TRK 5c:f3:fc:6b:10:a8 1008 54 TRK bbn-8052.bbn.xo>show mac-address-table address 5c:f3:fc:6b:10:a8 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 5c:f3:fc:6b:10:a8 1007 54 TRK 5c:f3:fc:6b:10:a8 1006 54 TRK 5c:f3:fc:6b:10:a8 1008 54 TRK bbn-8052.bbn.xo>show mac-address-table address 5c:f3:fc:6b:10:aa MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 5c:f3:fc:6b:10:aa 1007 54 TRK bbn-8052.bbn.xo>show mac-address-table address 5c:f3:fc:ba:13:34 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 5c:f3:fc:ba:13:34 1010 48 FWD bbn-8052.bbn.xo>show mac-address-table address 5c:f3:fc:ba:13:36 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 5c:f3:fc:ba:13:36 1012 53 TRK bbn-8052.bbn.xo>show mac-address-table address 5c:f3:fc:ba:4e:78 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 5c:f3:fc:ba:4e:78 1007 38 FWD bbn-8052.bbn.xo>show mac-address-table address 5c:f3:fc:ba:51:d4 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 5c:f3:fc:ba:51:d4 1007 40 FWD bbn-8052.bbn.xo>show mac-address-table address 5c:f3:fc:ba:56:68 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 5c:f3:fc:ba:56:68 1011 37 FWD 5c:f3:fc:ba:56:68 1007 37 FWD bbn-8052.bbn.xo>show mac-address-table address 5c:f3:fc:ba:56:68 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 5c:f3:fc:ba:56:68 1011 37 FWD 5c:f3:fc:ba:56:68 1007 37 FWD bbn-8052.bbn.xo>show mac-address-table address 5c:f3:fc:ba:65:c0 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 5c:f3:fc:ba:65:c0 1007 39 FWD bbn-8052.bbn.xo>show mac-address-table address 5c:f3:fc:c0:14:c5 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 5c:f3:fc:c0:14:c5 1006 1 FWD bbn-8052.bbn.xo>show mac-address-table address 5c:f3:fc:c0:50:09 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 5c:f3:fc:c0:50:09 1006 6 FWD bbn-8052.bbn.xo>show mac-address-table address 5c:f3:fc:c0:53:65 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 5c:f3:fc:c0:53:65 1006 2 FWD bbn-8052.bbn.xo>show mac-address-table address 5c:f3:fc:c0:57:f9 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 5c:f3:fc:c0:57:f9 1006 5 FWD bbn-8052.bbn.xo>show mac-address-table address 5c:f3:fc:c0:67:51 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 5c:f3:fc:c0:67:51 1006 7 FWD bbn-8052.bbn.xo>show mac-address-table address 6c:ae:8b:1b:e7:8a MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 6c:ae:8b:1b:e7:8a 1008 46 FWD bbn-8052.bbn.xo>show mac-address-table address 6c:ae:8b:1b:e7:8e MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 6c:ae:8b:1b:e7:8e 1006 15 FWD bbn-8052.bbn.xo>show mac-address-table address 6c:ae:8b:1b:e7:f2 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 6c:ae:8b:1b:e7:f2 1007 42 FWD bbn-8052.bbn.xo>show mac-address-table address 6c:ae:8b:1b:e7:f6 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 6c:ae:8b:1b:e7:f6 1006 11 FWD bbn-8052.bbn.xo>show mac-address-table address 6c:ae:8b:1b:e9:d2 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 6c:ae:8b:1b:e9:d2 1008 45 FWD bbn-8052.bbn.xo>show mac-address-table address 6c:ae:8b:1b:e9:d6 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 6c:ae:8b:1b:e9:d6 1006 14 FWD bbn-8052.bbn.xo>show mac-address-table address 6c:ae:8b:1b:ee:fa MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 6c:ae:8b:1b:ee:fa 1007 41 FWD bbn-8052.bbn.xo>show mac-address-table address 6c:ae:8b:1b:ee:fe MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 6c:ae:8b:1b:ee:fe 1006 10 FWD bbn-8052.bbn.xo>show mac-address-table address 6c:ae:8b:1b:ef:52 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 6c:ae:8b:1b:ef:52 1007 43 FWD bbn-8052.bbn.xo>show mac-address-table address 6c:ae:8b:1b:ef:56 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 6c:ae:8b:1b:ef:56 1006 12 FWD bbn-8052.bbn.xo>show mac-address-table address 6c:ae:8b:1b:f0:aa MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 6c:ae:8b:1b:f0:aa 1007 44 FWD bbn-8052.bbn.xo>show mac-address-table address 6c:ae:8b:1b:f0:ae MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- 6c:ae:8b:1b:f0:ae 1006 13 FWD bbn-8052.bbn.xo>show mac-address-table address cc:ef:48:7a:7a:a9 MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- cc:ef:48:7a:7a:a9 1010 48 FWD bbn-8052.bbn.xo>show mac-address-table address f8:c0:01:bb:f8:cb MAC address VLAN Port Trnk State Permanent ----------------- -------- ------- ---- ----- --------- f8:c0:01:bb:f8:cb 1006 9 FWD }}}