Changes between Initial Version and Version 1 of PlasticSlices/BaselineEvaluation/Baseline7Details

Timestamp:

06/29/11 11:21:25 (13 years ago)

Author:

Josh Smift

Comment:

--

PlasticSlices/BaselineEvaluation/Baseline7Details

@@ +1 @@
+[[PageOutline]]
+
+Here are the details of [wiki:PlasticSlices/BaselineEvaluation#Baseline7 Baseline 7], divided by slice.
+
+The raw logs of each experiment are at http://www.gpolab.bbn.com/plastic-slices/baseline-logs/baseline-7/.
+
+= plastic-101 =
+
+[PlasticSlices/Experiments#GigaPing GigaPing], using count=100000, and this table of client/server pairs:
+
+|| '''client'''                   || '''server'''                   || '''server address''' ||
+|| ganel.gpolab.bbn.com           || planetlab5.clemson.edu         || server=10.42.101.105 ||
+|| planetlab4.clemson.edu         || plnode2.cip.gatech.edu         || server=10.42.101.101 ||
+|| plnode1.cip.gatech.edu         || pl5.myplc.grnoc.iu.edu         || server=10.42.101.73  ||
+|| pl4.myplc.grnoc.iu.edu         || orbitplc2.orbit-lab.org        || server=10.42.101.112 ||
+|| orbitplc1.orbit-lab.org        || of-planet2.stanford.edu        || server=10.42.101.91  ||
+|| of-planet1.stanford.edu        || pl02.cs.washington.edu         || server=10.42.101.81  ||
+|| pl01.cs.washington.edu         || wings-openflow-3.wail.wisc.edu || server=10.42.101.96  ||
+|| wings-openflow-2.wail.wisc.edu || gardil.gpolab.bbn.com          || server=10.42.101.52  ||
+
+== Commands run on each client ==
+
+{{{
+server=<ipaddr>
+sudo ping -i .006 -s $((1500-8-20)) $server
+}}}
+
+== Results ==
+
+Generated from the logs with
+
+{{{
+subnet=101
+for host in ganel.gpolab.bbn.com planetlab4.clemson.edu plnode1.cip.gatech.edu pl4.myplc.grnoc.iu.edu orbitplc1.orbit-lab.org of-planet1.stanford.edu pl01.cs.washington.edu wings-openflow-2.wail.wisc.edu ; do echo -e "$host:\n\n{{{" ; grep -A 2 statistics pgenigpolabbbncom_plastic$subnet\@$host.log ; echo -e "}}}\n" ; done
+}}}
+
+== Analysis ==
+
+''(forthcoming)''
+
+= plastic-102 =
+
+[PlasticSlices/Experiments#GigaPing GigaPing], using count=100000, and this table of client/server pairs:
+
+|| '''client'''                   || '''server'''                   || '''server address''' ||
+|| sardis.gpolab.bbn.com          || planetlab4.clemson.edu         || server=10.42.102.104 ||
+|| planetlab5.clemson.edu         || plnode1.cip.gatech.edu         || server=10.42.102.100 ||
+|| plnode2.cip.gatech.edu         || pl4.myplc.grnoc.iu.edu         || server=10.42.102.72  ||
+|| pl5.myplc.grnoc.iu.edu         || orbitplc1.orbit-lab.org        || server=10.42.102.111 ||
+|| orbitplc2.orbit-lab.org        || of-planet2.stanford.edu        || server=10.42.102.91  ||
+|| of-planet1.stanford.edu        || pl01.cs.washington.edu         || server=10.42.102.80  ||
+|| pl02.cs.washington.edu         || wings-openflow-2.wail.wisc.edu || server=10.42.102.95  ||
+|| wings-openflow-3.wail.wisc.edu || bain.gpolab.bbn.com            || server=10.42.102.54  ||
+
+== Commands run on each client ==
+
+{{{
+server=<ipaddr>
+sudo ping -i .006 -s $((1500-8-20)) $server
+}}}
+
+== Results ==
+
+Generated from the logs with
+
+{{{
+subnet=102
+for host in sardis.gpolab.bbn.com planetlab5.clemson.edu plnode2.cip.gatech.edu pl5.myplc.grnoc.iu.edu orbitplc2.orbit-lab.org of-planet1.stanford.edu pl02.cs.washington.edu wings-openflow-3.wail.wisc.edu ; do echo -e "$host:\n\n{{{" ; grep -A 2 statistics pgenigpolabbbncom_plastic$subnet\@$host.log ; echo -e "}}}\n" ; done
+}}}
+
+== Analysis ==
+
+''(forthcoming)''
+
+= plastic-103 =
+
+[PlasticSlices/Experiments#GigaPerfTCP GigaPerf TCP], using port=5103, size=350, and this table of client/server pairs:
+
+|| '''client'''                   || '''server'''                   || '''server address''' ||
+|| of-planet1.stanford.edu        || navis.gpolab.bbn.com           || server=10.42.103.55  ||
+|| ganel.gpolab.bbn.com           || orbitplc1.orbit-lab.org        || server=10.42.103.111 ||
+|| pl02.cs.washington.edu         || bain.gpolab.bbn.com            || server=10.42.103.54  ||
+|| of-planet2.stanford.edu        || pl01.cs.washington.edu         || server=10.42.103.80  ||
+
+== One-time prep commands run on each client and server ==
+
+{{{
+sudo yum -y install iperf
+}}}
+
+== Commands run on each server ==
+
+{{{
+server=<ipaddr>
+nice -n 19 iperf -B $server -p 5103 -s -i 1
+}}}
+
+== Commands run on each client ==
+
+{{{
+server=<ipaddr>
+nice -n 19 iperf -c $server -p 5103 -t 518400
+}}}
+
+== Results ==
+
+Generated with
+
+{{{
+subnet=103
+for host in of-planet1.stanford.edu ganel.gpolab.bbn.com pl02.cs.washington.edu of-planet2.stanford.edu orbitplc2.orbit-lab.org ; do echo -e "$host:\n\n{{{" ; grep -a -A 5 -B 1 "Client connecting" pgenigpolabbbncom_plastic$subnet\@$host.log ; echo -e "}}}\n" ; done
+}}}
+
+and then edited slightly to remove artifacts (like control characters, my prompt, etc).
+
+== Analysis ==
+
+''(forthcoming)''
+
+= plastic-104 =
+
+[PlasticSlices/Experiments#GigaPerfUDP GigaPerf UDP], using port=5104, size=500, rate=100, and this table of client/server pairs:
+
+|| '''client'''                   || '''server'''                   || '''server address''' ||
+|| navis.gpolab.bbn.com           || wings-openflow-2.wail.wisc.edu || server=10.42.104.95  ||
+|| wings-openflow-3.wail.wisc.edu || orbitplc2.orbit-lab.org        || server=10.42.104.112 ||
+|| orbitplc1.orbit-lab.org        || pl4.myplc.grnoc.iu.edu         || server=10.42.104.72  ||
+|| pl5.myplc.grnoc.iu.edu         || gardil.gpolab.bbn.com          || server=10.42.104.52  ||
+
+== One-time prep commands run on each client and server ==
+
+{{{
+sudo yum -y install iperf
+}}}
+
+== Commands run on each server ==
+
+{{{
+server=<ipaddr>
+nice -n 19 iperf -u -B $server -p 5104 -s -i 1
+}}}
+
+== Commands run on each client ==
+
+{{{
+server=<ipaddr>
+nice -n 19 iperf -u -c $server -p 5104 -t 518400 -b 3M
+}}}
+
+== Results ==
+
+Generated with
+
+{{{
+subnet=104
+for host in navis.gpolab.bbn.com wings-openflow-3.wail.wisc.edu orbitplc1.orbit-lab.org pl5.myplc.grnoc.iu.edu ; do echo -e "$host:\n\n{{{" ; grep -A 10 -B 1 "Client connecting" pgenigpolabbbncom_plastic$subnet\@$host.log ; echo -e "}}}\n" ; done
+}}}
+
+and then edited slightly to remove artifacts (like control characters, my prompt, etc).
+
+== Analysis ==
+
+''(forthcoming)''
+
+= plastic-105 =
+
+[PlasticSlices/Experiments#GigaPerfTCP GigaPerf TCP], using port=5105, size=250, and this table of client/server pairs:
+
+|| '''client'''                   || '''server'''                   || '''server address''' ||
+|| wings-openflow-2.wail.wisc.edu || planetlab5.clemson.edu         || server=10.42.105.105 ||
+|| planetlab4.clemson.edu         || sardis.gpolab.bbn.com          || server=10.42.105.53  ||
+|| bain.gpolab.bbn.com            || plnode2.cip.gatech.edu         || server=10.42.105.101 ||
+|| plnode1.cip.gatech.edu         || wings-openflow-3.wail.wisc.edu || server=10.42.105.96  ||
+
+== One-time prep commands run on each client and server ==
+
+{{{
+sudo yum -y install iperf
+}}}
+
+== Commands run on each server ==
+
+{{{
+server=<ipaddr>
+nice -n 19 iperf -B $server -p 5105 -s -i 1
+}}}
+
+== Commands run on each client ==
+
+{{{
+server=<ipaddr>
+nice -n 19 iperf -c $server -p 5105 -t 518400
+}}}
+
+== Results ==
+
+Generated with
+
+{{{
+subnet=105
+for host in wings-openflow-2.wail.wisc.edu planetlab4.clemson.edu bain.gpolab.bbn.com plnode1.cip.gatech.edu ; do echo -e "$host:\n\n{{{" ; grep -A 5 -B 1 "Client connecting" pgenigpolabbbncom_plastic$subnet\@$host.log ; echo -e "}}}\n" ; done
+}}}
+
+and then edited slightly to remove artifacts (like control characters, my prompt, etc).
+
+== Analysis ==
+
+''(forthcoming)''
+
+= plastic-106 =
+
+[PlasticSlices/Experiments#GigaPerfUDP GigaPerf UDP], using port=5106, size=500, rate=100, and this table of client/server pairs:
+
+|| '''client'''                   || '''server'''                   || '''server address''' ||
+|| planetlab5.clemson.edu         || wings-openflow-2.wail.wisc.edu || server=10.42.106.95  ||
+|| wings-openflow-3.wail.wisc.edu || plnode1.cip.gatech.edu         || server=10.42.106.100 ||
+|| plnode2.cip.gatech.edu         || ganel.gpolab.bbn.com           || server=10.42.106.51  ||
+|| gardil.gpolab.bbn.com          || planetlab4.clemson.edu         || server=10.42.106.104 ||
+
+== One-time prep commands run on each client and server ==
+
+{{{
+sudo yum -y install iperf
+}}}
+
+== Commands run on each server ==
+
+{{{
+server=<ipaddr>
+nice -n 19 iperf -u -B $server -p 5106 -s -i 1
+}}}
+
+== Commands run on each client ==
+
+{{{
+server=<ipaddr>
+nice -n 19 iperf -u -c $server -p 5106 -t 518400 -b 3M
+}}}
+
+== Results ==
+
+Generated with
+
+{{{
+subnet=106
+for host in planetlab5.clemson.edu wings-openflow-3.wail.wisc.edu plnode2.cip.gatech.edu gardil.gpolab.bbn.com ; do echo -e "$host:\n\n{{{" ; grep -A 10 -B 1 "Client connecting" pgenigpolabbbncom_plastic$subnet\@$host.log ; echo -e "}}}\n" ; done
+}}}
+
+and then edited slightly to remove artifacts (like control characters, my prompt, etc).
+
+== Analysis ==
+
+''(forthcoming)''
+
+= plastic-107 =
+
+[PlasticSlices/Experiments#GigaWeb GigaWeb], using count=30, port=4107, file=substrate.doc, md5sum=d4fcf71833327fbfef98be09deef8bfb, and this table of client/server pairs:
+
+|| '''client'''                   || '''server'''                   || '''server address''' ||
+|| planetlab5.clemson.edu         || pl4.myplc.grnoc.iu.edu         || server=10.42.107.72  ||
+|| pl5.myplc.grnoc.iu.edu         || plnode2.cip.gatech.edu         || server=10.42.107.101 ||
+|| plnode1.cip.gatech.edu         || pl02.cs.washington.edu         || server=10.42.107.81  ||
+|| pl01.cs.washington.edu         || planetlab4.clemson.edu         || server=10.42.107.104 ||
+
+== One-time prep commands run on each server ==
+
+{{{
+sudo yum -y install pyOpenSSL patch
+cd
+rm -rf ~/gigaweb
+mkdir -p ~/gigaweb/docroot
+
+cd ~/gigaweb
+wget http://code.activestate.com/recipes/442473-simple-http-server-supporting-ssl-secure-communica/download/1/ -O httpsd.py
+wget http://groups.geni.net/geni/attachment/wiki/PlasticSlices/Experiments/httpsd.py.patch?format=raw -O httpsd.py.patch
+patch httpsd.py httpsd.py.patch
+rm httpsd.py.patch
+
+cd ~/gigaweb/docroot
+wget http://groups.geni.net/geni/attachment/wiki/DeliverablePage/Spiral1%20substrate%20catalog.doc?format=raw -O substrate.doc
+
+cd ~/gigaweb
+openssl genrsa -passout pass:localhost -des3 -rand /dev/urandom -out localhost.localdomain.key 1024
+openssl req -subj /CN=localhost.localdomain -passin pass:localhost -new -key localhost.localdomain.key -out localhost.localdomain.csr
+openssl x509 -passin pass:localhost -req -days 3650 -in localhost.localdomain.csr -signkey localhost.localdomain.key -out localhost.localdomain.crt
+openssl rsa -passin pass:localhost -in localhost.localdomain.key -out decrypted.localhost.localdomain.key
+mv -f decrypted.localhost.localdomain.key localhost.localdomain.key
+cat localhost.localdomain.key localhost.localdomain.crt > localhost.localdomain.pem
+rm localhost.localdomain.key localhost.localdomain.crt localhost.localdomain.csr
+}}}
+
+== Commands run on each server ==
+
+{{{
+server=<ipaddr>
+cd ~/gigaweb/docroot
+python ../httpsd.py $server 4107
+}}}
+
+== Commands run on each client ==
+
+{{{
+server=<ipaddr>
+cd
+rm -rf ~/gigaweb
+mkdir ~/gigaweb
+cd ~/gigaweb
+while true ; do wget --no-check-certificate https://$server:4107/substrate.doc -O substrate.doc ; echo -n "md5sum: " ; md5sum substrate.doc ; rm substrate.doc ; done
+}}}
+
+== Results ==
+
+Since we transfered the file to each client hundreds (if not thousands) of times, we didn't save all the copies, but we did generate an MD5 checksum after each download, and log that, so we ran some grep commands on the logs to count the number of total checksums printed in each log, and the number where the checksum had the expected value.
+
+Generated with
+
+{{{
+subnet=107
+for host in planetlab5.clemson.edu pl5.myplc.grnoc.iu.edu plnode1.cip.gatech.edu pl01.cs.washington.edu ; do echo -e -n "$host:\n\n{{{\ntotal md5sum lines: " ; grep "md5sum:" pgenigpolabbbncom_plastic$subnet\@$host.log | wc -l ; echo -e -n "matching md5sum lines: " ; grep "md5sum: d4fcf71833327fbfef98be09deef8bfb" pgenigpolabbbncom_plastic$subnet\@$host.log | wc -l ; echo -e "}}}\n" ; done
+}}}
+
+== Analysis ==
+
+''(forthcoming)''
+
+= plastic-108 =
+
+[PlasticSlices/Experiments#GigaWeb GigaWeb], using count=40, port=4108, file=substrate.doc, md5sum=d4fcf71833327fbfef98be09deef8bfb, and this table of client/server pairs:
+
+|| '''client'''                   || '''server'''                   || '''server address''' ||
+|| wings-openflow-3.wail.wisc.edu || of-planet2.stanford.edu        || server=10.42.108.91  ||
+|| of-planet1.stanford.edu        || orbitplc1.orbit-lab.org        || server=10.42.108.111 ||
+|| orbitplc2.orbit-lab.org        || pl5.myplc.grnoc.iu.edu         || server=10.42.108.73  ||
+|| pl4.myplc.grnoc.iu.edu         || wings-openflow-2.wail.wisc.edu || server=10.42.108.95  ||
+
+== One-time prep commands run on each server ==
+
+{{{
+sudo yum -y install pyOpenSSL patch
+cd
+rm -rf ~/gigaweb
+mkdir -p ~/gigaweb/docroot
+
+cd ~/gigaweb
+wget http://code.activestate.com/recipes/442473-simple-http-server-supporting-ssl-secure-communica/download/1/ -O httpsd.py
+wget http://groups.geni.net/geni/attachment/wiki/PlasticSlices/Experiments/httpsd.py.patch?format=raw -O httpsd.py.patch
+patch httpsd.py httpsd.py.patch
+rm httpsd.py.patch
+
+cd ~/gigaweb/docroot
+wget http://groups.geni.net/geni/attachment/wiki/DeliverablePage/Spiral1%20substrate%20catalog.doc?format=raw -O substrate.doc
+
+cd ~/gigaweb
+openssl genrsa -passout pass:localhost -des3 -rand /dev/urandom -out localhost.localdomain.key 1024
+openssl req -subj /CN=localhost.localdomain -passin pass:localhost -new -key localhost.localdomain.key -out localhost.localdomain.csr
+openssl x509 -passin pass:localhost -req -days 3650 -in localhost.localdomain.csr -signkey localhost.localdomain.key -out localhost.localdomain.crt
+openssl rsa -passin pass:localhost -in localhost.localdomain.key -out decrypted.localhost.localdomain.key
+mv -f decrypted.localhost.localdomain.key localhost.localdomain.key
+cat localhost.localdomain.key localhost.localdomain.crt > localhost.localdomain.pem
+rm localhost.localdomain.key localhost.localdomain.crt localhost.localdomain.csr
+}}}
+
+== Commands run on each server ==
+
+{{{
+server=<ipaddr>
+cd ~/gigaweb/docroot
+python ../httpsd.py $server 4108
+}}}
+
+== Commands run on each client ==
+
+{{{
+server=<ipaddr>
+cd
+rm -rf ~/gigaweb
+mkdir ~/gigaweb
+cd ~/gigaweb
+while true ; do wget --no-check-certificate https://$server:4108/substrate.doc -O substrate.doc ; echo -n "md5sum: " ; md5sum substrate.doc ; rm substrate.doc ; done
+}}}
+
+== Results ==
+
+Since we transfered the file to each client hundreds (if not thousands) of times, we didn't save all the copies, but we did generate an MD5 checksum after each download, and log that, so we ran some grep commands on the logs to count the number of total checksums printed in each log, and the number where the checksum had the expected value.
+
+Generated with
+
+{{{
+subnet=108
+for host in wings-openflow-3.wail.wisc.edu of-planet1.stanford.edu orbitplc2.orbit-lab.org pl4.myplc.grnoc.iu.edu ; do echo -e -n "$host:\n\n{{{\ntotal md5sum lines: " ; grep "md5sum:" pgenigpolabbbncom_plastic$subnet\@$host.log | wc -l ; echo -e -n "matching md5sum lines: " ; grep "md5sum: d4fcf71833327fbfef98be09deef8bfb" pgenigpolabbbncom_plastic$subnet\@$host.log | wc -l ; echo -e "}}}\n" ; done
+}}}
+
+== Analysis ==
+
+''(forthcoming)''
+
+= plastic-109 =
+
+[PlasticSlices/Experiments#GigaNetcat GigaNetcat], using count=25, port=6109, file=substrate.doc, and this table of client/server pairs:
+
+|| '''client'''                   || '''server'''                   || '''server address''' ||
+|| navis.gpolab.bbn.com           || pl5.myplc.grnoc.iu.edu         || server=10.42.109.73  ||
+|| pl4.myplc.grnoc.iu.edu         || pl02.cs.washington.edu         || server=10.42.109.81  ||
+|| pl01.cs.washington.edu         || planetlab5.clemson.edu         || server=10.42.109.105 ||
+|| planetlab4.clemson.edu         || of-planet1.stanford.edu        || server=10.42.109.90  ||
+|| of-planet2.stanford.edu        || wings-openflow-3.wail.wisc.edu || server=10.42.109.96  ||
+|| wings-openflow-2.wail.wisc.edu || ganel.gpolab.bbn.com           || server=10.42.109.51  ||
+
+== One-time prep commands run on each server ==
+
+{{{
+sudo yum -y install nc
+mkdir -p ~/giganetcat
+cd ~/giganetcat
+wget http://groups.geni.net/geni/attachment/wiki/DeliverablePage/Spiral1%20substrate%20catalog.doc?format=raw -O substrate.doc
+}}}
+
+== One-time prep commands run on each client ==
+
+{{{
+sudo yum -y install nc
+}}}
+
+== Commands run on each server ==
+
+{{{
+server=<ipaddr>
+cd ~/giganetcat
+while true ; do nc -l $server 6109 < substrate.doc ; done
+}}}
+
+== Commands run on each client ==
+
+{{{
+server=<ipaddr>
+cd
+rm -rf ~/giganetcat
+mkdir ~/giganetcat
+cd ~/giganetcat
+while true ; do nc $server 6109 > substrate.doc ; echo -n "$(date "+%F %T")  " ; md5sum substrate.doc ; rm substrate.doc ; done
+}}}
+
+== Results ==
+
+Since we transfered the file to each client hundreds (if not thousands) of times, we didn't save all the copies, but we did generate an MD5 checksum after each download, and log that, so we ran some grep commands on the logs to count the number of total checksums printed in each log, and the number where the checksum had the expected value.
+
+Generated with
+
+{{{
+subnet=109
+for host in navis.gpolab.bbn.com pl4.myplc.grnoc.iu.edu pl01.cs.washington.edu planetlab4.clemson.edu of-planet2.stanford.edu wings-openflow-2.wail.wisc.edu ; do echo -e -n "$host:\n\n{{{\ntotal md5sum lines: " ; grep "substrate.doc" pgenigpolabbbncom_plastic$subnet\@$host.log | wc -l ; echo -e -n "matching md5sum lines: " ; grep "d4fcf71833327fbfef98be09deef8bfb  substrate.doc" pgenigpolabbbncom_plastic$subnet\@$host.log | wc -l ; echo -e "}}}\n" ; done
+}}}
+
+== Analysis ==
+
+''(forthcoming)''
+
+= plastic-110 =
+
+[PlasticSlices/Experiments#GigaNetcat GigaNetcat], using count=25, port=6110, file=substrate.doc, and this table of client/server pairs:
+
+|| '''client'''                   || '''server'''                   || '''server address''' ||
+|| gardil.gpolab.bbn.com          || pl01.cs.washington.edu         || server=10.42.110.80  ||
+|| pl02.cs.washington.edu         || of-planet1.stanford.edu        || server=10.42.110.90  ||
+|| of-planet2.stanford.edu        || orbitplc2.orbit-lab.org        || server=10.42.110.112 ||
+|| orbitplc1.orbit-lab.org        || pl4.myplc.grnoc.iu.edu         || server=10.42.110.72  ||
+|| pl5.myplc.grnoc.iu.edu         || plnode1.cip.gatech.edu         || server=10.42.110.100 ||
+|| plnode2.cip.gatech.edu         || sardis.gpolab.bbn.com          || server=10.42.110.53  ||
+
+== One-time prep commands run on each server ==
+
+{{{
+sudo yum -y install nc
+mkdir -p ~/giganetcat
+cd ~/giganetcat
+wget http://groups.geni.net/geni/attachment/wiki/DeliverablePage/Spiral1%20substrate%20catalog.doc?format=raw -O substrate.doc
+}}}
+
+== One-time prep commands run on each client ==
+
+{{{
+sudo yum -y install nc
+}}}
+
+== Commands run on each server ==
+
+{{{
+server=<ipaddr>
+cd ~/giganetcat
+while true ; do nc -l $server 6110 < substrate.doc ; done
+}}}
+
+== Commands run on each client ==
+
+{{{
+server=<ipaddr>
+cd
+rm -rf ~/giganetcat
+mkdir ~/giganetcat
+cd ~/giganetcat
+while true ; do nc $server 6110 > substrate.doc ; echo -n "$(date "+%F %T")  " ; md5sum substrate.doc ; rm substrate.doc ; done
+}}}
+
+== Results ==
+
+Since we transfered the file to each client hundreds (if not thousands) of times, we didn't save all the copies, but we did generate an MD5 checksum after each download, and log that, so we ran some grep commands on the logs to count the number of total checksums printed in each log, and the number where the checksum had the expected value.
+
+Generated with
+
+{{{
+subnet=110
+for host in gardil.gpolab.bbn.com pl02.cs.washington.edu of-planet2.stanford.edu orbitplc1.orbit-lab.org pl5.myplc.grnoc.iu.edu plnode2.cip.gatech.edu ; do echo -e -n "$host:\n\n{{{\ntotal md5sum lines: " ; grep "substrate.doc" pgenigpolabbbncom_plastic$subnet\@$host.log | wc -l ; echo -e -n "matching md5sum lines: " ; grep "d4fcf71833327fbfef98be09deef8bfb  substrate.doc" pgenigpolabbbncom_plastic$subnet\@$host.log | wc -l ; echo -e "}}}\n" ; done
+}}}
+
+== Analysis ==
+
+''(forthcoming)''