Changes between Initial Version and Version 1 of OMF5.3ExperimentAnalysis


Ignore:
Timestamp:
09/07/11 17:22:24 (10 years ago)
Author:
agosain@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • OMF5.3ExperimentAnalysis

    v1 v1  
     1[[PageOutline]]
     2
     3==   Processing and plotting measurement data  ==
     4
     5       
     6The collected results corresponding to each run of the experiment would be stored in a sqlite3(*.sq3) file. We have created scripts that analyze the collected results, and create CSV files based on the analysis. These CSV files are later fed into R scripts to plot graphs. The whole process is automated. These scripts are available at http://software.geni.net/local-sw/
     7
     8        Processing can be done in any Linux server, mobile node or another server
     9
     10
     11The post processing scripts run on any flavor of Linux. This machine can be a experimenters laptop or the Central OML server.
     12
     13
     14- Install '''sqlite3''' using aptitude if you're running ubuntu, or get it from http://www.sqlite.org/ .
     15
     16- Install '''R''' and its packages from: http://cran.r-project.org/doc/manuals/R-admin.html
     17
     18- Install '''gplots''' add-on for '''R''' ( http://cran.r-project.org/web/packages/gplots/index.html ) in order to run the sample '''R''' plotting script.
     19
     20        Measurement data is retrieved from OML server sql db;  how?
     21
     22        Measurement data is processed with a script
     23
     24The following code, describes a sample ruby script for the analysis of the results of a set of "udp-dual" experiments. It assumes that all the relevant sq3 files are in the "basedir" directory with a name with the following pattern: point#{i}_*.sq3 where i is a number/index indicating the position of measurement.
     25
     26The script takes as input a text file with the list of the points of interest (one position index per line) and prints a csv file with each line containing the point#,avgBW,stdBW,avgRSSI,stdRSSI,avgCINR,stdCINR , that is, the average and standard error of bandwidth, rssi, and cinr for each point, based on the experiments performed on that point. Note that in this analysis we only consider the downlink throughput in the dual-mode test, the data for which is collected on the mobile client (for uplink throughput, you will have to look at the data collected on the server side. A very similar analysis can be performed on the sq3 files collected there). Downlink connection ID can be determined knowing the local_address and local_port to which the connection is made.
     27
     28{{{
     29#----------------------------------------------------------------------
     30# Copyright (c) 2011 Raytheon BBN Technologies
     31#
     32# Permission is hereby granted, free of charge, to any person obtaining
     33# a copy of this software and/or hardware specification (the "Work") to
     34# deal in the Work without restriction, including without limitation the
     35# rights to use, copy, modify, merge, publish, distribute, sublicense,
     36# and/or sell copies of the Work, and to permit persons to whom the Work
     37# is furnished to do so, subject to the following conditions:
     38#
     39# The above copyright notice and this permission notice shall be
     40# included in all copies or substantial portions of the Work.
     41#
     42# THE WORK IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
     43# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
     44# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
     45# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
     46# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
     47# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
     48# OUT OF OR IN CONNECTION WITH THE WORK OR THE USE OR OTHER DEALINGS
     49# IN THE WORK.
     50#----------------------------------------------------------------------
     51
     52#!/usr/bin/ruby
     53
     54require 'rubygems'
     55require 'sqlite3'
     56
     57######### Analysis File Dir#############
     58
     59# MAKE SURE THESE PARAMETERS ARE RIGHT #
     60
     61basedir = './dual_udp'
     62local_server_addr='192.1.240.106'
     63local_server_port='5001'
     64foreign_server_addr='192.1.240.126'
     65foreign_server_port='5001'
     66
     67########################################
     68
     69
     70module Enumerable
     71
     72        def sum
     73                return self.inject(0) { |acc,i|acc +i }
     74        end
     75
     76        def average
     77                return self.sum/self.length.to_f
     78        end
     79
     80        def sample_variance
     81                avg=self.average
     82                sum=self.inject(0) {|acc,i| acc + (i-avg)**2}
     83                return(1/self.length.to_f*sum)
     84        end
     85
     86        def standard_deviation
     87                return Math.sqrt(self.sample_variance)
     88        end
     89
     90        def standard_error
     91                return self.standard_deviation/Math.sqrt(self.length.to_f)
     92        end
     93
     94end #module Enumerable
     95
     96##################### Analysis starts here ######################
     97
     98# read path points from the input_file. Note that input_file must
     99# have a valid point number and only a valid point number in each
     100# line.
     101#################################################################
     102
     103
     104unless ARGV.length == 1
     105        puts "Usage: new_clientside_analyze_dual_udp input_file(s)"
     106        puts "Note: Each input_file must have a valid point number"
     107        puts "      and only a valid point number in each line."
     108        exit
     109end
     110
     111# read path_points for each input_file
     112input_file = ARGV[0]
     113if File.exists?(input_file)
     114        path_points=[]
     115        f = File.open(input_file) or die "Unable to open file..."
     116        f.each_line{ |line|
     117                path_points.push line.to_i
     118        }
     119        #puts path_points.inspect
     120
     121        # uncomment if you're interested in having headers
     122        #puts "pos#,avgBW,stdBW,avgRSSI,stdRSSI,avgCINR,stdCINR"
     123
     124        # for each point on the path print out the measurements:
     125        for i in path_points
     126
     127                avgBandwidth = Array.new
     128                avgRSSI = Array.new
     129                avgCINR = Array.new
     130
     131
     132                # get corresponding sqlite3 files
     133                files = Dir["#{basedir}/point#{i}_*.sq3"]
     134                if files.empty? then
     135                        #puts "No proper files found for i=#{i}"
     136                        puts "#{i},0.0,0.0,0.0,0.0,0.0,0.0"
     137                        next
     138                end
     139               
     140
     141                # each file has the average throughput as its last entry. get that.
     142                files.each do |f|
     143                        db = SQLite3::Database.new(f)
     144
     145                        # throughput measurements of both directions are pushed into the same table.
     146                        # they can be distinguished by their different connection ids.
     147                        # connection ids can be found in a separate table.
     148               
     149                        # fist find the downlink and uplink connection ids.
     150                       
     151                        #uplink_ID_query = "select connection_id from iperf_connection where foreign_address='#{foreign_server_addr}' and foreign_port='#{foreign_server_port}' ;"
     152                       
     153                        #uplink_ID   = db.get_first_value(uplink_ID_query);
     154
     155                        downlink_conID_query = "select connection_id from iperf_connection where local_address='#{local_server_addr}' and local_port='#{local_server_port}';"                   
     156                        downlink_conID = db.get_first_value(downlink_conID_query);
     157                        #puts downlink_conID;
     158                               
     159                        # now calucate the downlink bandwidth
     160                        downlink_result = db.get_first_value( "select size*8/(1024*(end_interval-begin_interval)) from iperf_transfer where (end_interval-begin_interval > 1) and connection_id='#{downlink_conID}' order by oml_seq desc limit 1;" )   
     161
     162                        # note that if the file exists but query result is nil, throughput is 0.0
     163                        if downlink_result.nil? then
     164                                #puts "Query result for (Bandwidth) is nil. using 0.0 for i=#{i}"
     165                                # this could be due to the problem with the last element not being the average (iperf issue).
     166                                downlink_result2 = db.get_first_value( "select sum(size)*8/(1024*End_interval) from iperf_transfer" )
     167                                #puts "for i=#{i} , result2 is #{result2}"
     168                                if downlink_result2.nil? or (downlink_result2<0) then
     169                                        avgBandwidth << 0.0
     170                                else
     171                                        avgBandwidth << downlink_result2
     172                                end
     173                        else
     174                                avgBandwidth << downlink_result
     175                        end
     176
     177
     178                        # now let's get average/stdD RSSI
     179                        result = db.get_first_value( "select avg(RSSI) from wimaxcu_wimaxstat" )
     180                        if result.nil? then
     181                                #puts "Query result for (RSSI) is nil. using 0.0 for i=#{i}"
     182                                avgRSSI << 0.0
     183                        else
     184                                avgRSSI << result
     185                        end
     186
     187                        # now let's get average/stdD CINR
     188                        result = db.get_first_value( "select avg(CINR) from wimaxcu_wimaxstat" )
     189                        if result.nil? then
     190                                #puts "Query result is nil. using 0.0 for i=#{i}"
     191                                avgCINR << 0.0
     192                        else
     193                                avgCINR << result
     194                        end
     195
     196                end
     197                puts "#{i},#{avgBandwidth.average},#{avgBandwidth.standard_error},#{avgRSSI.average},#{avgRSSI.standard_error},#{avgCINR.average},#{avgCINR.standard_error}"
     198        end # end for each point
     199
     200else
     201        abort("Invalid input_file name.")               
     202end     
     203}}}
     204
     205
     206
     207       
     208The CSV file created by the above script, can be fed into R scripts that plot the extracted data. The following is an example of a R script that plots throughput data:
     209
     210{{{
     211#----------------------------------------------------------------------
     212# Copyright (c) 2011 Raytheon BBN Technologies
     213#
     214# Permission is hereby granted, free of charge, to any person obtaining
     215# a copy of this software and/or hardware specification (the "Work") to
     216# deal in the Work without restriction, including without limitation the
     217# rights to use, copy, modify, merge, publish, distribute, sublicense,
     218# and/or sell copies of the Work, and to permit persons to whom the Work
     219# is furnished to do so, subject to the following conditions:
     220#
     221# The above copyright notice and this permission notice shall be
     222# included in all copies or substantial portions of the Work.
     223#
     224# THE WORK IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
     225# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
     226# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
     227# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
     228# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
     229# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
     230# OUT OF OR IN CONNECTION WITH THE WORK OR THE USE OR OTHER DEALINGS
     231# IN THE WORK.
     232#----------------------------------------------------------------------
     233
     234#! /usr/local/bin/Rscript
     235
     236##########################################################################
     237#       This is the R script to plot dual udp 'throughput' data                  #
     238##########################################################################
     239
     240
     241# get input/output files as arguments
     242args <- commandArgs(TRUE);
     243
     244print (args);
     245infile <- args[[1]];
     246outfile <- args[[2]];
     247
     248print(infile);
     249print(outfile);
     250
     251# skip list (puts annotation markers for points in this list). This to mark points where the node could not connect or no experiment was done.
     252# skip_note_y : y coordinate of where to put the marker. This should be manually modified based on the range of y axis.
     253skip_list = c();
     254skip_note_y = c(1060);
     255
     256## setup the device
     257pdf(outfile, width=4, height=3, pointsize=10);
     258oldpar <- par(font.lab=2, font.axis=2,
     259                mar = c(4,4,.5,.5),
     260                oma = c(0,0,0,0),
     261                mgp = c(2.5,1,0)
     262             );
     263
     264## read in the data
     265data <- read.table(infile, header=FALSE, sep=",");
     266locations <- data[[1]];
     267means.throughput <- data[[2]];
     268
     269#stderr.throughput <- data[[3]];
     270names(means.throughput) <- locations
     271
     272# use gplots library to draw error bars
     273library(gplots);
     274plot(x=means.throughput, pch=20, lty=1, xaxt="n",  ylim=c(0,max(means.throughput)*1.5), gap=0, ylab="Avg. Downlink Throughput (Kb/s)", xlab="Location" );
     275
     276# turn on grids
     277abline(h=seq(0,max(means.throughput)*1.5,by=1000),lty=2, col="gray");
     278
     279# Put markers for skip_list
     280for (i in locations) {
     281        if (i %in% skip_list)
     282        text(which(locations==i,arr.ind=TRUE),skip_note_y,labels="*",col="red");
     283}
     284
     285# Draw the x-axis and y-axis (omitted above)
     286axis(side=1, at=seq(1,length(locations), by=1), labels=paste("P",names(means.throughput),sep=""), cex=1.0);
     287}}}
     288
     289You can automate the process of graph creation by writing a bash script (make_plots.sh) similar to the following and running it in shell:
     290
     291{{{
     292#----------------------------------------------------------------------
     293# Copyright (c) 2011 Raytheon BBN Technologies
     294#
     295# Permission is hereby granted, free of charge, to any person obtaining
     296# a copy of this software and/or hardware specification (the "Work") to
     297# deal in the Work without restriction, including without limitation the
     298# rights to use, copy, modify, merge, publish, distribute, sublicense,
     299# and/or sell copies of the Work, and to permit persons to whom the Work
     300# is furnished to do so, subject to the following conditions:
     301#
     302# The above copyright notice and this permission notice shall be
     303# included in all copies or substantial portions of the Work.
     304#
     305# THE WORK IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
     306# OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
     307# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
     308# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
     309# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
     310# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
     311# OUT OF OR IN CONNECTION WITH THE WORK OR THE USE OR OTHER DEALINGS
     312# IN THE WORK.
     313#----------------------------------------------------------------------
     314
     315#!/bin/bash
     316
     317./new_clientside_analyze_dual_udp.rb ptest > text/ptest_dual_udp_log.txt
     318
     319R --no-save --no-restore --no-environ --args text/ptest_dual_udp_log.txt figures/dual/ptest_udp_dual_throughput.pdf < Rscripts/plot_udp_dual_throughput.R
     320}}}
     321
     322The first line uses the analysis script to pull data out of the sq3 files and create a csv file, putting it in the 'text/ptest_dual_udp_log.txt'. The file "ptest" contains EXACTLY two lines, one with number 1 and the other with number 2 on it indicating the point indices. The next line calls the R script to read from the that text file, and create a pdf figure containing throughput data. Similar R scripts can be used for plotting RSSI and CINR, being called in the same way.
     323
     324
     325       
     326
     327