Changes between Version 12 and Version 13 of GEC17Agenda/GettingStartedWithGENI_III_GIMI/Procedure/Finish


Ignore:
Timestamp:
07/18/13 16:01:33 (11 years ago)
Author:
divyashri.bhat@gmail.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • GEC17Agenda/GettingStartedWithGENI_III_GIMI/Procedure/Finish

    v12 v13  
    11= Finish =
    22
    3 == D. Push to iRODS ==
    4 
    5 === D.1 Overview ===
    6 
    7 In GIMI, [https://www.irods.org/ iRODS] is used as the repository for measurement data. At the moment our iRODS data system consist of three servers (RENCI, NICTA, and UMass) and a metadata catalog (located at RENCI).
    8 
    9 ----
    10 
    11 === D.2 Storing data on iRODS ===
    12 
    13 * After successful completion of an experiment the measurement data from the experiment have been stored in iRODS. We will now look into two options on how this data can be handled.
    14  
    15 === D.2.1 iRODS command line tools in the user work space ===
    16   * The GENI [http://groups.geni.net/geni/wiki/GECTutorialVMInstructions User Workspace] comes already with the iRODS command line tools installed.
    17   * If you have not done so far use [https://www.irods.org/index.php/iinit iinit] to create a file that contains your iRODS password in a scrambled form. This will then be used automatically by the [https://www.irods.org/index.php/icommands icommands] for authentication with the server.
    18 {{{
    19 $ iinit
    20 }}}
    21     You will be prompted for a password. Please enter the one given on the paper handout.
    22 
    23   * The iRODS client uses a configuration file (~/.irods/.irodsEnv) that sets certain parameter for the icommands. Here is and example:
    24 {{{
    25 # iRODS personal configuration file.
    26 #
    27 # This file was automatically created during iRODS installation.
    28 #   Created Thu Feb 16 14:06:27 2012
    29 #
    30 # iRODS server host name:
    31 irodsHost 'emmy9.casa.umass.edu'
    32 # iRODS server port number:
    33 irodsPort 1247
    34 
    35 # Default storage resource name:
    36 irodsDefResource 'iRODSUmass2'
    37 # Home directory in iRODS:
    38 irodsHome '/geniRenci/home/gimiXX'
    39 # Current directory in iRODS:
    40 irodsCwd '/geniRenci/home/gimiXX'
    41 # Account name:
    42 irodsUserName 'gimiXX'
    43 # Zone:
    44 irodsZone 'geniRenci'
    45 }}}
    46   * Retrieve file from your iRODS home directory into user workspace.
    47 {{{
    48 $ iget <file_name>
    49 }}}
    50 
    51     Assuming you wanted to retrieve a file called gimitesting.sq3 the command would look as follows:
    52 {{{
    53 $ iget gimitesting.sq3
    54 }}}
    55 
    56   * Store data from user workspace into your iRODS home directory.
    57 {{{
    58 $ iput <file_name>
    59 }}}
    60 
    61   * Add metadata information to stored file.
    62 {{{
    63 $ imeta add -d <file_name> <name> <value> <unit>
    64 }}}
    65     For example if you wanted to add the slice name of the experiment as metadata for file gimi01-ping_all.sq3, the command would look as follows:
    66 {{{
    67 $ imeta add -d gimi01-ping_all.sq3 SliceName gimi01
    68 }}}
    69     (<unit> is not required.)
    70 
    71 === D.2.2 iRODS web interface ===
    72 iRODS also provides a nice and easy to use [https://www.irods.org/web/index.php web interface], which we will explore in the following.
    73   * Point the browser in your user workspace to the following link: https://www.irods.org/web/index.php
    74   * Input the following information to sign in:
    75 {{{
    76 Host/IP: emmy9.casa.umass.edu
    77 Port: 1247
    78 Username: as given on printout
    79 Password: as given on printout
    80 }}}
    81   * The following screenshot shows an example for the web interface: [[Image(irods_screen.png)]]
    82 
    83 ----
    84 
    85 === D.3 IRODS and OML ===
    86 In GIMI we have enabled an option in OML2.8 that allows the execution of script. We use this functionality to automatically save measurement results in IRODS after a measurement has successfully completed.
    87 
    88 ----
    89 
    90 === D.4 DISCLAIMER ===
    91 The iRODS service we are offering within the scope of GIMI does NOT guarantee 100% reliable data storage (i.e., we do NOT back up the data). If you are performing your own experiments and want to use iRODS you are absolutely welcome but be aware that we do NOT guarantee recovery from data loss.
    92 
    93 ----
    94 
    95 === E.1 iRODS Environment Settings ===
    96 If you haven't already performed this step in A.1, you first have to configure the your local iRODS environment:
    97 
    98 * The iRODS client uses a configuration file (~/.irods/.irodsEnv) that sets certain parameter for the icommands. Here is and example:
    99 {{{
    100 # iRODS personal configuration file.
    101 #
    102 # This file was automatically created during iRODS installation.
    103 #   Created Thu Feb 16 14:06:27 2012
    104 #
    105 # iRODS server host name:
    106 irodsHost 'emmy9.casa.umass.edu'
    107 # iRODS server port number:
    108 irodsPort 1247
    109 
    110 # Default storage resource name:
    111 irodsDefResource 'iRODSUmass2'
    112 # Home directory in iRODS:
    113 irodsHome '/geniRenci/home/gimiXX'
    114 # Current directory in iRODS:
    115 irodsCwd '/geniRenci/home/gimiXX'
    116 # Account name:
    117 irodsUserName 'gimiXX'
    118 # Zone:
    119 irodsZone 'geniRenci'
    120 }}}
    121  
    122   * First of all copy the iRODS configuration file to .irodsEnv with the following command (replace gimiXX with your username):
    123 {{{
    124 $ cp ~/Tutorials/GIMI/gimiXX/gimiXXIrodsEnv ~/.irods/.irodsEnv
    125 }}}
    126  * Register with iRODS server by issuing the following command (more details on iRODS will be given shortly):
    127 {{{
    128 $ iinit
    129 }}}
    130  
    131  * You will be prompted for a password. Please type in the password you were provided with on the paper handout!!
    132 
    133 ----
    134 
    135 === E.2 Retrieving Data from iRODS to Local System ===
    136 In the following we list the steps and commands that will allow you to retrieve data from the iRODS storage to your
    137 local storage:
    138 
    139   * List content of your home directory
    140 {{{
    141 $ ils
    142 }}}
    143   * To change to a different directory use
    144 {{{
    145 $ icd
    146 }}}
    147   * Retrieve file from your iRODS home directory into user workspace.
    148 {{{
    149 $ iget <file_name>
    150 }}}
    151 
    152   Assuming you wanted to retrieve a file called gimitesting.sq3 the command would look as follows:
    153 {{{
    154 $ iget gimitesting.sq3
    155 }}}
    156   * A complete overview on the iRODS commands can be found [https://www.irods.org/index.php/icommands here].
    157 
    158 === E.3 Searching the Catalogue for Data ===
    159 {{{
    160 $ imeta qu -d SliceName like gimi28
    161 }}}
    162 
    163 
    164 
    165 [[BR]]
    166 [[BR]]
    167 
    168 In Section E we went through the exercise of retrieving data from iRODS to a local computer. In this Section, we will introduce
    169 two different methods that can be used to analyze the measurement data. Analysis of measurement data obtained with OMF/OML is not
    170 limited to these two methods, we simply use them for demonstration purposes.
    171 
    172 ----
    173 
    174 === F.1 R Scripts ===
    175 
    176 One potential way to visualize the data is making use of [http://www.r-project.org/ R], which provides a visualization language.
    177 For this tutorial, we have create a set of R scripts, which we briefly discuss in the following.
    178 
    179 The first [http://emmy9.casa.umass.edu/GEC15-GIMI-Tutorial/R_script_ping.r R script] creates a plot of the RTTs for each ping that's carried out in the experiment in the initial experiment we ran in the tutorial.
    180 
    181 {{{
    182 library(RSQLite)
    183 con <- dbConnect(dbDriver("SQLite"), dbname = "gimi20-2012-10-18t14.03.42-04.00.sq3")
    184 dbListTables(con)
    185 dbReadTable(con,"pingmonitor_myping")
    186 
    187 mydata1 <- dbGetQuery(con, "select dest_addr, rtt from pingmonitor_myping where dest_addr='192.168.4.10'")
    188 rtt1 <- abs(mydata1$rtt)
    189 
    190 mydata2 <- dbGetQuery(con, "select dest_addr, rtt from pingmonitor_myping where dest_addr='192.168.5.12'")
    191 rtt2 <- abs(mydata2$rtt)
    192 
    193 mydata3 <- dbGetQuery(con, "select dest_addr, rtt from pingmonitor_myping where dest_addr='192.168.4.11'")
    194 rtt3 <- abs(mydata3$rtt)
    195 
    196 mydata4 <- dbGetQuery(con, "select dest_addr, rtt from pingmonitor_myping where dest_addr='192.168.2.12'")
    197 rtt4 <- abs(mydata4$rtt)
    198 
    199 mydata5 <- dbGetQuery(con, "select dest_addr, rtt from pingmonitor_myping where dest_addr='192.168.1.13'")
    200 rtt5 <- abs(mydata5$rtt)
    201 
    202 mydata6 <- dbGetQuery(con, "select dest_addr, rtt from pingmonitor_myping where dest_addr='192.168.5.11'")
    203 rtt6 <- abs(mydata6$rtt)
    204 
    205 mydata7 <- dbGetQuery(con, "select dest_addr, rtt from pingmonitor_myping where dest_addr='192.168.2.10'")
    206 rtt7 <- abs(mydata7$rtt)
    207 
    208 mydata8 <- dbGetQuery(con, "select dest_addr, rtt from pingmonitor_myping where dest_addr='192.168.3.13'")
    209 rtt8 <- abs(mydata8$rtt)
    210 
    211 mydata9 <- dbGetQuery(con, "select dest_addr, rtt from pingmonitor_myping where dest_addr='192.168.6.14'")
    212 rtt9 <- abs(mydata9$rtt)
    213 
    214 mydata10 <- dbGetQuery(con, "select dest_addr, rtt from pingmonitor_myping where dest_addr='192.168.1.10'")
    215 rtt10 <- abs(mydata10$rtt)
    216 
    217 mydata11 <- dbGetQuery(con, "select dest_addr, rtt from pingmonitor_myping where dest_addr='192.168.3.12'")
    218 rtt11 <- abs(mydata11$rtt)
    219 
    220 mydata12 <- dbGetQuery(con, "select dest_addr, rtt from pingmonitor_myping where dest_addr='192.168.6.12'")
    221 rtt12 <- abs(mydata12$rtt)
    222 
    223 png(filename="gimi20-nmetrics-eth", height=650, width=900,
    224  bg="white")
    225 g_range <- range(0,rtt1,rtt2,rtt3,rtt4,rtt5,rtt6,rtt7,rtt8,rtt9,rtt10,rtt11,rtt12)
    226 
    227 plot(rtt1,type="o",col="red",ylim= g_range, lty=2, xlab="Experiment Interval",ylab="RTT")
    228 lines(rtt2,type="o",col="blue",xlab="Experiment Interval",ylab="Received Data")
    229 lines(rtt3,type="o",col="green",xlab="Experiment Interval",ylab="Received Data")
    230 lines(rtt4,type="o",col="purple",xlab="Experiment Interval",ylab="Received Data")
    231 lines(rtt5,type="o",col="violetred",xlab="Experiment Interval",ylab="Received Data")
    232 lines(rtt6,type="o",col="springgreen",xlab="Experiment Interval",ylab="Received Data")
    233 lines(rtt7,type="o",col="skyblue",xlab="Experiment Interval",ylab="Received Data")
    234 lines(rtt8,type="o",col="sienna",xlab="Experiment Interval",ylab="Received Data")
    235 lines(rtt9,type="o",col="pink",xlab="Experiment Interval",ylab="Received Data")
    236 lines(rtt10,type="o",col="yellow",xlab="Experiment Interval",ylab="Received Data")
    237 lines(rtt11,type="o",col="thistle",xlab="Experiment Interval",ylab="Received Data")
    238 lines(rtt12,type="o",col="orange",xlab="Experiment Interval",ylab="Received Data")
    239 
    240 title(main="nmetrics experiment on ExoGENI (Received Data)", col.main="red", font.main=4)
    241 legend("topright", g_range[4], legend=c("192.168.4.10","192.168.5.12","192.168.4.11","192.168.2.12","192.168.5.11","192.168.2.10","192.168.3.13","192.168.6.14","192.168.1.10","192.168.3.12","192.168.6.12"), cex=0.8,
    242    col=c("blue","red","green","purple","violetred","springgreen","skyblue","sienna","pink","yellow","thistle","orange"), pch=15:16:17:18:19:20:21:22:23:24:25:26, lty=1:2:3:4:5:6:7:8:9:10:11:12);
    243 
    244 dev.off()
    245 }}}
    246 
    247 The resulting plot is shown below.
    248 
    249 [[Image(gimi20-ping.png)]]
    250 
    251 ----
    252 
    253 The following [http://emmy9.casa.umass.edu/GEC15-GIMI-Tutorial/R_script_otr.r R script] plots otr results from the
    254 4th experiment we executed in Section C.
    255 
    256 {{{
    257 
    258 library(RSQLite)
    259 con <- dbConnect(dbDriver("SQLite"), dbname = "gimi20-otg-nmetrics.sq3")
    260 dbListTables(con)
    261 dbReadTable(con,"otr2_udp_in")
    262 
    263 mydata1 <- dbGetQuery(con, "select oml_sender_id, pkt_length from otr2_udp_in where src_host='192.168.4.10'")
    264 pkt_length <- mydata1$pkt_length
    265 #plot(rx_bytes1, type="o", color="red", xlab="Experiment Interval", ylab="Received data")
    266 
    267 
    268 png(filename="gimi20_otg1.png", height=650, width=900,
    269  bg="white")
    270 g_range <- range(0,pkt_length)
    271 
    272 plot(pkt_length,type="o",col="red",ylim= g_range, lty=2, xlab="Experiment Interval",ylab="Packet Size")
    273 
    274 title(main="Received packet size with sender address 192.168.4.10", col.main="red", font.main=4)
    275 legend("bottomright", g_range[1], legend=c("interface1"), cex=0.8,
    276    col=c("blue"), pch=21, lty=1);
    277 
    278 dev.off()
    279 }}}
    280 
    281 
    282 The resulting plot is shown below.
    283 
    284 [[Image(gimi20_otg1.png)]]
    285 
    286 ----
    287 
    288 The following [http://emmy9.casa.umass.edu/GEC15-GIMI-Tutorial/R_script_nmetrics.r script] plots part of nmetrics results from the
    289 4th experiment we executed.
    290 
    291 {{{
    292 library(RSQLite)
    293 con <- dbConnect(dbDriver("SQLite"), dbname = "gimi20-otg-nmetrics.sq3")
    294 dbListTables(con)
    295 dbReadTable(con,"nmetrics_net_if")
    296 
    297 mydata1 <- dbGetQuery(con, "select oml_sender_id, rx_bytes from nmetrics_net_if where oml_sender_id=1")
    298 rx_bytes1 <- abs(mydata1$rx_bytes)
    299 #plot(rx_bytes1, type="o", color="red", xlab="Experiment Interval", ylab="Received data")
    300 
    301 mydata2 <- dbGetQuery(con, "select oml_sender_id, rx_bytes from nmetrics_net_if where oml_sender_id=2")
    302 rx_bytes2 <- abs(mydata2$rx_bytes)
    303 
    304 mydata3 <- dbGetQuery(con, "select oml_sender_id, rx_bytes from nmetrics_net_if where oml_sender_id=3")
    305 rx_bytes3 <- abs(mydata3$rx_bytes)
    306 
    307 mydata4 <- dbGetQuery(con, "select oml_sender_id, rx_bytes from nmetrics_net_if where oml_sender_id=4")
    308 rx_bytes4 <- abs(mydata4$rx_bytes)
    309 
    310 png(filename="gimi20-nmetrics-eth", height=650, width=900,
    311  bg="white")
    312 g_range <- range(0,rx_bytes1,rx_bytes2,rx_bytes3,rx_bytes4)
    313 
    314 plot(rx_bytes1,type="o",col="red",ylim= g_range, lty=2, xlab="Experiment Interval",ylab="Received Data")
    315 lines(rx_bytes2,type="o",col="blue",xlab="Experiment Interval",ylab="Received Data")
    316 lines(rx_bytes3,type="o",col="green",xlab="Experiment Interval",ylab="Received Data")
    317 lines(rx_bytes4,type="o",col="purple",xlab="Experiment Interval",ylab="Received Data")
    318 title(main="nmetrics experiment on ExoGENI (Received Data)", col.main="red", font.main=4)
    319 legend("bottomright", g_range[4], legend=c("interface1","interface2","interface3","interface4"), cex=0.8,
    320    col=c("blue","red","green","purple"), pch=21:22:23:24, lty=1:2:3:4);
    321 }}}
    322 
    323 
    324 The resulting plot is shown below.
    325 
    326 [[Image(gimi20-nmetrics-eth.png)]]
    327 
    328 All three scripts are simply executed with:
    329 
    330 {{{
    331 R -f <script_name>
    332 }}}
    333 
    334 The benefit of using R scripts is that they can produce graphs that can be used in documents!
    335 
    336 The results can then be stored into iRODS using the itools presented in Section D.2.
    337 
    338 
    339 
    340  
    341 [[BR]]
    342 [[BR]]
    3433
    3444