Changes between Version 8 and Version 9 of GENIExperimenter/Tutorials/HadoopInASlice/ExecuteExperiment
- Timestamp:
- 01/07/14 21:26:58 (10 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
GENIExperimenter/Tutorials/HadoopInASlice/ExecuteExperiment
v8 v9 23 23 24 24 == 1. Login to Hadoop Master == 25 25 26 {{{ 26 27 #!html … … 42 43 == 2. Check the status/properties of the VMs. == 43 44 44 {{{ 45 #!html 46 47 48 <table border="0"> 49 <tr> 50 51 <td > 52 <ol type="A"> 53 54 <li> Observe the properties of the network interfaces </li> 55 56 57 58 <pre><code> 45 === A. Observe the properties of the network interfaces === 46 47 {{{ 59 48 # /sbin/ifconfig 60 49 eth0 Link encap:Ethernet HWaddr fa:16:3e:72:ad:a6 … … 85 74 collisions:0 txqueuelen:0 86 75 RX bytes:4010954 (3.8 MiB) TX bytes:4010954 (3.8 MiB) 87 </code></pre>76 }}} 88 77 89 78 90 91 92 <li> Observe the contents of the NEuca user data file. This file includes a script that will install and execute the script that you configured for the VM. </li> 93 <pre><code> 79 === Observe the contents of the NEuca user data file. This file includes a script that will install and execute the script that you configured for the VM. === 80 {{{ 94 81 # neuca-user-data 95 82 [global] … … 120 107 fi 121 108 eval "/bin/sh -c \"chmod +x /tmp/master.sh; /tmp/master.sh\"" 122 </code></pre> 123 124 125 <li> Observe the contents of the of the script that was installed and executed on the VM. </li> 126 <pre><code> 109 }}} 110 111 112 === Observe the contents of the of the script that was installed and executed on the VM. === 113 {{{ 127 114 # cat /tmp/master.sh 128 115 #!/bin/bash … … 158 145 /home/hadoop/hadoop-euca-init.sh 172.16.1.1 -master 159 146 echo "Done starting daemons" >> /home/hadoop/log 160 </code></pre> 161 162 163 <li>Test for connectivity between the VMs.</li> 164 <pre><code> 147 }}} 148 149 150 === Test for connectivity between the VMs. === 151 152 {{{ 165 153 # ping hadoop-worker-0 166 154 PING hadoop-worker-0 (172.16.1.10) 56(84) bytes of data. … … 181 169 3 packets transmitted, 3 received, 0% packet loss, time 1999ms 182 170 rtt min/avg/max/mdev = 0.468/0.607/0.852/0.174 ms 183 </code></pre>184 </op>185 186 </td></tr>187 </table>188 171 }}} 189 172 190 173 == 3. Check the status of the Hadoop filesystem. == 191 174 192 {{{ 193 #!html 194 195 196 <table border="0"> 197 <tr> 198 199 <td > 200 <ol type="A"> 201 202 203 <li> Query for the status of the filesystem and its associated workers. </li> 204 <pre><code> 175 === Query for the status of the filesystem and its associated workers. === 176 177 {{{ 205 178 # hadoop dfsadmin -report 206 179 Configured Capacity: 54958481408 (51.18 GB) … … 238 211 DFS Remaining%: 88.58% 239 212 Last contact: Sat Jan 04 21:49:33 UTC 2014 240 </code></pre> 241 242 243 244 <li> Test the filesystem with a small file </li> 245 246 <ol type="a"> 247 <li> Create a small test file </li> 248 <pre><code> 213 }}} 214 215 216 217 == Test the filesystem with a small file == 218 219 220 === Create a small test file === 221 {{{ 249 222 # hadoop fs -put hello.txt hello.txt 250 </code></pre> 251 252 <li> Push the file into the Hadoop filesystem</li> 253 <pre><code> 223 }}} 224 225 === Push the file into the Hadoop filesystem === 226 {{{ 254 227 # hadoop fs -put hello.txt hello.txt 255 </code></pre> 256 257 <li> Check for the file's existence </li> 258 <pre><code> 228 }}} 229 230 === Check for the file's existence === 231 {{{ 259 232 # hadoop fs -ls 260 233 Found 1 items 261 234 -rw-r--r-- 3 root supergroup 12 2014-01-04 21:59 /user/root/hello.txt 262 </code></pre> 263 264 <li> Check the contents of the file </li> 265 <pre><code> 235 }}} 236 237 === Check the contents of the file === 238 {{{ 266 239 # hadoop fs -cat hello.txt 267 240 Hello GENI World 268 </code></pre>269 270 </ol>271 </op>272 273 </td></tr>274 </table>275 241 }}} 276 242 … … 279 245 Test the true power of the Hadoop filesystem by creating and sorting a large random dataset. It may be useful/interesting to login to the master and/or worker VMs and use tools like \verb$top$, \verb$iotop$, and \verb$iftop$ to observe the resource utilization on each of the VMs during the sort test. 280 246 281 {{{ 282 #!html 283 284 285 <table border="0"> 286 <tr> 287 288 <td > 289 <ol type="A"> 290 <li> Create a 1 GB random data set. After the data is created, use the \verb$ls$ functionally to confirm the data exists. Note that the data is composed of several files in a directory. </li> 291 <pre><code> 247 == Create a 1 GB random data set. After the data is created, use the \verb$ls$ functionally to confirm the data exists. Note that the data is composed of several files in a directory. == 248 {{{ 292 249 # hadoop jar /usr/local/hadoop-0.20.2/hadoop-0.20.2-examples.jar teragen 10000000 random.data.1G 293 250 Generating 10000000 using 2 maps with step of 5000000 … … 309 266 14/01/05 18:48:28 INFO mapred.JobClient: Map input bytes=10000000 310 267 14/01/05 18:48:28 INFO mapred.JobClient: Map output records=10000000 311 </code></pre> 312 313 <li> Sort the datasets. On your own, you can use the \verb$cat$ and/or \verb$get$ functionally to look at the random and sorted files to confirm their size and that the sort actually worked. 314 </li> 315 <pre><code> 268 }}} 269 270 == Sort the datasets. On your own, you can use the \verb$cat$ and/or \verb$get$ functionally to look at the random and sorted files to confirm their size and that the sort actually worked. == 271 272 {{{ 316 273 # hadoop jar /usr/local/hadoop-0.20.2/hadoop-0.20.2-examples.jar terasort random.data.1G sorted.data.1G 317 274 14/01/05 18:50:49 INFO terasort.TeraSort: starting … … 372 329 14/01/05 18:52:48 INFO mapred.JobClient: Reduce input records=10000000 373 330 14/01/05 18:52:48 INFO terasort.TeraSort: done 374 </code></pre> 375 376 </ol> 377 378 </td></tr> 379 </table> 380 }}} 331 }}} 332 381 333 382 334 == 5. Advanced Example == … … 384 336 Re-do the tutorial with a different number of workers, amount of bandwidth, and/or worker instance types. Warning: Be courteous to other users and do not take all the resources. 385 337 386 {{{ 387 #!html 388 389 390 <table border="0"> 391 <tr> 392 393 <td > 394 <ol type="A"> 395 396 397 <li> Time the performance of runs with different resources </li> 398 <li> Observe largest size file you can create with different settings. </li> 399 </ol> 400 401 402 </ol> 403 404 405 406 </ol> 407 </td> 408 </tr> 409 410 411 </table> 412 }}} 413 338 A. Time the performance of runs with different resources 339 B. Observe largest size file you can create with different settings. 414 340 415 341