Changes between Version 5 and Version 6 of GENIExperimenter/Tutorials/GettingStarted_PartII_Hadoop/Procedure/Execute
- Timestamp:
- 03/07/14 15:07:28 (10 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
GENIExperimenter/Tutorials/GettingStarted_PartII_Hadoop/Procedure/Execute
v5 v6 230 230 231 231 232 == 4. Test the filesystem with a small file == 233 234 235 === A. Create a small test file === 232 233 === 5.3 Test the filesystem with a small file === 234 235 236 ==== 5.3.1 Create a small test file ==== 236 237 {{{ 237 238 # echo Hello GENI World > hello.txt 238 239 }}} 239 240 240 === B.Push the file into the Hadoop filesystem ===241 ==== 5.3.2 Push the file into the Hadoop filesystem === 241 242 {{{ 242 243 # hadoop fs -put hello.txt hello.txt 243 244 }}} 244 245 245 === C.Check for the file's existence ===246 ==== 5.3.3 Check for the file's existence === 246 247 {{{ 247 248 # hadoop fs -ls … … 250 251 }}} 251 252 252 === D.Check the contents of the file ===253 ==== 5.3.4 Check the contents of the file === 253 254 {{{ 254 255 # hadoop fs -cat hello.txt … … 256 257 }}} 257 258 258 == 4. Run the Hadoop Sort Testcase==259 === 5.4 Run the Hadoop Sort Testcase === 259 260 260 261 Test the true power of the Hadoop filesystem by creating and sorting a large random dataset. It may be useful/interesting to login to the master and/or worker VMs and use tools like top, iotop, and iftop to observe the resource utilization on each of the VMs during the sort test. Note: on these VMs iotop and iftop must be run as root. 261 262 262 === A. Create a 1 GB random data set.===263 ==== 5.4.1 Create a 1 GB random data set. ==== 263 264 264 265 After the data is created, use the ls functionally to confirm the data exists. Note that the data is composed of several files in a directory. … … 286 287 }}} 287 288 288 === B. Sort the dataset.===289 ==== 5.4.2 Sort the dataset. ==== 289 290 290 291 Note: you can use Hadoop's cat and/or get functionally to look at the random and sorted files to confirm their size and that the sort actually worked. … … 351 352 }}} 352 353 353 354 == 5. Advanced Example == 354 === 5.5 Advanced Example === 355 355 356 356 Re-do the tutorial with a different number of workers, amount of bandwidth, and/or worker instance types. Warning: be courteous to other users and do not use too many of the resources. 357 357 358 === A. Time the performance of runs with different resources.===359 === B. Observe largest size file you can create with different resources.===358 ==== 5.5.1 Time the performance of runs with different resources. ==== 359 ==== 5.5.2 Observe largest size file you can create with different resources. ==== 360 360 361 361