Changes between Version 5 and Version 6 of GENIExperimenter/Tutorials/jacks/HadoopInASlice/ExecuteExperiment

Timestamp:

09/17/15 08:13:15 (9 years ago)

Author:

nriga@bbn.com

Comment:

--

GENIExperimenter/Tutorials/jacks/HadoopInASlice/ExecuteExperiment

@@ -153 +153 @@
        start-yarn.sh
 }}}
-
-== 3. Check the status of the Hadoop filesystem. ==
-
-===  A. Query for the status of the filesystem and its associated workers. ===
-
-{{{
-# hadoop dfsadmin -report
-Configured Capacity: 54958481408 (51.18 GB)
-Present Capacity: 48681934878 (45.34 GB)
-DFS Remaining: 48681885696 (45.34 GB)
-DFS Used: 49182 (48.03 KB)
-DFS Used%: 0%
-Under replicated blocks: 1
+ a. Check the status of the Hadoop filesystem. ==
+ {{{
+# hdfs dfsadmin -report
+Configured Capacity: 54824083456 (51.06 GB)
+Present Capacity: 48522035200 (45.19 GB)
+DFS Remaining: 48521986048 (45.19 GB)
+DFS Used: 49152 (48 KB)
+DFS Used%: 0.00%
+Under replicated blocks: 0
 Blocks with corrupt replicas: 0
 Missing blocks: 0
+Missing blocks (with replication factor 1): 0
 
 -------------------------------------------------
-Datanodes available: 2 (2 total, 0 dead)
-
-Name: 172.16.1.11:50010
-Rack: /default/rack0
+Live datanodes (2):
+
+Name: 172.16.1.10:50010 (worker-0)
+Hostname: worker-0
 Decommission Status : Normal
-Configured Capacity: 27479240704 (25.59 GB)
-DFS Used: 24591 (24.01 KB)
-Non DFS Used: 3137957873 (2.92 GB)
-DFS Remaining: 24341258240(22.67 GB)
-DFS Used%: 0%
-DFS Remaining%: 88.58%
-Last contact: Sat Jan 04 21:49:32 UTC 2014
-
-
-Name: 172.16.1.10:50010
-Rack: /default/rack0
+Configured Capacity: 27412041728 (25.53 GB)
+DFS Used: 24576 (24 KB)
+Non DFS Used: 3151020032 (2.93 GB)
+DFS Remaining: 24260997120 (22.59 GB)
+DFS Used%: 0.00%
+DFS Remaining%: 88.50%
+Configured Cache Capacity: 0 (0 B)
+Cache Used: 0 (0 B)
+Cache Remaining: 0 (0 B)
+Cache Used%: 100.00%
+Cache Remaining%: 0.00%
+Xceivers: 1
+Last contact: Thu Sep 17 12:04:32 UTC 2015
+
+
+Name: 172.16.1.11:50010 (worker-1)
+Hostname: worker-1
 Decommission Status : Normal
-Configured Capacity: 27479240704 (25.59 GB)
-DFS Used: 24591 (24.01 KB)
-Non DFS Used: 3138588657 (2.92 GB)
-DFS Remaining: 24340627456(22.67 GB)
-DFS Used%: 0%
-DFS Remaining%: 88.58%
-Last contact: Sat Jan 04 21:49:33 UTC 2014
-}}}
-
-
-
-== 4. Test the filesystem with a small file ==
-
-
-=== A. Create a small test file ===
-{{{
-# echo Hello GENI World > hello.txt
-}}}
-
-=== B. Push the file into the Hadoop filesystem ===
-{{{
-# hadoop fs -put hello.txt hello.txt
-}}}
-
-=== C. Check for the file's existence ===
-{{{
-# hadoop fs -ls
+Configured Capacity: 27412041728 (25.53 GB)
+DFS Used: 24576 (24 KB)
+Non DFS Used: 3151028224 (2.93 GB)
+DFS Remaining: 24260988928 (22.59 GB)
+DFS Used%: 0.00%
+DFS Remaining%: 88.50%
+Configured Cache Capacity: 0 (0 B)
+Cache Used: 0 (0 B)
+Cache Remaining: 0 (0 B)
+Cache Used%: 100.00%
+Cache Remaining%: 0.00%
+Xceivers: 1
+Last contact: Thu Sep 17 12:04:32 UTC 2015
+}}}
+
+
+== 2. Run the experiment ==
+
+=== 2.1 Test the hadoop cluster with a small file ===
+ a. Create a small test file
+ {{{
+# echo Hello GENI World > /tmp/hello.txt
+}}}
+ a. Push the file into the Hadoop filesystem ===
+ {{{
+# hdfs dfs -put  /tmp/hello.txt /hello.txt
+}}}
+ a. Check for the file's existence ===
+ {{{
+# hdfs dfs -ls /
 Found 1 items
--rw-r--r--   3 root supergroup         12 2014-01-04 21:59 /user/root/hello.txt
-}}}
-
-===  D. Check the contents of the file ===
-{{{
-# hadoop fs -cat hello.txt
+-rw-r--r--   2 hadoop supergroup         17 2015-09-17 12:09 /hello.txt
+}}}
+ a. Check the contents of the file ===
+ {{{
+# hdfs dfs -cat /hello.txt
 Hello GENI World
 }}}
 
-== 4.   Run the Hadoop Sort Testcase ==
+=== 2.2 Run the Hadoop Sort Testcase ===
 
  Test the true power of the Hadoop filesystem by creating and sorting a large random dataset.   It may be useful/interesting to login to the master and/or worker VMs and use tools like top, iotop, and iftop to observe the resource utilization on each of the VMs during the sort test.  Note: on these VMs iotop and iftop must be run as root.
 
-===  A. Create a 1 GB random data set.   ===
-
-After the data is created, use the ls functionally to confirm the data exists.  Note that the data is composed of several files in a directory.
-
-{{{
-#  hadoop jar /usr/local/hadoop-0.20.2/hadoop-0.20.2-examples.jar teragen 10000000 random.data.1G
-Generating 10000000 using 2 maps with step of 5000000
+ a. Create a 1 GB random data set
+ {{{
+#  hadoop jar /home/hadoop/hadoop-2.7.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar teragen 10000000 /input
 14/01/05 18:47:58 INFO mapred.JobClient: Running job: job_201401051828_0003
 14/01/05 18:47:59 INFO mapred.JobClient:  map 0% reduce 0%
@@ -253 +254 @@
 14/01/05 18:48:28 INFO mapred.JobClient:     Map output records=10000000
 }}}
-
-=== B. Sort the dataset. ===  
-
-{{{
-# hadoop jar /usr/local/hadoop-0.20.2/hadoop-0.20.2-examples.jar terasort random.data.1G sorted.data.1G
+ After the data is created, use the ls functionally to confirm the data exists.  Note that the data is composed of several files in a directory.
+ a. Sort the dataset: 
+ {{{
+# hadoop jar /home/hadoop/hadoop-2.7.1/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.1.jar terasort /input /output
 14/01/05 18:50:49 INFO terasort.TeraSort: starting
 14/01/05 18:50:49 INFO mapred.FileInputFormat: Total input paths to process : 2
@@ -316 +316 @@
 14/01/05 18:52:48 INFO terasort.TeraSort: done
 }}}
-
-==== C. Look at the output. ====  
-
-You can use Hadoop's cat and/or get functionally to look at the random and sorted files to confirm their size and that the sort actually worked.
-
-Try some or all of these commands.  Does the output make sense to you?
-
-{{{
+ a. Look at the output: You can use Hadoop's cat and/or get functionally to look at the random and sorted files to confirm their size and that the sort actually worked.
+ Try some or all of these commands.  Does the output make sense to you?
+ {{{
 hadoop fs -ls random.data.1G
 hadoop fs -ls sorted.data.1G
@@ -330 +325 @@
 }}}
 
-== 5.   Advanced Example ==
+== 3.   Advanced Example ==
 
  Re-do the tutorial with a different number of workers, amount of bandwidth, and/or worker  instance types.  Warning:  be courteous to  other users and do not use too many of the resources.