Version 29 (modified by 9 years ago) (diff) | ,
---|
Hadoop in a Slice
We are actively updating this tutorial. If you need help with this tutorial please contact: help@geni.net
Part I: Obtain Resources: create a slice and reserve resources
1. Establish the Environment
1.1 Pre-work: Ensure SSH keys are setup
Verify that you have at least one public key associated with your account. To do that, after you login to the portal check under your Profile, under the SSH keys
tab. If you do not have SSH keys associated yet, please follow the instructions on that tab of the Portal.
1.2 Configure Omni
If you have not installed and configured omni yet, please follow these instructions.
2. Obtain Resources
2.1 Create a slice Slice
Create a slice using omni
and the slice name of your choice. From now on that slice name will be referred to as SLICENAME
.
$ omni createslice SLICENAME
2.2 Create your RSpec
We are going to use the graphical interface of the portal (Jacks) to create the RSpec file for this tutorial but we are going to use Omni for reserving the resources.
2.2.1 Load a simple topology in Jacks
|
The topology you loaded has two node-types: master
and worker
. Let's see what each node-type is comprised of:
|
2.2.2 Create the topology for the experiment
Now we will create our Hadoop cluster. For the cluster we need:
- 1 hadoop master
- 2 or more workers
- all the nodes need to be on the same Layer 2 Domain and IP Subnet (172.16.1.0/24)
So let's go ahead and draw our topology:
|
Now based on the loaded node types we will create our Hadoop cluster. For the cluster we need
- 1 hadoop master
- 2 or more workers
- all the nodes need to be on the same Layer 2 Domain
Let's design our topology.
|
|
|
|
|
|
- Edit each worker’s attributes
- Names: hadoop-worker-0 and hadoop-worker-1
- Node Type: ExoGENI Medium
- Disk Image Name: http://geni-images.renci.org/images/standard/hadoop/hadoop_debian6.v1.0/hadoop_debian6.v1.0.xml
- Disk Image Version: 54ddc5ed7d4a22331651d7b78dc5d39138f8a165
- Add an Install Script:
URL:http://geni-images.renci.org/images/tutorials/GENI-hadoop/worker.sh
Path:/tmp
- Add an execute service to execute the script at boot time. For each VM, substitute
the VM’s name for where the following uses “hadoop-worker-0”. (Note: the following
should be placed on one line)
chmod +x /tmp/worker.sh; /tmp/worker.sh $hadoop-master.Name() $hadoop-master.IP("link-0") $hadoop-worker-0.Name() $hadoop-worker-0.IP("link-0")
- Repeat for worker 1
|
|
Attachments (24)
-
add-master.png (228.0 KB) - added by 10 years ago.
HadoopInASlice - add-master.png
-
edit-master.png (216.1 KB) - added by 10 years ago.
HadoopInASlice - edit-master.png
-
add-workers.png (243.2 KB) - added by 10 years ago.
HadoopInASlice - add-workers.png
-
edit-workers.png (219.9 KB) - added by 10 years ago.
HadoopInASlice - edit-workers.png
-
add-link.png (266.8 KB) - added by 10 years ago.
HadoopInASlice - add-link.png
-
edit-link-interfaces.png (193.5 KB) - added by 10 years ago.
HadoopInASlice - edit-link-interfaces.png
-
edit-link-properties.png (201.2 KB) - added by 10 years ago.
HadoopInASlice - edit-link-properties.png
-
slice-up.png (240.5 KB) - added by 10 years ago.
HadoopInASlice - slice-up.png
-
Bind-Slice.png (59.4 KB) - added by 9 years ago.
Bind-Slice
- Add-Master.png (49.2 KB) - added by 9 years ago.
- Set-Master-Name.png (57.2 KB) - added by 9 years ago.
- Set-Master-Disk-Image.png (63.4 KB) - added by 9 years ago.
- Set-Master-Install-Script.png (54.2 KB) - added by 9 years ago.
- Set-Master-Execute-Script.png (53.6 KB) - added by 9 years ago.
- Add-Workers.png (51.8 KB) - added by 9 years ago.
- Set-Worker-Name.png (65.5 KB) - added by 9 years ago.
- Set-Worker-Disk-Image.png (62.7 KB) - added by 9 years ago.
- Set-Worker-Install-Script.png (57.5 KB) - added by 9 years ago.
- Set-Worker-Execute-Script.png (56.0 KB) - added by 9 years ago.
- Setup-Worker2.png (53.2 KB) - added by 9 years ago.
- Add-Link.png (54.4 KB) - added by 9 years ago.
- Set-Master-IP.png (60.5 KB) - added by 9 years ago.
- Set-Worker0-IP.png (60.5 KB) - added by 9 years ago.
- Set-Worker1-IP.png (56.3 KB) - added by 9 years ago.