wiki:GENIExperimenter/Tutorials/jacks/HadoopInASlice/ObtainResources

Version 29 (modified by nriga@bbn.com, 9 years ago) (diff)

--

Hadoop in a Slice

We are actively updating this tutorial. If you need help with this tutorial please contact: help@geni.net

Part I: Obtain Resources: create a slice and reserve resources

Image Map

1. Establish the Environment

1.1 Pre-work: Ensure SSH keys are setup

Verify that you have at least one public key associated with your account. To do that, after you login to the portal check under your Profile, under the SSH keys tab. If you do not have SSH keys associated yet, please follow the instructions on that tab of the Portal.

1.2 Configure Omni

If you have not installed and configured omni yet, please follow these instructions.

2. Obtain Resources

2.1 Create a slice Slice

Create a slice using omni and the slice name of your choice. From now on that slice name will be referred to as SLICENAME.

$ omni createslice SLICENAME 

2.2 Create your RSpec

We are going to use the graphical interface of the portal (Jacks) to create the RSpec file for this tutorial but we are going to use Omni for reserving the resources.

2.2.1 Load a simple topology in Jacks

  1. In the Portal, open the Slice page for the slice you just created. Notice that you created the slice with omni and it is available via the Portal.
  2. Press the Add Resources button to launch Jacks for this slice.
  3. From the Choose RSpec menu (see figure), select the URL button.
  4. Enter the URL for the RSpec:
    http://geni-images.renci.org/images/tutorials/GENI-hadoop/hadoop_dynamic_request_rspec.xml
    then click Select.
  5. After you click Select, a couple of nodes will appear.

The topology you loaded has two node-types: master and worker. Let's see what each node-type is comprised of:

  1. Inspect the properties of the hadoop master. Note that not only the usual attributes (e.g. node name, node type) are set, but there are also custom OS Image, install and execute attributes. These attributes is what customizes a generic GENI node a specialized one like a hadoop master.

2.2.2 Create the topology for the experiment

Now we will create our Hadoop cluster. For the cluster we need:

  • 1 hadoop master
  • 2 or more workers
  • all the nodes need to be on the same Layer 2 Domain and IP Subnet (172.16.1.0/24)

So let's go ahead and draw our topology:

  1. Select the worker node.
  2. Press the "Duplicate Nodes only" button.
  3. Repeat the above two steps so that you have at least three workers but no more than six (please be respectful to the cluster size you create, especially if you are doing this in the context of a tutorial).
  4. Delete the original worker node (select it and on the attributes on the left click the Delete button).
    DO NOT press the delete button since this is equivalent to pressing the Back button on the browser.
  5. Ensure that all your workers are named in the pattern `worker-i` with i=0,1.2,... (e.g. `worker-0, worker-1, worker-2`)
    WARNING This step is important, if the workers are not named based on the above convention your cluster WILL NOT be configured correctly
  6. On the master node make sure that on the Execute script the input number matches the number of workers, e.g. if you have 3 workers the entry should be
    /home/hadoop/hadoop_config_dynamic.sh 3
  7. Connect all your nodes in one LAN
    1. Drag a line between two of your nodes
    2. From each unconnected node draw a line from that node to the middle of the link where there is a small square
  8. Assign IPs to your nodes:
    Click on the small square in the middle of your lan and set the IPs and netmask for each interface according to this list:
    • the netmask for all interfaces should be 255.255.255.0
    • master IP: 172.16.1.1
    • worker-i IP: 172.16.1.<10+i>, e.g. worker-0: 172.16.1.10, worker-1: 172.16.1.11, worker-5: 172.16.1.15, etc

Now based on the loaded node types we will create our Hadoop cluster. For the cluster we need

  • 1 hadoop master
  • 2 or more workers
  • all the nodes need to be on the same Layer 2 Domain

Let's design our topology.

  1. Select the worker node
  2. Press the Add Resources button to launch Jacks for this slice.
  3. From the Choose RSpec menu (see figure), select the URL button.
  4. Enter the URL for the RSpec:
    http://geni-images.renci.org/images/tutorials/GENI-hadoop/hadoop_dynamic_request_rspec.xml
    then click Select.
  5. After you click Select, a couple of nodes will appear.
{{{ #!html
  1. Click "Site 0" in the Jacks window
  2. Select an ExoGENI site (If you are participating in an organized tutorial, please bind the VMs to the rack assigned to you)
Bind the Slice
}}} == 3. Create the Hadoop Master == {{{ #!html
  1. Add an ExoGENI VM to your slice.
  2. Add the Hadoop Master
  1. Select the VM to set its properties:
    1. Name: hadoop-master
    2. Node Type: ExoGENI Medium
    3. Set the Master Name
    4. Custom Disk Image Name: http://geni-images.renci.org/images/standard/hadoop/hadoop_debian6.v1.0/hadoop_debian6.v1.0.xml
    5. Disk Image Version: 54ddc5ed7d4a22331651d7b78dc5d39138f8a165
    6. Set the Master Image
    7. Add an Install Script:
      URL: http://geni-images.renci.org/images/tutorials/GENI-hadoop/master.sh
      Path: /tmp
    8. Set the Master Install Script
    9. Add an execute service to execute the script at boot time:
      chmod +x /tmp/master.sh; /tmp/master.sh
    10. Set the Master Execute Scripts
}}} == 3. Create the Hadoop Workers == {{{ #!html
  1. Add a 2 more VMs to the same rack as the first VM.
Set the Master Execute Scripts
  1. Edit each worker’s attributes
    1. Names: hadoop-worker-0 and hadoop-worker-1
    2. Node Type: ExoGENI Medium
    3. Set the Master Execute Scripts
    4. Disk Image Name: http://geni-images.renci.org/images/standard/hadoop/hadoop_debian6.v1.0/hadoop_debian6.v1.0.xml
    5. Disk Image Version: 54ddc5ed7d4a22331651d7b78dc5d39138f8a165
    6. Set the Master Execute Scripts
    7. Add an Install Script:
      URL: http://geni-images.renci.org/images/tutorials/GENI-hadoop/worker.sh
      Path: /tmp
    8. Set the Master Execute Scripts
    9. Add an execute service to execute the script at boot time. For each VM, substitute the VM’s name for where the following uses “hadoop-worker-0”. (Note: the following should be placed on one line)
      chmod +x /tmp/worker.sh; /tmp/worker.sh $hadoop-master.Name() $hadoop-master.IP("link-0") $hadoop-worker-0.Name() $hadoop-worker-0.IP("link-0")
    10. Set the Master Execute Scripts
    11. Repeat for worker 1
    12. Set the Master Execute Scripts
}}} == 4. Create the Network == {{{ #!html
  1. Link all three VMs with a broadcast network
  2. Set the Master Execute Scripts
    1. Set the master's IP to 172.16.1.1
    2. Set the master's netmask to 255.255.255.0
    3. Set the Master Execute Scripts
    4. Set the worker0's IP to 172.16.1.10
    5. Set the worker0's netmask to 255.255.255.0
    6. Set the worker1's IP to172.16.1.11
    7. Set the worker1's netmask to 255.255.255.0
}}} == 5. Instantiate the Slice == You can use any tool to reserve this topology, today we are going to use Omni. To do that you will need to: {{{ #!html
  1. Download the rspec and save it to your local disk by clicking the download button
  2. Submit the request by running this omni command in a command line window:
    omni createsliver slicename rspec_filename -a AM_NICKNAME
  3. Wait until the slice is up
  4. Log into the master node
}}} ---- = [.. Introduction] = = [../ExecuteExperiment Next: Execute the Hadoop Experiment] =

Attachments (24)