wiki:GENIExperimenter/Tutorials/GettingStarted_PartII_Hadoop/Procedure/DesignSetup

Version 33 (modified by sedwards@bbn.com, 10 years ago) (diff)

--

Understanding the AM API using a simple Hadoop experiment

Image Map

1. Design the Experiment

  1. In today's experiment you will use resources at the aggregate listed on the worksheet. If you don't have a worksheet use RENCI ExoGENI (aka exogeni.net.rcivmsite)

2. Establish the Environment

2.1 Pre-work: Ensure SSH keys are setup

Verify that you have at least one public key associated with your account. To do that, after you login to the portal check under your Profile, under the SSH keys tab. If you do not have SSH keys associated yet, please follow the instructions on that tab of the Portal.

2.2 Configure Omni

  1. Login to the GENI Portal
  2. Click on the `Profile tab` link on the top of your screen. Then click on the `Configure omni` tab under `PROFILE`.

Figure 2-1 Click on the Configure omni tab under Profile.
  1. Click on the `Download your omni data` button under step 2.

Figure 2-2 Download your omni data under step 2.
  1. If this is the first time you try to access your GENI certificate you will have to generate one. Click on the `generate a certificate` link.

Figure 2-3 Click on Generate a certificate.
  1. Unless you really understand how SSL certificates work, choose the simple option. Click on the `Generate Combined Certificate and Key File` button and then click on `Close`. You will be taken back to the download page with the warning. Reload the page to enable the download button.

Figure 2-4 Click on Generate Combined Certificate and Key File.
  1. If you are a member of more than one project, select which project you would like to be the default one for running experiments in GENI. You can always change the project that is used by the `-r` command line option of Omni. Then click on `Download your omni data`.

Figure 2-5 Click on Download your omni data.
  1. The bundle will be saved at ~/Downloads/omni-bundle.zip
  2. Open a terminal window and type:
    omni-configure
    The cert and key files you need will be installed in the appropriate folders.

3. Obtain Resources

3.1 Create a slice

Create a slice using omni and the slice name of your choice. From now on that slice name will be referred to as SLICENAME.

$ omni createslice SLICENAME

3.2. Load a simple topology in Flack

For this exercise, we will edit an existing RSpec file. Start by loading this predefined topology into Flack.

  1. In the Portal, launch Flack for this slice.
  2. From the Import menu (see figure), select the Import from the web item.
Import an RSpec into Flack
Figure 3-2 Import an RSpec into Flack.
  1. Enter the URL for the RSpec we are going to edit, then click Ok.
  2. When Flack pops up a message asking you to pick a manager to use click OK and then select the manager listed on your worksheet from the list (otherwise select exogeni.net.rcivmsite). After you click Accept, a network topology with three nodes should appear on the canvas.
    Note Because this RSpec file contains a generic description of resources and does not specify any particular aggregate manager, Flack will pop up a message asking you to specify an aggregate to use:
    There were resources detected without a manager selected, please select which manager you would like to use.

3.3. Modify the RSpec to automatically install and execute network test software

For this experiment, we'd like to run a simple experiment with Hadoop using one Hadoop master and two Hadoop worker nodes.

  1. Click on the the i button on each node (see figure) to manipulate details of the node configuration.
    Note When you open the node information, you should see options like those in the figure. The controls available to you will vary depending on the types of the aggregate manager and the component, but many features are widely available throughout GENI.

Figure 3-3 Edit the nodes
  1. The way we will request installation of the proper software and execution of our experiment is to specify a custom image and then add install and execute services to run on that image.
  2. For each of the three nodes, fill in the following fields using the values specified in Tables 3-1, 3-2 and 3-3:
    1. specify a "Disk Image Name" and "Disk Image Version",
    2. click "+ Add Install Service" and fill in the location to download from and the location on the node to install it,
    3. click "+ Add Execute Service" button to request that the appropriate command be executed at boot time.
    4. Complete the various fields using the appropriate information from the tables below.
    5. When you are done, please choose Apply.
    Be very careful when entering this information -- these commands will not be executed yet, so it will be some time before you will see any relevant error messages if there is a mistake here. To reduce errors, copy and paste the fields directly from the online instructions.

Figure 3-4 Add the install and execute services
Field Value
Name hadoop-master
Sliver Type XOLarge
Disk Image Name http://geni-images.renci.org/images/GENIWinterCamp/images/gwc-hadoop.v0.4a.xml
Disk Image Version 16ff128df4cf10f2472a8d20796146bcd5a5ddc3
Install Archive URL http://geni-images.renci.org/images/GENIWinterCamp/master.sh
Install Location /tmp
Execute Service chmod +x /tmp/master.sh; /tmp/master.sh
Execute Service Shell sh

Table 3-1 The details for the hadoop-master node.
Configure the hadoop-master node

Figure 3-6 Configure the details on the hadoop-master node.
Field Value
Name hadoop-worker-0
Sliver Type XOMedium
Disk Image Name http://geni-images.renci.org/images/GENIWinterCamp/images/gwc-hadoop.v0.4a.xml
Disk Image Version 16ff128df4cf10f2472a8d20796146bcd5a5ddc3
Install Archive URL http://geni-images.renci.org/images/GENIWinterCamp/worker.sh
Install Location /tmp
Execute Service chmod +x /tmp/worker.sh; /tmp/worker.sh $hadoop-master.Name() $hadoop-master.IP("link0") $hadoop-worker-0.Name() $hadoop-worker-0.IP("link0")
Execute Service Shell sh

Table 3-2 The details for the hadoop-worker-0 node.

Field Value
Name hadoop-worker-1
Sliver Type XOMedium
Disk Image Name http://geni-images.renci.org/images/GENIWinterCamp/images/gwc-hadoop.v0.4a.xml
Disk Image Version 16ff128df4cf10f2472a8d20796146bcd5a5ddc3
Install Archive URL http://geni-images.renci.org/images/GENIWinterCamp/worker.sh
Install Location /tmp
Execute Service chmod +x /tmp/worker.sh; /tmp/worker.sh $hadoop-master.Name() $hadoop-master.IP("link0") $hadoop-worker-1.Name() $hadoop-worker-1.IP("link0")
Execute Service Shell sh

Table 3-3 The details for the hadoop-worker-1 node.
Configure the hadoop-worker-0 node

Figure 3-7 Configure the details on the hadoop-worker-0 node.
  1. Now set the properties of the links in your topology: IP addresses, net masks and link capacity. To do this click on the i button on link0. Then go to the Interfaces tab and set the IP addresses and masks as specified in Table 3-4 below.
  2. Switch to the Properties tab and set the link’s capacity to 100Mbps by entering 100000000 in the box labeled Capacity. (Note: Flack will report that this is 100Gb/s but it is actually 100Mb/s.)

Figure 3-5 Configure the link bandwidth
Node IP Mask
hadoop-master 172.16.1.1 255.255.255.0
hadoop-worker-0 172.16.1.10 255.255.255.0
hadoop-worker-1 172.16.1.11 255.255.255.0
Property Value
Capacity100000000

Table 3-4 The details for link0.
Configure link0

Figure 3-8 Configure link0.

If you have time, it would be instructive to download the worker.sh and master.sh files yourself, and inspect the contents. This is a good example of how you can ask for GENI components to perform tasks for you without any manual intervention. This is a very useful facility for large experiments!

3.4. Export the modified request RSpec

Now we will pull back some of the covers and inspect exactly what Flack has been doing for us when preparing the RSpecs for the experiments we design. Each node and link has a corresponding element in the RSpec, and the details of the component configuration (such as the install and execute services we requested above) are specified with attributes, or sometimes child elements, within those portions of the document.

  1. From the View menu (see figure), select the Preview request document(s) item. This will bring up a window showing the current RSpec -- please take a moment to inspect it. The XML elements under the "flack" namespace were added for Flack's internal use (containing information about the canvas layout, editing history, etc.), and are ignored by aggregate managers. The `<node>` and `<link>` elements contain the specification for the components we will request, and if you look closely, you will be able to see the install and execute service elements you added earlier.
  2. Use the Save to file button (in the upper left) to a local copy of your RSpec with the name lab1_rspec.xml. We'll use this in the next step to demonstrate how other client tools also use RSpec files to communicate requests to aggregate managers.

Figure 3-6 View and save the final request RSpec

3.5. Instantiate the new experiment using Omni

For this step, we'll change the approach a bit and switch to a new client tool, the command line Omni client.

From a terminal, please enter the command:

$ omni -a AM_NICKNAME createsliver SLICENAME RSPEC_FILE

where AM_NICKNAME is the nickname for your assigned aggregate manager and SLICENAME is the name of the slice you created earlier (both of these are given on your worksheet). RSPEC_FILE should be replaced with the filename of the RSpec you saved in step 4.

If all is well, Omni should give you a number of informational messages, such as:

INFO:omni:Loading config file /home/geni/.gcf/omni_config

It should quickly proceed to the point where it makes the request to the remote manager:

INFO:omni:Creating sliver(s) from rspec file /home/geni/Downloads/experiments.rspec for slice ...

This step can sometimes be time-consuming, so please be patient. If it succeeds, within a couple of minutes Omni should report:

INFO:omni: Completed createsliver:

and your resource reservation is complete!


Introduction

Next: Execute