Changes between Version 3 and Version 4 of GENIExperimenter/Tutorials/jacks/HadoopInASlice/ExecuteExperiment


Ignore:
Timestamp:
09/17/15 07:52:26 (9 years ago)
Author:
nriga@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • GENIExperimenter/Tutorials/jacks/HadoopInASlice/ExecuteExperiment

    v3 v4  
     1[[PageOutline]]
    12= [.. Hadoop in a Slice] =
    2 == Part II: Execute Experiment: Login to the nodes and execute the Hadoop experiment ==
    33{{{
    44#!html
     
    1818</div>
    1919}}}
    20 = Instructions =
    2120
    2221Now that you have reserved your resources, you are ready to login to the slice and run some Hadoop examples.
    2322
    24 == 1. Login to Hadoop Master ==
     23== 1. Configure the hadoop cluster ==
     24
     25=== 1.1. Login to Hadoop Master ===
    2526
    2627{{{
     
    3233
    3334       <td >
    34          <ol type="A">
    35            <li>Login (ssh) to the hadoop-master using the credentials associated with the
    36 GENI Portal and the IP address displayed by Flack. The ssh application you use will
     35         <ol type="a">
     36           <li>Login (ssh) to the <code> master</code>  using the output of the readyToLogin script or gathering the necessary information from any tool you used for the reservation.  The ssh application you use will
    3737depend on the configuration of your laptop/desktop.</li>
    38          </op>
     38         </ol>
    3939</td></tr>
    4040</table>
    4141}}}
    4242
    43 == 2. Check the status/properties of the VMs. ==
    44 
    45 === A. Observe the properties of the network interfaces ===
    46 
    47 {{{
    48 # /sbin/ifconfig
    49 eth0      Link encap:Ethernet  HWaddr fa:16:3e:72:ad:a6 
    50           inet addr:10.103.0.20  Bcast:10.103.0.255  Mask:255.255.255.0
    51           inet6 addr: fe80::f816:3eff:fe72:ada6/64 Scope:Link
    52           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
    53           RX packets:1982 errors:0 dropped:0 overruns:0 frame:0
    54           TX packets:1246 errors:0 dropped:0 overruns:0 carrier:0
    55           collisions:0 txqueuelen:1000
    56           RX bytes:301066 (294.0 KiB)  TX bytes:140433 (137.1 KiB)
    57           Interrupt:11 Base address:0x2000
    58 
    59 eth1      Link encap:Ethernet  HWaddr fe:16:3e:00:6d:af 
    60           inet addr:172.16.1.1  Bcast:172.16.1.255  Mask:255.255.255.0
    61           inet6 addr: fe80::fc16:3eff:fe00:6daf/64 Scope:Link
    62           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
    63           RX packets:21704 errors:0 dropped:0 overruns:0 frame:0
    64           TX packets:4562 errors:0 dropped:0 overruns:0 carrier:0
    65           collisions:0 txqueuelen:1000
    66           RX bytes:3100262 (2.9 MiB)  TX bytes:824572 (805.2 KiB)
    67 
    68 lo        Link encap:Local Loopback 
    69           inet addr:127.0.0.1  Mask:255.0.0.0
    70           inet6 addr: ::1/128 Scope:Host
    71           UP LOOPBACK RUNNING  MTU:16436  Metric:1
    72           RX packets:19394 errors:0 dropped:0 overruns:0 frame:0
    73           TX packets:19394 errors:0 dropped:0 overruns:0 carrier:0
    74           collisions:0 txqueuelen:0
    75           RX bytes:4010954 (3.8 MiB)  TX bytes:4010954 (3.8 MiB)
     43=== 1.2. Look around your node. ==
     44
     45 a. Observe the properties of the network interfaces.
     46 {{{
     47# ifconfig
     48ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
     49        inet 10.103.0.17  netmask 255.255.255.0  broadcast 10.103.0.255
     50        inet6 fe80::f816:3eff:fe31:db4  prefixlen 64  scopeid 0x20<link>
     51        ether fa:16:3e:31:0d:b4  txqueuelen 1000  (Ethernet)
     52        RX packets 99237  bytes 12928515 (12.3 MiB)
     53        RX errors 0  dropped 0  overruns 0  frame 0
     54        TX packets 55213  bytes 44478177 (42.4 MiB)
     55        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
     56
     57eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
     58        inet 172.16.1.1  netmask 255.255.255.0  broadcast 172.16.1.255
     59        inet6 fe80::fc16:3eff:fe00:5b82  prefixlen 64  scopeid 0x20<link>
     60        ether fe:16:3e:00:5b:82  txqueuelen 1000  (Ethernet)
     61        RX packets 765818  bytes 1450907680 (1.3 GiB)
     62        RX errors 0  dropped 0  overruns 0  frame 0
     63        TX packets 511355  bytes 42217300 (40.2 MiB)
     64        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
     65
     66lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
     67        inet 127.0.0.1  netmask 255.0.0.0
     68        inet6 ::1  prefixlen 128  scopeid 0x10<host>
     69        loop  txqueuelen 0  (Local Loopback)
     70        RX packets 13502  bytes 1189668 (1.1 MiB)
     71        RX errors 0  dropped 0  overruns 0  frame 0
     72        TX packets 13502  bytes 1189668 (1.1 MiB)
     73        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    7674}}}
    77  
    78 
    79 ===  B. Observe the contents of the NEuca user data file.  This file includes a script that will install and execute the script that you configured for the VM. ===
     75 a. Observe the contents of the NEuca user data file.  This file includes a script that will install and execute the script that you configured for the VM. ===
    8076{{{
    8177# neuca-user-data
    8278[global]
    83 actor_id=67C4EFB4-7CBF-48C9-8195-934FF81434DC
    84 slice_id=39672f6e-610a-4d86-8810-30e02d20cc99
    85 reservation_id=55676541-5221-483d-bb60-429de025f275
    86 unit_id=902709a4-32f2-41fc-b85c-b4791c779580
     79actor_id=b0e0e413-77ef-4775-b476-040ca8377d1d
     80slice_id=e61547ac-a45d-4228-8483-dfe51945288d
     81reservation_id=7163823e-cf26-45ec-960b-a833522d2693
     82unit_id=8679cd5d-7cf2-4fea-9f51-6832e6ce0358
    8783;router= Not Specified
    8884;iscsi_initiator_iqn= Not Specified
    89 slice_name=urn:publicid:IDN+ch.geni.net:ADAMANT+slice+pruth-winter-camp
    90 unit_url=http://geni-orca.renci.org/owl/8210b4d7-4afc-4838-801f-c20a8f1f75ae#hadoop-master
    91 host_name=hadoop-master
     85slice_name=urn:publicid:IDN+ch.geni.net:GRW15Illinois+slice+testhadoop
     86unit_url=http://geni-orca.renci.org/owl/f3427516-3356-4525-97a6-752df5a7ec68#master
     87host_name=master
     88management_ip=128.120.83.37
     89physical_host=ucd-w4
     90nova_id=b7a54b3d-1410-4540-b57f-e5cf4827b3c3
     91[users]
     92....
    9293[interfaces]
    93 fe163e006daf=up:ipv4:172.16.1.1/24
     94fe163e005b82=up:ipv4:172.16.1.1/24
    9495[storage]
    9596[routes]
     
    9899        # Automatically generated boot script
    99100        # wget or curl must be installed on the image
    100         mkdir -p /tmp
    101         cd /tmp
     101        mkdir -p /home/hadoop/
     102        cd /home/hadoop/
    102103        if [ -x `which wget 2>/dev/null` ]; then
    103           wget -q -O `basename http://geni-images.renci.org/images/tutorials/GENI-hadoop/master.sh` http://geni-images.renci.org/images/tutorials/GENI-hadoop/master.sh
     104          wget -q -O `basename http://geni-images.renci.org/images/tutorials/GENI-hadoop/hadoop_config_dynamic.sh.tgz` http://geni-images.renci.org/images/tutorials/GENI-hadoop/hadoop_config_dynamic.sh.tgz
    104105        else if [ -x `which curl 2>/dev/null` ]; then
    105           curl http://geni-images.renci.org/images/tutorials/GENI-hadoop/master.sh > `basename http://geni-images.renci.org/images/tutorials/GENI-hadoop/master.sh`
     106          curl http://geni-images.renci.org/images/tutorials/GENI-hadoop/hadoop_config_dynamic.sh.tgz > `basename http://geni-images.renci.org/images/tutorials/GENI-hadoop/hadoop_config_dynamic.sh.tgz`
    106107        fi
    107108        fi
    108         eval "/bin/sh -c \"chmod +x /tmp/master.sh; /tmp/master.sh\""
     109        # untar
     110        tar -zxf `basename http://geni-images.renci.org/images/tutorials/GENI-hadoop/hadoop_config_dynamic.sh.tgz`
     111        eval "/bin/sh -c \"/home/hadoop/hadoop_config_dynamic.sh 2\""
    109112}}}
    110113
     
    112115=== C. Observe the contents of the of the script that was installed and executed on the VM. ===
    113116{{{
    114 # cat /tmp/master.sh
     117# sudo cat /home/hadoop/hadoop_config_dynamic.sh
     118
    115119#!/bin/bash
    116120
    117  echo "Hello from neuca script" > /home/hadoop/log
    118  MY_HOSTNAME=hadoop-master
    119  hostname $MY_HOSTNAME
    120  echo 172.16.1.1  hadoop-master  >> /etc/hosts
    121   echo 172.16.1.10 hadoop-worker-0 >> /etc/hosts
    122   echo 172.16.1.11 hadoop-worker-1 >> /etc/hosts
    123   echo 172.16.1.12 hadoop-worker-2 >> /etc/hosts
    124   echo 172.16.1.13 hadoop-worker-3 >> /etc/hosts
    125   echo 172.16.1.14 hadoop-worker-4 >> /etc/hosts
    126   echo 172.16.1.15 hadoop-worker-5 >> /etc/hosts
    127   echo 172.16.1.16 hadoop-worker-6 >> /etc/hosts
    128   echo 172.16.1.17 hadoop-worker-7 >> /etc/hosts
    129   echo 172.16.1.18 hadoop-worker-8 >> /etc/hosts
    130   echo 172.16.1.19 hadoop-worker-9 >> /etc/hosts
    131   echo 172.16.1.20 hadoop-worker-10 >> /etc/hosts
    132   echo 172.16.1.21 hadoop-worker-11 >> /etc/hosts
    133   echo 172.16.1.22 hadoop-worker-12 >> /etc/hosts
    134   echo 172.16.1.23 hadoop-worker-13 >> /etc/hosts
    135   echo 172.16.1.24 hadoop-worker-14 >> /etc/hosts
    136   echo 172.16.1.25 hadoop-worker-15 >> /etc/hosts
    137   while true; do
    138       PING=`ping -c 1 172.16.1.1 > /dev/null 2>&1`
    139       if [ "$?" = "0" ]; then
    140           break
    141       fi
    142       sleep 5
    143   done
    144   echo '/home/hadoop/hadoop-euca-init.sh 172.16.1.1 -master' >> /home/hadoop/log
    145   /home/hadoop/hadoop-euca-init.sh 172.16.1.1 -master
    146   echo "Done starting daemons" >> /home/hadoop/log
     121HADOOP_LOG='/home/hadoop/hadoop_boot.log'
     122
     123echo "Hello from neuca script" > $HADOOP_LOG
     124
     125....
    147126}}}
    148127