Changes between Version 3 and Version 4 of GENIExperimenter/Tutorials/jacks/HadoopInASlice/ExecuteExperiment
- Timestamp:
- 09/17/15 07:52:26 (9 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
GENIExperimenter/Tutorials/jacks/HadoopInASlice/ExecuteExperiment
v3 v4 1 [[PageOutline]] 1 2 = [.. Hadoop in a Slice] = 2 == Part II: Execute Experiment: Login to the nodes and execute the Hadoop experiment ==3 3 {{{ 4 4 #!html … … 18 18 </div> 19 19 }}} 20 = Instructions =21 20 22 21 Now that you have reserved your resources, you are ready to login to the slice and run some Hadoop examples. 23 22 24 == 1. Login to Hadoop Master == 23 == 1. Configure the hadoop cluster == 24 25 === 1.1. Login to Hadoop Master === 25 26 26 27 {{{ … … 32 33 33 34 <td > 34 <ol type="A"> 35 <li>Login (ssh) to the hadoop-master using the credentials associated with the 36 GENI Portal and the IP address displayed by Flack. The ssh application you use will 35 <ol type="a"> 36 <li>Login (ssh) to the <code> master</code> using the output of the readyToLogin script or gathering the necessary information from any tool you used for the reservation. The ssh application you use will 37 37 depend on the configuration of your laptop/desktop.</li> 38 </o p>38 </ol> 39 39 </td></tr> 40 40 </table> 41 41 }}} 42 42 43 == 2. Check the status/properties of the VMs. == 44 45 === A. Observe the properties of the network interfaces === 46 47 {{{ 48 # /sbin/ifconfig 49 eth0 Link encap:Ethernet HWaddr fa:16:3e:72:ad:a6 50 inet addr:10.103.0.20 Bcast:10.103.0.255 Mask:255.255.255.0 51 inet6 addr: fe80::f816:3eff:fe72:ada6/64 Scope:Link 52 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 53 RX packets:1982 errors:0 dropped:0 overruns:0 frame:0 54 TX packets:1246 errors:0 dropped:0 overruns:0 carrier:0 55 collisions:0 txqueuelen:1000 56 RX bytes:301066 (294.0 KiB) TX bytes:140433 (137.1 KiB) 57 Interrupt:11 Base address:0x2000 58 59 eth1 Link encap:Ethernet HWaddr fe:16:3e:00:6d:af 60 inet addr:172.16.1.1 Bcast:172.16.1.255 Mask:255.255.255.0 61 inet6 addr: fe80::fc16:3eff:fe00:6daf/64 Scope:Link 62 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 63 RX packets:21704 errors:0 dropped:0 overruns:0 frame:0 64 TX packets:4562 errors:0 dropped:0 overruns:0 carrier:0 65 collisions:0 txqueuelen:1000 66 RX bytes:3100262 (2.9 MiB) TX bytes:824572 (805.2 KiB) 67 68 lo Link encap:Local Loopback 69 inet addr:127.0.0.1 Mask:255.0.0.0 70 inet6 addr: ::1/128 Scope:Host 71 UP LOOPBACK RUNNING MTU:16436 Metric:1 72 RX packets:19394 errors:0 dropped:0 overruns:0 frame:0 73 TX packets:19394 errors:0 dropped:0 overruns:0 carrier:0 74 collisions:0 txqueuelen:0 75 RX bytes:4010954 (3.8 MiB) TX bytes:4010954 (3.8 MiB) 43 === 1.2. Look around your node. == 44 45 a. Observe the properties of the network interfaces. 46 {{{ 47 # ifconfig 48 ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 49 inet 10.103.0.17 netmask 255.255.255.0 broadcast 10.103.0.255 50 inet6 fe80::f816:3eff:fe31:db4 prefixlen 64 scopeid 0x20<link> 51 ether fa:16:3e:31:0d:b4 txqueuelen 1000 (Ethernet) 52 RX packets 99237 bytes 12928515 (12.3 MiB) 53 RX errors 0 dropped 0 overruns 0 frame 0 54 TX packets 55213 bytes 44478177 (42.4 MiB) 55 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 56 57 eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 58 inet 172.16.1.1 netmask 255.255.255.0 broadcast 172.16.1.255 59 inet6 fe80::fc16:3eff:fe00:5b82 prefixlen 64 scopeid 0x20<link> 60 ether fe:16:3e:00:5b:82 txqueuelen 1000 (Ethernet) 61 RX packets 765818 bytes 1450907680 (1.3 GiB) 62 RX errors 0 dropped 0 overruns 0 frame 0 63 TX packets 511355 bytes 42217300 (40.2 MiB) 64 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 65 66 lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 67 inet 127.0.0.1 netmask 255.0.0.0 68 inet6 ::1 prefixlen 128 scopeid 0x10<host> 69 loop txqueuelen 0 (Local Loopback) 70 RX packets 13502 bytes 1189668 (1.1 MiB) 71 RX errors 0 dropped 0 overruns 0 frame 0 72 TX packets 13502 bytes 1189668 (1.1 MiB) 73 TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 76 74 }}} 77 78 79 === B. Observe the contents of the NEuca user data file. This file includes a script that will install and execute the script that you configured for the VM. === 75 a. Observe the contents of the NEuca user data file. This file includes a script that will install and execute the script that you configured for the VM. === 80 76 {{{ 81 77 # neuca-user-data 82 78 [global] 83 actor_id= 67C4EFB4-7CBF-48C9-8195-934FF81434DC84 slice_id= 39672f6e-610a-4d86-8810-30e02d20cc9985 reservation_id= 55676541-5221-483d-bb60-429de025f27586 unit_id= 902709a4-32f2-41fc-b85c-b4791c77958079 actor_id=b0e0e413-77ef-4775-b476-040ca8377d1d 80 slice_id=e61547ac-a45d-4228-8483-dfe51945288d 81 reservation_id=7163823e-cf26-45ec-960b-a833522d2693 82 unit_id=8679cd5d-7cf2-4fea-9f51-6832e6ce0358 87 83 ;router= Not Specified 88 84 ;iscsi_initiator_iqn= Not Specified 89 slice_name=urn:publicid:IDN+ch.geni.net:ADAMANT+slice+pruth-winter-camp 90 unit_url=http://geni-orca.renci.org/owl/8210b4d7-4afc-4838-801f-c20a8f1f75ae#hadoop-master 91 host_name=hadoop-master 85 slice_name=urn:publicid:IDN+ch.geni.net:GRW15Illinois+slice+testhadoop 86 unit_url=http://geni-orca.renci.org/owl/f3427516-3356-4525-97a6-752df5a7ec68#master 87 host_name=master 88 management_ip=128.120.83.37 89 physical_host=ucd-w4 90 nova_id=b7a54b3d-1410-4540-b57f-e5cf4827b3c3 91 [users] 92 .... 92 93 [interfaces] 93 fe163e00 6daf=up:ipv4:172.16.1.1/2494 fe163e005b82=up:ipv4:172.16.1.1/24 94 95 [storage] 95 96 [routes] … … 98 99 # Automatically generated boot script 99 100 # wget or curl must be installed on the image 100 mkdir -p / tmp101 cd / tmp101 mkdir -p /home/hadoop/ 102 cd /home/hadoop/ 102 103 if [ -x `which wget 2>/dev/null` ]; then 103 wget -q -O `basename http://geni-images.renci.org/images/tutorials/GENI-hadoop/ master.sh` http://geni-images.renci.org/images/tutorials/GENI-hadoop/master.sh104 wget -q -O `basename http://geni-images.renci.org/images/tutorials/GENI-hadoop/hadoop_config_dynamic.sh.tgz` http://geni-images.renci.org/images/tutorials/GENI-hadoop/hadoop_config_dynamic.sh.tgz 104 105 else if [ -x `which curl 2>/dev/null` ]; then 105 curl http://geni-images.renci.org/images/tutorials/GENI-hadoop/ master.sh > `basename http://geni-images.renci.org/images/tutorials/GENI-hadoop/master.sh`106 curl http://geni-images.renci.org/images/tutorials/GENI-hadoop/hadoop_config_dynamic.sh.tgz > `basename http://geni-images.renci.org/images/tutorials/GENI-hadoop/hadoop_config_dynamic.sh.tgz` 106 107 fi 107 108 fi 108 eval "/bin/sh -c \"chmod +x /tmp/master.sh; /tmp/master.sh\"" 109 # untar 110 tar -zxf `basename http://geni-images.renci.org/images/tutorials/GENI-hadoop/hadoop_config_dynamic.sh.tgz` 111 eval "/bin/sh -c \"/home/hadoop/hadoop_config_dynamic.sh 2\"" 109 112 }}} 110 113 … … 112 115 === C. Observe the contents of the of the script that was installed and executed on the VM. === 113 116 {{{ 114 # cat /tmp/master.sh 117 # sudo cat /home/hadoop/hadoop_config_dynamic.sh 118 115 119 #!/bin/bash 116 120 117 echo "Hello from neuca script" > /home/hadoop/log 118 MY_HOSTNAME=hadoop-master 119 hostname $MY_HOSTNAME 120 echo 172.16.1.1 hadoop-master >> /etc/hosts 121 echo 172.16.1.10 hadoop-worker-0 >> /etc/hosts 122 echo 172.16.1.11 hadoop-worker-1 >> /etc/hosts 123 echo 172.16.1.12 hadoop-worker-2 >> /etc/hosts 124 echo 172.16.1.13 hadoop-worker-3 >> /etc/hosts 125 echo 172.16.1.14 hadoop-worker-4 >> /etc/hosts 126 echo 172.16.1.15 hadoop-worker-5 >> /etc/hosts 127 echo 172.16.1.16 hadoop-worker-6 >> /etc/hosts 128 echo 172.16.1.17 hadoop-worker-7 >> /etc/hosts 129 echo 172.16.1.18 hadoop-worker-8 >> /etc/hosts 130 echo 172.16.1.19 hadoop-worker-9 >> /etc/hosts 131 echo 172.16.1.20 hadoop-worker-10 >> /etc/hosts 132 echo 172.16.1.21 hadoop-worker-11 >> /etc/hosts 133 echo 172.16.1.22 hadoop-worker-12 >> /etc/hosts 134 echo 172.16.1.23 hadoop-worker-13 >> /etc/hosts 135 echo 172.16.1.24 hadoop-worker-14 >> /etc/hosts 136 echo 172.16.1.25 hadoop-worker-15 >> /etc/hosts 137 while true; do 138 PING=`ping -c 1 172.16.1.1 > /dev/null 2>&1` 139 if [ "$?" = "0" ]; then 140 break 141 fi 142 sleep 5 143 done 144 echo '/home/hadoop/hadoop-euca-init.sh 172.16.1.1 -master' >> /home/hadoop/log 145 /home/hadoop/hadoop-euca-init.sh 172.16.1.1 -master 146 echo "Done starting daemons" >> /home/hadoop/log 121 HADOOP_LOG='/home/hadoop/hadoop_boot.log' 122 123 echo "Hello from neuca script" > $HADOOP_LOG 124 125 .... 147 126 }}} 148 127