Changes between Initial Version and Version 1 of BBN_Emulab

Timestamp:

07/15/09 15:32:08 (15 years ago)

Author:

chaos@bbn.com

Comment:

--

BBN_Emulab

@@ +1 @@
+These pages describe integration work carried out by [mailto:czhang@bbn.com Chunhui (Evan) Zhang] as part of a GENI Internship at BBN Technologies.  Feedback and comments are welcome!
+
+
+== Project Goals ==
+
+
+1. Set up a BBN Emulab instance as aggregate manager of BBN Emulab aggregate and associate with Utah ProtoGENI clearing house.
+
+2. Evaluate the design and implementation of ProtoGENI in this environment, and document the experience for others in GENI.
+
+
+== Related Documnets ==
+
+http://groups.geni.net/geni/wiki/ProtoGENI-1Q09-status
+
+https://www.protogeni.net/trac/protogeni/wiki/BecomingAProtoGENISite
+
+ProtoGENI Clearing House Installation Document: http://www.protogeni.net/trac/protogeni/wiki/InstallingAClearingHouse
+
+Instructions for enabling the AM APIs on an Emulab installation: https://www.protogeni.net/trac/protogeni/wiki/ProtoGENIInstall
+
+Emulab Installation Document: http://users.emulab.net/trac/emulab/wiki/InstallDocs
+
+
+== Overall ProtoGENI Building Strategy ==
+
+1. Run 'Boss' and 'Ops' as virtual machine on top of a physical machine.
+
+2. Set up BBN Clearinghouse and BBN Emulab with one experiment node inside BBN firewall first.
+
+3. Then rebuild the BBN Emulab outside the firewall with public IPs and federate it with Utah clearing house.
+
+
+== Status Report and related instructions ==
+
+1. '''Joined Utah 'geni' project on www.emulab.net''' -- ''Completed'' (5 June, 2009)
+
+   Became a member of ProtoGENI developer's group in order to get updated information and ability to login to www.protogeni.net
+
+2. '''Requested for a account on Utah's CVS server''' -- ''Completed'' (8 June, 2009)
+
+   In order to check out the latest Emulab code(including the ProtoGENI code). Instructions.[http://users.emulab.net/trac/emulab/wiki/kb270].  
+
+3. '''Talked with Harry and Heidi to make concrete building plan and arrange the hardware resources''' -- ''Completed'' (22 June, 2009)
+
+   '''''First Stage BBN ProtoGENI Plan,'''''
+
+   [[Image(intern_plan.jpg)]]
+[[BR]]
+[[BR]]
+
+   '''''Second Stage BBN ProtoGENI Plan (May not necessary),'''''
+
+   [[Image(intern_plan2.jpg)]]
+[[BR]]
+[[BR]]
+
+   '''''Detailed Internal configuration for 'First Stage''''''
+
+   [[Image(protogeni_local_new.jpg)]]
+[[BR]]
+[[BR]]
+
+
+4. '''Basic software installation''' -- ''Completed'' (10 June, 2009)
+
+   Installed Fedora 10 x86_32bit (updated to kernel 2.6.27.24-170.2.68.fc10.i686) on host machine.[[BR]]
+   Installed [http://www.virtualbox.org/wiki/Linux_Downloads VirtualBox] for Fedora 10 32bit on emulabhost machine.[[BR]]
+ Step 1:  Download the software 
+ {{{
+ [root@geni-emuhost software]# wget http://download.virtualbox.org/virtualbox/2.2.4/VirtualBox-2.2.4_47978_fedora9-1.i386.rpm
+ }}}
+
+ Step 2: Install the dependencies
+ {{{
+ [root@geni-emuhost software]# yum -y install kernel-devel qt qt-x11
+ }}}
+
+ Step 3: Install the Virtualbox
+ {{{
+ [root@geni-emuhost software]# rpm -ivh VirtualBox-2.2.4_47978_fedora9-1.i386.rpm 
+  Preparing...                ########################################### [100%]
+    1:VirtualBox             ########################################### [100%]
+
+ Creating group 'vboxusers'. VM users must be member of that group!
+
+ No precompiled module for this kernel found -- trying to build one. Messages
+ emitted during module compilation will be logged to /var/log/vbox-install.log.
+
+ Success!
+
+ }}}
+[[BR]]
+[[BR]]
+
+
+    
+5. '''Virtual machine (FreeBSD guest) installation and virtual network design''' -- ''Completed'' (26 June, 2009)
+    
+   Reference: 
+              http://users.emulab.net/trac/emulab/wiki/install/fs-freebsd.html [[BR]]
+              http://users.emulab.net/trac/emulab/wiki/install/ops-freebsd.html [[BR]]
+              http://users.emulab.net/trac/emulab/wiki/install/boss-freebsd.html
+
+   Installed three freeBSD virtual machines: 
+
+     'clearing house' with ip 192.168.56.2(em1),[[BR]]
+     'boss'           with ip 192.168.56.3(em1),[[BR]]
+     'ops'            with ip 192.168.56.4(em1)
+
+   '''Installation Instructions.''' [wiki:BBN_Emulab/freebsdInstall]
+
+   All above NICs 'em1' are bridged to the host machine's NIC eth1.
+
+   Now we need(required by the Emulab installation process) a network for above servers to access internet. So we do,
+
+     'clearing house' with ip 128.89.80.197(em0),[[BR]]
+     'boss'           with ip 128.89.80.195(em0),[[BR]]
+     'ops'            with ip 128.89.80.196(em0)
+
+   All above NICs 'em0' are bridged to the host machine's NIC eth0.
+
+   At this point, we still need a mechanism to allow those virtual machines to access the host's hard disk so that virtual machines can access the software they need. 
+   I plan to use the 'shared folder' function provided by [http://www.virtualbox.org VirtualBox].(Doesn't work with FreeBSD guest!!!). Other plans: use third virtual NIC to build a 'host-only' network.
+   
+     "host"           with ip 192.168.57.1(vboxnet0),[[BR]]
+     'clearing house' with ip 192.168.57.2(em2),[[BR]]
+     'boss'           with ip 192.168.57.3(em2),[[BR]]
+     'ops'            with ip 192.168.57.4(em2)
+
+
+   '''Tip 1:''' You have two ways to change the IP address permanently in FreeBSD.
+     
+      1) Use command 'sysinstall' --> 'Configure' --> 'Networking' --> 'Interfaces', then change the IP.[[BR]]
+      2) Edit /etc/rc.conf, add 'ifconfig_em1="inet 192.168.57.2 netmask 255.255.255.0"' for example.
+
+      Finally, you need to do '/etc/rc.d/netif restart' to take the change into effect (You have to reboot if it failed to make change).
+
+   '''Tip 2:''' use 'scp root@192.168.57.1:/home/czhang/protogeni/software/testbed.tgz /root/protogeni' inside vm to copy stuff from host. 
+[[BR]]
+[[BR]]
+
+6. '''Build Private BBN Emulab and federate with Utah Clearing House''' -- ''Completed'' (13 July, 2009)
+[[BR]]
+
+ A). '''Build 'Ops' server'''
+
+ Reference: 
+
+ http://users.emulab.net/trac/emulab/wiki/install/ops-emusoft.html
+
+ '''Step 1:''' Get the required software from http://users.emulab.net/trac/emulab/wiki/install/required_software.html and cvs-public.flux.utah.edu
+ On host:
+ {{{
+ # cd /home/czhang/protogeni/software
+ # wget http://www.emulab.net/downloads/FreeBSD-6.3-20080901.tar.gz
+ # wget http://www.emulab.net/downloads/ports-20080901.tar.gz
+ # cvs -d cvs-public.flux.utah.edu:/flux/CVS checkout testbed
+ # tar zcvf testbed.tgz testbed //make it easier to copy to VM later
+ }}} 
+
+ '''Step 2:''' Make some required folders on 'Ops'
+ {{{
+ # mkdir /users /proj /groups
+ }}}
+ 
+ '''Step 3:''' Create a folder on 'Ops' to put the required software.
+ {{{
+ # mkdir /root/protogeni
+ }}}
+
+ '''Step 4:''' Copy required software from host.
+ {{{
+ # scp root@192.168.57.1:/home/czhang/protogeni/software/testbed.tgz /root/protogeni
+ # scp root@192.168.57.1:/home/czhang/protogeni/software/FreeBSD-6.3-20080901.tar.gz /root/protogeni
+ # scp root@192.168.57.1:/home/czhang/protogeni/software/ports-20080901.tar.gz /root/protogeni  
+ }}}
+
+ '''Step 5:''' Install perl and python on 'Ops'
+ {{{
+ # cd /root/protogeni
+ # tar xvf FreeBSD-6.3-20080901.tar.gz //it produces folder 'packages'
+ # cd /root/protogeni/packages
+ # pkg_add perl-5.8.8_1.tbz
+ # ln -s /usr/local/bin/perl5.8.8 /usr/bin/perl
+ # pkg_add python25-2.5.1_1.tbz
+ }}}
+
+ '''Step 6:''' Install 'ops' packages
+ {{{
+ # cd /root/protogeni
+ # pkg_add emulab-ops-2.1.tbz
+ }}}
+ 
+ The '''output''' is here. [attachment:wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt]
+
+ '''Step 7:''' Prepare 'defs-bbn-emulab' file.
+ {{{
+ # mkdir -p /usr/testbed/src
+
+ # cd /usr/testbed/src
+ # tar xvf /root/protogeni/testbed.tgz
+
+ # cd /usr/testbed/src/testbed
+ # cp defs-example-privatecnet defs-bbn-emulab
+ # vi defs-bbn-emulab
+ }}}
+ My ''''defs-bbn-emulab'''' [attachment:wiki:BBN_Emulab:defs-bbn-emulab].
+
+ '''Step 8:''' Run 'configure'.
+ {{{
+ # mkdir -p /usr/testbed/obj/testbed
+ # cd /usr/testbed/obj/testbed
+ # /usr/testbed/src/testbed/configure --with-TBDEFS=/usr/testbed/src/testbed/defs-bbn-emulab
+ }}}
+ The '''output''' is here. [attachment:wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt]
+
+ '''Step 9:''' Run the ops installation script
+ {{{
+ # cd /usr/testbed/obj/testbed/install
+ # /usr/bin/perl ops-install -p /root/protogeni/packages
+ }}}
+ The '''output''' is here. [attachment:wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt]
+
+ '''Step 10:''' Install Emulab software from source
+ {{{
+ # cd /usr/testbed/obj/testbed
+ # /usr/local/bin/gmake ops-install
+ }}}
+ The '''output''' is here. [attachment:wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt]
+[[BR]]
+
+ B). '''Build 'Boss' server and protogeni support'''
+
+ Reference:
+
+ http://users.emulab.net/trac/emulab/wiki/install/boss-emusoft.html [[BR]]
+ https://www.protogeni.net/trac/protogeni/wiki/ProtoGENIInstall
+
+ '''Step 1:''' Create a folder in 'Boss' VM to put the required software.
+ {{{
+ # mkdir /root/protogeni
+ }}}
+
+ '''Step 2:''' Copy required software from host.
+ {{{
+ # scp root@192.168.57.1:/home/czhang/protogeni/software/testbed.tgz /root/protogeni
+ # scp root@192.168.57.1:/home/czhang/protogeni/software/FreeBSD-6.3-20080901.tar.gz /root/protogeni
+ # scp root@192.168.57.1:/home/czhang/protogeni/software/ports-20080901.tar.gz /root/protogeni
+ }}}
+
+ '''Step 3:''' Install perl and python on 'Boss'.
+ {{{
+ # cd /root/protogeni
+ # tar xvf FreeBSD-6.3-20080901.tar.gz //it produces folder 'packages'
+ # cd /root/protogeni/packages
+ # pkg_add perl-5.8.8_1.tbz
+ # ln -s /usr/local/bin/perl5.8.8 /usr/bin/perl
+ # pkg_add python25-2.5.1_1.tbz
+ # pkg_add rsync-2.6.9_2.tbz
+ }}}
+
+ '''Step 4:''' Prepare 'defs-bbn-emulab' file.
+ {{{
+ # mkdir -p /usr/testbed/src
+
+ # cd /usr/testbed/src
+ # tar xvf /root/protogeni/testbed.tgz
+
+ # scp root@192.168.57.4:/usr/testbed/src/testbed/defs-bbn-emulab /usr/testbed/src/testbed/   //copy from 'Ops'
+ }}}
+ My ''''defs-bbn-emulab'''' [attachment:wiki:BBN_Emulab:defs-bbn-emulab].
+
+ '''Tips''': I choose to not enable ProtoGENI support when installing the regular Emulab. Enable it when installing ProtoGENI. 
+
+ '''Step 5:''' Install 'boss' packages.
+ {{{
+ # cd /root/protogeni
+ # pkg_add emulab-boss-2.1.tbz
+ }}}
+ The '''output''' is here. [attachment:wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt]
+
+
+ '''Step 6:''' Run 'configure' and apply named patch.
+ {{{
+ # mkdir -p /usr/testbed/obj/testbed
+ # cd /usr/testbed/obj/testbed
+ # /usr/testbed/src/testbed/configure --with-TBDEFS=/usr/testbed/src/testbed/defs-bbn-emulab
+ # patch -p0 < /usr/testbed/src/testbed/named/rc.named.6.3.patch
+ }}}
+ The '''output''' is here. [attachment:wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt]
+
+ '''Step 7:''' Run the boss installation script.
+ {{{
+ # cd /usr/testbed/obj/testbed/install
+ # /usr/bin/perl boss-install -p /root/protogeni/packages
+ }}}
+ The '''output''' is here. [attachment:wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt]
+
+ At this point, we have a functional Emulab website up and runing, here is the screenshot,
+
+ [[Image(bbn_emulab_website.jpg)]]
+
+ '''Step 8:''' Deal with 'named' and 'dhcpd' configuration.
+
+ '''For 'named':'''
+
+ The configuration file is '/etc/namedb/named.conf' and other zone files in the same folder.[[BR]]
+ The command to restart it is '/etc/rc.d/named restart'.[[BR]]
+
+ During the last step, I encountered a error with named setup. By temporarily ignore that error, I redid the last step and this time I successfully installed it.
+ Then I checked /etc/namedb and found named configuration file:'named.conf'(created by the boss installation script with two 'views', 
+ one is for the internal network and the other one for the external network), 'bbn.com.db.head' and 'bbn.com.internal.db.head'. As the 
+ Utah Emulab document said, run /usr/testbed/sbin/named_setup should produces two zone files 'bbn.com.db' and 'bbn.com.internal.db'. 
+ Then I ran it but it did not produce the zone files. So I manually edited those two files. The file is here.[attachment:wiki:BBN_Emulab:bbn.com.db] and [attachment:wiki:BBN_Emulab:bbn.com.internal.db]   
+ The named.conf is here. [attachment:wiki:BBN_Emulab:named.conf]
+
+ '''For 'dhcpd':'''
+
+ The configuration file is '/usr/local/etc/dhcpd.conf'[[BR]]
+ The way to modify it is: edit '/usr/local/etc/dhcpd.conf.template' first, then run '/usr/testbed/sbin/dhcpd_makeconf -i -r'[[BR]]
+ The command to start and stop it is '/usr/local/etc/rc.d/2.dhcpd.sh start/stop'  //2.dhcpd.sh is a dhcpd wrapper provided by Utah
+
+ When you done with the installation, the first thing to do is to add a option 'deny unknown-clients' to the dhcpd.conf. Because your dhcpd may cause conflict with other existing dhcp servers.
+
+ '''Step 9:''' Try to install protogeni(Failed!!! at this point)
+ 
+ At this point, I did not go further with the testbed configuration including switch configuration and adding node to the testbed. I'd like to try if I could enable the protogeni support at this stage (Follow the wiki page https://www.protogeni.net/trac/protogeni/wiki/ProtoGENIInstall ).(However, this failed to install protogeni with the below error message:)
+
+ [[Image(no_osid_error.jpg)]]
+
+ It seems caused by the fact that I did not customize the disk images and put related information into the database. So I plan to go ahead to do it.
+[[BR]]
+
+ C). '''Customize Memory Filesystem and continue to install protogeni''' (Do not "add node" and "customize the generic image" yet)
+
+ Reference:
+
+ http://users.emulab.net/trac/emulab/wiki/install/mfs.html
+
+ '''Step 1:''' Download tftpboot MFS tarball to host and copy to 'Boss'(I want to keep a copy on host for late use).
+
+ On Host:
+ {{{
+ # cd /home/czhang/protogeni/software
+ # wget http://www.emulab.net/downloads/tftpboot-latest.tar.gz
+ }}}
+ On 'Boss'
+ {{{
+ # scp root@192.168.57.1:/home/czhang/protogeni/software/tftpboot-latest.tar.gz /root/protogeni
+ # cd /tftpboot
+ # tar xvf /root/protogeni/tftpboot-latest.tar.gz
+ }}}
+
+ '''Step 2:''' Follow the 'README' extracted from tftpboot-latest.tar.gz to customize the mfs and add related information into database.
+
+ At this point, I tried to run 'sudo /usr/testbed/sbin/protogeni/initsite' again and was able to get federated with Utah Clearing House successfully!!
+
+ This is the output screenshot,
+  
+ [[Image(initsite_success.jpg)]]
+
+[[BR]]
+[[BR]]
+ 
+ D). '''Add a experiment node and customize generic images'''
+
+ Reference:
+
+ http://users.emulab.net/trac/emulab/wiki/install/switches-db.html [[BR]]
+ http://users.emulab.net/trac/emulab/wiki/install/new_nodes.html [[BR]]
+ http://users.emulab.net/trac/emulab/wiki/install/customize_images.html
+ 
+ '''Step 1:''' Initial Switch Setup.
+ {{{
+ # mysql tbdb
+ mysql> insert into node_types set class='switch', type='cisco2950-ios';  //Just assume our NEC switches are Cisco 2950
+ mysql> insert into nodes set node_id='cisco1',phys_nodeid='cisco1',type='cisco2950-ios',role='ctrlswitch';
+ mysql> insert into nodes set node_id='cisco2',phys_nodeid='cisco2',type='cisco2950-ios',role='testswitch';
+ mysql> insert into switch_stack_types set stack_id='Control', stack_type='generic', supports_private=0, single_domain=0, min_vlan=20, max_vlan=64;
+ mysql> insert into switch_stack_types set stack_id='Experiment', stack_type='generic', supports_private=0, single_domain=0, min_vlan=20, max_vlan=64;
+ mysql> insert into switch_stacks values ('cisco1','Control',1),('cisco2','Experiment',1);
+ }}}
+
+ At this point, we need to configure a vlan as IP interface for control switch and assign IP 192.168.56.5 to it. Then put lines '192.168.56.5    cisco1' and '192.168.56.6    cisco2' in file Boss:/etc/hosts.
+
+ As of now, we could ping from boss to both switches.
+
+ Detailed cable connection figure will be provided here.
+
+
+ '''Step 2:''' Add a node.
+ {{{
+ }}}
+ 
+ '''Step 3:''' Customize generic images
+[[BR]]
+[[BR]]
+
+
+8. '''Build BBN Clearinghouse (Optional, not complete)''' 
+
+  Reference: http://www.protogeni.net/trac/protogeni/wiki/InstallingAClearingHouse
+
+ '''Step 1:''' Create a folder in 'Clearinghouse' to put the required software.
+ {{{
+ # mkdir /root/protogeni
+ }}}
+
+ '''Step 2:''' Copy required software from host.
+ {{{
+ # scp root@192.168.57.1:/home/czhang/protogeni/software/testbed.tgz /root/protogeni
+ # scp root@192.168.57.1:/home/czhang/protogeni/software/FreeBSD-6.3-20080901.tar.gz /root/protogeni
+ # scp root@192.168.57.1:/home/czhang/protogeni/software/ports-20080901.tar.gz /root/protogeni  
+ }}}
+
+ '''Step 3:''' Install perl and python on 'Clearinghouse'
+ {{{
+ # cd /root/protogeni
+ # tar xvf FreeBSD-6.3-20080901.tar.gz //it produces folder 'packages'
+ # cd /root/protogeni/packages
+ # pkg_add perl-5.8.8_1.tbz
+ # ln -s /usr/local/bin/perl5.8.8 /usr/bin/perl
+ # pkg_add python25-2.5.1_1.tbz
+ }}}
+
+
+ '''Step 4:''' Prepare 'defs-clrhouse' file.
+ {{{
+ # mkdir -p /usr/testbed/src
+
+ # cd /usr/testbed/src
+ # tar xvf /root/protogeni/testbed.tgz
+
+
+
+
+ # cd /usr/testbed/src/testbed/protogeni/install
+ # cp defs-clrhouse defs-bbn-clrhouse
+ # vi defs-bbn-clrhouse
+ }}}
+ My ''''defs-bbn-clrhouse'''' [attachment:wiki:BBN_Emulab:defs-bbn-clrhouse].
+
+ '''Step 5:''' Do configure.
+ {{{
+ # mkdir -p /usr/testbed/obj/testbed
+ # cd /usr/testbed/obj/testbed
+ # /usr/testbed/src/testbed/configure --with-TBDEFS=/usr/testbed/src/testbed/protogeni/install/defs-clrhouse
+ }}}
+ The '''output''' is here. [attachment:wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt]
+
+ Problem: Configure did not produce 'clrhouse-install' script as indicated in the document from Utah(Solved! Do 'cvs up' and re-configure).
+