Version 1 (modified by 15 years ago) (diff) | ,
---|
These pages describe integration work carried out by Chunhui (Evan) Zhang as part of a GENI Internship at BBN Technologies. Feedback and comments are welcome!
Project Goals
- Set up a BBN Emulab instance as aggregate manager of BBN Emulab aggregate and associate with Utah ProtoGENI clearing house.
- Evaluate the design and implementation of ProtoGENI in this environment, and document the experience for others in GENI.
Related Documnets
http://groups.geni.net/geni/wiki/ProtoGENI-1Q09-status
https://www.protogeni.net/trac/protogeni/wiki/BecomingAProtoGENISite
ProtoGENI Clearing House Installation Document: http://www.protogeni.net/trac/protogeni/wiki/InstallingAClearingHouse
Instructions for enabling the AM APIs on an Emulab installation: https://www.protogeni.net/trac/protogeni/wiki/ProtoGENIInstall
Emulab Installation Document: http://users.emulab.net/trac/emulab/wiki/InstallDocs
Overall ProtoGENI Building Strategy
- Run 'Boss' and 'Ops' as virtual machine on top of a physical machine.
- Set up BBN Clearinghouse and BBN Emulab with one experiment node inside BBN firewall first.
- Then rebuild the BBN Emulab outside the firewall with public IPs and federate it with Utah clearing house.
Status Report and related instructions
- Joined Utah 'geni' project on www.emulab.net -- Completed (5 June, 2009)
Became a member of ProtoGENI developer's group in order to get updated information and ability to login to www.protogeni.net
- Requested for a account on Utah's CVS server -- Completed (8 June, 2009)
In order to check out the latest Emulab code(including the ProtoGENI code). Instructions.http://users.emulab.net/trac/emulab/wiki/kb270.
- Talked with Harry and Heidi to make concrete building plan and arrange the hardware resources -- Completed (22 June, 2009)
First Stage BBN ProtoGENI Plan,
Second Stage BBN ProtoGENI Plan (May not necessary),
Detailed Internal configuration for 'First Stage'
- Basic software installation -- Completed (10 June, 2009)
Installed Fedora 10 x86_32bit (updated to kernel 2.6.27.24-170.2.68.fc10.i686) on host machine.
Installed VirtualBox for Fedora 10 32bit on emulabhost machine.
Step 1: Download the software
[root@geni-emuhost software]# wget http://download.virtualbox.org/virtualbox/2.2.4/VirtualBox-2.2.4_47978_fedora9-1.i386.rpm
Step 2: Install the dependencies
[root@geni-emuhost software]# yum -y install kernel-devel qt qt-x11
Step 3: Install the Virtualbox
[root@geni-emuhost software]# rpm -ivh VirtualBox-2.2.4_47978_fedora9-1.i386.rpm Preparing... ########################################### [100%] 1:VirtualBox ########################################### [100%] Creating group 'vboxusers'. VM users must be member of that group! No precompiled module for this kernel found -- trying to build one. Messages emitted during module compilation will be logged to /var/log/vbox-install.log. Success!
- Virtual machine (FreeBSD guest) installation and virtual network design -- Completed (26 June, 2009)
Reference:
http://users.emulab.net/trac/emulab/wiki/install/fs-freebsd.html
http://users.emulab.net/trac/emulab/wiki/install/ops-freebsd.html
http://users.emulab.net/trac/emulab/wiki/install/boss-freebsd.html
Installed three freeBSD virtual machines:
'clearing house' with ip 192.168.56.2(em1),
'boss' with ip 192.168.56.3(em1),
'ops' with ip 192.168.56.4(em1)
Installation Instructions. BBN_Emulab/freebsdInstall
All above NICs 'em1' are bridged to the host machine's NIC eth1.
Now we need(required by the Emulab installation process) a network for above servers to access internet. So we do,
'clearing house' with ip 128.89.80.197(em0),
'boss' with ip 128.89.80.195(em0),
'ops' with ip 128.89.80.196(em0)
All above NICs 'em0' are bridged to the host machine's NIC eth0.
At this point, we still need a mechanism to allow those virtual machines to access the host's hard disk so that virtual machines can access the software they need. I plan to use the 'shared folder' function provided by VirtualBox.(Doesn't work with FreeBSD guest!!!). Other plans: use third virtual NIC to build a 'host-only' network.
"host" with ip 192.168.57.1(vboxnet0),
'clearing house' with ip 192.168.57.2(em2),
'boss' with ip 192.168.57.3(em2),
'ops' with ip 192.168.57.4(em2)
Tip 1: You have two ways to change the IP address permanently in FreeBSD.
1) Use command 'sysinstall' --> 'Configure' --> 'Networking' --> 'Interfaces', then change the IP.
2) Edit /etc/rc.conf, add 'ifconfig_em1="inet 192.168.57.2 netmask 255.255.255.0"' for example.
Finally, you need to do '/etc/rc.d/netif restart' to take the change into effect (You have to reboot if it failed to make change).
Tip 2: use 'scp root@192.168.57.1:/home/czhang/protogeni/software/testbed.tgz /root/protogeni' inside vm to copy stuff from host.
- Build Private BBN Emulab and federate with Utah Clearing House -- Completed (13 July, 2009)
A). Build 'Ops' server
Reference:
http://users.emulab.net/trac/emulab/wiki/install/ops-emusoft.html
Step 1: Get the required software from http://users.emulab.net/trac/emulab/wiki/install/required_software.html and cvs-public.flux.utah.edu On host:
# cd /home/czhang/protogeni/software # wget http://www.emulab.net/downloads/FreeBSD-6.3-20080901.tar.gz # wget http://www.emulab.net/downloads/ports-20080901.tar.gz # cvs -d cvs-public.flux.utah.edu:/flux/CVS checkout testbed # tar zcvf testbed.tgz testbed //make it easier to copy to VM later
Step 2: Make some required folders on 'Ops'
# mkdir /users /proj /groups
Step 3: Create a folder on 'Ops' to put the required software.
# mkdir /root/protogeni
Step 4: Copy required software from host.
# scp root@192.168.57.1:/home/czhang/protogeni/software/testbed.tgz /root/protogeni # scp root@192.168.57.1:/home/czhang/protogeni/software/FreeBSD-6.3-20080901.tar.gz /root/protogeni # scp root@192.168.57.1:/home/czhang/protogeni/software/ports-20080901.tar.gz /root/protogeni
Step 5: Install perl and python on 'Ops'
# cd /root/protogeni # tar xvf FreeBSD-6.3-20080901.tar.gz //it produces folder 'packages' # cd /root/protogeni/packages # pkg_add perl-5.8.8_1.tbz # ln -s /usr/local/bin/perl5.8.8 /usr/bin/perl # pkg_add python25-2.5.1_1.tbz
Step 6: Install 'ops' packages
# cd /root/protogeni # pkg_add emulab-ops-2.1.tbz
The output is here. wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt
Step 7: Prepare 'defs-bbn-emulab' file.
# mkdir -p /usr/testbed/src # cd /usr/testbed/src # tar xvf /root/protogeni/testbed.tgz # cd /usr/testbed/src/testbed # cp defs-example-privatecnet defs-bbn-emulab # vi defs-bbn-emulabMy 'defs-bbn-emulab' wiki:BBN_Emulab:defs-bbn-emulab.
Step 8: Run 'configure'.
# mkdir -p /usr/testbed/obj/testbed # cd /usr/testbed/obj/testbed # /usr/testbed/src/testbed/configure --with-TBDEFS=/usr/testbed/src/testbed/defs-bbn-emulabThe output is here. wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt
Step 9: Run the ops installation script
# cd /usr/testbed/obj/testbed/install # /usr/bin/perl ops-install -p /root/protogeni/packagesThe output is here. wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt
Step 10: Install Emulab software from source
# cd /usr/testbed/obj/testbed # /usr/local/bin/gmake ops-installThe output is here. wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt
B). Build 'Boss' server and protogeni support
Reference:
http://users.emulab.net/trac/emulab/wiki/install/boss-emusoft.html
https://www.protogeni.net/trac/protogeni/wiki/ProtoGENIInstall
Step 1: Create a folder in 'Boss' VM to put the required software.
# mkdir /root/protogeni
Step 2: Copy required software from host.
# scp root@192.168.57.1:/home/czhang/protogeni/software/testbed.tgz /root/protogeni # scp root@192.168.57.1:/home/czhang/protogeni/software/FreeBSD-6.3-20080901.tar.gz /root/protogeni # scp root@192.168.57.1:/home/czhang/protogeni/software/ports-20080901.tar.gz /root/protogeni
Step 3: Install perl and python on 'Boss'.
# cd /root/protogeni # tar xvf FreeBSD-6.3-20080901.tar.gz //it produces folder 'packages' # cd /root/protogeni/packages # pkg_add perl-5.8.8_1.tbz # ln -s /usr/local/bin/perl5.8.8 /usr/bin/perl # pkg_add python25-2.5.1_1.tbz # pkg_add rsync-2.6.9_2.tbz
Step 4: Prepare 'defs-bbn-emulab' file.
# mkdir -p /usr/testbed/src # cd /usr/testbed/src # tar xvf /root/protogeni/testbed.tgz # scp root@192.168.57.4:/usr/testbed/src/testbed/defs-bbn-emulab /usr/testbed/src/testbed/ //copy from 'Ops'My 'defs-bbn-emulab' wiki:BBN_Emulab:defs-bbn-emulab.
Tips: I choose to not enable ProtoGENI support when installing the regular Emulab. Enable it when installing ProtoGENI.
Step 5: Install 'boss' packages.
# cd /root/protogeni # pkg_add emulab-boss-2.1.tbzThe output is here. wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt
Step 6: Run 'configure' and apply named patch.
# mkdir -p /usr/testbed/obj/testbed # cd /usr/testbed/obj/testbed # /usr/testbed/src/testbed/configure --with-TBDEFS=/usr/testbed/src/testbed/defs-bbn-emulab # patch -p0 < /usr/testbed/src/testbed/named/rc.named.6.3.patchThe output is here. wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt
Step 7: Run the boss installation script.
# cd /usr/testbed/obj/testbed/install # /usr/bin/perl boss-install -p /root/protogeni/packagesThe output is here. wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt
At this point, we have a functional Emulab website up and runing, here is the screenshot,
Step 8: Deal with 'named' and 'dhcpd' configuration.
For 'named':
The configuration file is '/etc/namedb/named.conf' and other zone files in the same folder.
The command to restart it is '/etc/rc.d/named restart'.
During the last step, I encountered a error with named setup. By temporarily ignore that error, I redid the last step and this time I successfully installed it. Then I checked /etc/namedb and found named configuration file:'named.conf'(created by the boss installation script with two 'views', one is for the internal network and the other one for the external network), 'bbn.com.db.head' and 'bbn.com.internal.db.head'. As the Utah Emulab document said, run /usr/testbed/sbin/named_setup should produces two zone files 'bbn.com.db' and 'bbn.com.internal.db'. Then I ran it but it did not produce the zone files. So I manually edited those two files. The file is here.wiki:BBN_Emulab:bbn.com.db and wiki:BBN_Emulab:bbn.com.internal.db The named.conf is here. wiki:BBN_Emulab:named.conf
For 'dhcpd':
The configuration file is '/usr/local/etc/dhcpd.conf'
The way to modify it is: edit '/usr/local/etc/dhcpd.conf.template' first, then run '/usr/testbed/sbin/dhcpd_makeconf -i -r'
The command to start and stop it is '/usr/local/etc/rc.d/2.dhcpd.sh start/stop' 2.dhcpd.sh is a dhcpd wrapper provided by Utah
When you done with the installation, the first thing to do is to add a option 'deny unknown-clients' to the dhcpd.conf. Because your dhcpd may cause conflict with other existing dhcp servers.
Step 9: Try to install protogeni(Failed!!! at this point)
At this point, I did not go further with the testbed configuration including switch configuration and adding node to the testbed. I'd like to try if I could enable the protogeni support at this stage (Follow the wiki page https://www.protogeni.net/trac/protogeni/wiki/ProtoGENIInstall ).(However, this failed to install protogeni with the below error message:)
It seems caused by the fact that I did not customize the disk images and put related information into the database. So I plan to go ahead to do it.
C). Customize Memory Filesystem and continue to install protogeni (Do not "add node" and "customize the generic image" yet)
Reference:
Step 1: Download tftpboot MFS tarball to host and copy to 'Boss'(I want to keep a copy on host for late use).
On Host:
# cd /home/czhang/protogeni/software # wget http://www.emulab.net/downloads/tftpboot-latest.tar.gzOn 'Boss'
# scp root@192.168.57.1:/home/czhang/protogeni/software/tftpboot-latest.tar.gz /root/protogeni # cd /tftpboot # tar xvf /root/protogeni/tftpboot-latest.tar.gz
Step 2: Follow the 'README' extracted from tftpboot-latest.tar.gz to customize the mfs and add related information into database.
At this point, I tried to run 'sudo /usr/testbed/sbin/protogeni/initsite' again and was able to get federated with Utah Clearing House successfully!!
This is the output screenshot,
D). Add a experiment node and customize generic images
Reference:
http://users.emulab.net/trac/emulab/wiki/install/switches-db.html
http://users.emulab.net/trac/emulab/wiki/install/new_nodes.html
http://users.emulab.net/trac/emulab/wiki/install/customize_images.html
Step 1: Initial Switch Setup.
# mysql tbdb mysql> insert into node_types set class='switch', type='cisco2950-ios'; //Just assume our NEC switches are Cisco 2950 mysql> insert into nodes set node_id='cisco1',phys_nodeid='cisco1',type='cisco2950-ios',role='ctrlswitch'; mysql> insert into nodes set node_id='cisco2',phys_nodeid='cisco2',type='cisco2950-ios',role='testswitch'; mysql> insert into switch_stack_types set stack_id='Control', stack_type='generic', supports_private=0, single_domain=0, min_vlan=20, max_vlan=64; mysql> insert into switch_stack_types set stack_id='Experiment', stack_type='generic', supports_private=0, single_domain=0, min_vlan=20, max_vlan=64; mysql> insert into switch_stacks values ('cisco1','Control',1),('cisco2','Experiment',1);
At this point, we need to configure a vlan as IP interface for control switch and assign IP 192.168.56.5 to it. Then put lines '192.168.56.5 cisco1' and '192.168.56.6 cisco2' in file Boss:/etc/hosts.
As of now, we could ping from boss to both switches.
Detailed cable connection figure will be provided here.
Step 2: Add a node.
Step 3: Customize generic images
- Build BBN Clearinghouse (Optional, not complete)
Reference: http://www.protogeni.net/trac/protogeni/wiki/InstallingAClearingHouse
Step 1: Create a folder in 'Clearinghouse' to put the required software.
# mkdir /root/protogeni
Step 2: Copy required software from host.
# scp root@192.168.57.1:/home/czhang/protogeni/software/testbed.tgz /root/protogeni # scp root@192.168.57.1:/home/czhang/protogeni/software/FreeBSD-6.3-20080901.tar.gz /root/protogeni # scp root@192.168.57.1:/home/czhang/protogeni/software/ports-20080901.tar.gz /root/protogeni
Step 3: Install perl and python on 'Clearinghouse'
# cd /root/protogeni # tar xvf FreeBSD-6.3-20080901.tar.gz //it produces folder 'packages' # cd /root/protogeni/packages # pkg_add perl-5.8.8_1.tbz # ln -s /usr/local/bin/perl5.8.8 /usr/bin/perl # pkg_add python25-2.5.1_1.tbz
Step 4: Prepare 'defs-clrhouse' file.
# mkdir -p /usr/testbed/src # cd /usr/testbed/src # tar xvf /root/protogeni/testbed.tgz # cd /usr/testbed/src/testbed/protogeni/install # cp defs-clrhouse defs-bbn-clrhouse # vi defs-bbn-clrhouseMy 'defs-bbn-clrhouse' wiki:BBN_Emulab:defs-bbn-clrhouse.
Step 5: Do configure.
# mkdir -p /usr/testbed/obj/testbed # cd /usr/testbed/obj/testbed # /usr/testbed/src/testbed/configure --with-TBDEFS=/usr/testbed/src/testbed/protogeni/install/defs-clrhouseThe output is here. wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt
Problem: Configure did not produce 'clrhouse-install' script as indicated in the document from Utah(Solved! Do 'cvs up' and re-configure).
Attachments (15)
- em0_config.JPG (185.9 KB) - added by 15 years ago.
- em1_config.JPG (190.9 KB) - added by 15 years ago.
- em1_config1.JPG (269.0 KB) - added by 15 years ago.
- intern_plan.jpg (193.9 KB) - added by 15 years ago.
- intern_plan2.jpg (192.9 KB) - added by 15 years ago.
- protogeni_local_new.jpg (269.0 KB) - added by 15 years ago.
- no_osid_error.jpg (45.1 KB) - added by 15 years ago.
- initsite_success.jpg (85.5 KB) - added by 15 years ago.
- bbn_emulab_website.jpg (265.0 KB) - added by 15 years ago.
- defs-bbn-clrhouse (465 bytes) - added by 15 years ago.
- defs-bbn-emulab (7.9 KB) - added by 15 years ago.
- named.conf (1.4 KB) - added by 15 years ago.
- bbn.com.db (425 bytes) - added by 15 years ago.
- bbn.com.internal.db (441 bytes) - added by 15 years ago.
- BBN_Emulab_commandoutput.txt (39.1 KB) - added by 15 years ago.
Download all attachments as: .zip