wiki:BBN_Emulab

Version 1 (modified by chaos@bbn.com, 15 years ago) (diff)

--

These pages describe integration work carried out by Chunhui (Evan) Zhang as part of a GENI Internship at BBN Technologies. Feedback and comments are welcome!

Project Goals

  1. Set up a BBN Emulab instance as aggregate manager of BBN Emulab aggregate and associate with Utah ProtoGENI clearing house.
  1. Evaluate the design and implementation of ProtoGENI in this environment, and document the experience for others in GENI.

Related Documnets

http://groups.geni.net/geni/wiki/ProtoGENI-1Q09-status

https://www.protogeni.net/trac/protogeni/wiki/BecomingAProtoGENISite

ProtoGENI Clearing House Installation Document: http://www.protogeni.net/trac/protogeni/wiki/InstallingAClearingHouse

Instructions for enabling the AM APIs on an Emulab installation: https://www.protogeni.net/trac/protogeni/wiki/ProtoGENIInstall

Emulab Installation Document: http://users.emulab.net/trac/emulab/wiki/InstallDocs

Overall ProtoGENI Building Strategy

  1. Run 'Boss' and 'Ops' as virtual machine on top of a physical machine.
  1. Set up BBN Clearinghouse and BBN Emulab with one experiment node inside BBN firewall first.
  1. Then rebuild the BBN Emulab outside the firewall with public IPs and federate it with Utah clearing house.

Status Report and related instructions

  1. Joined Utah 'geni' project on www.emulab.net -- Completed (5 June, 2009)

Became a member of ProtoGENI developer's group in order to get updated information and ability to login to www.protogeni.net

  1. Requested for a account on Utah's CVS server -- Completed (8 June, 2009)

In order to check out the latest Emulab code(including the ProtoGENI code). Instructions.http://users.emulab.net/trac/emulab/wiki/kb270.

  1. Talked with Harry and Heidi to make concrete building plan and arrange the hardware resources -- Completed (22 June, 2009)

First Stage BBN ProtoGENI Plan,



Second Stage BBN ProtoGENI Plan (May not necessary),



Detailed Internal configuration for 'First Stage'



  1. Basic software installation -- Completed (10 June, 2009)

Installed Fedora 10 x86_32bit (updated to kernel 2.6.27.24-170.2.68.fc10.i686) on host machine.
Installed VirtualBox for Fedora 10 32bit on emulabhost machine.

Step 1: Download the software

[root@geni-emuhost software]# wget http://download.virtualbox.org/virtualbox/2.2.4/VirtualBox-2.2.4_47978_fedora9-1.i386.rpm

Step 2: Install the dependencies

[root@geni-emuhost software]# yum -y install kernel-devel qt qt-x11

Step 3: Install the Virtualbox

[root@geni-emuhost software]# rpm -ivh VirtualBox-2.2.4_47978_fedora9-1.i386.rpm 
 Preparing...                ########################################### [100%]
   1:VirtualBox             ########################################### [100%]

Creating group 'vboxusers'. VM users must be member of that group!

No precompiled module for this kernel found -- trying to build one. Messages
emitted during module compilation will be logged to /var/log/vbox-install.log.

Success!



  1. Virtual machine (FreeBSD guest) installation and virtual network design -- Completed (26 June, 2009)

Reference:

http://users.emulab.net/trac/emulab/wiki/install/fs-freebsd.html
http://users.emulab.net/trac/emulab/wiki/install/ops-freebsd.html
http://users.emulab.net/trac/emulab/wiki/install/boss-freebsd.html

Installed three freeBSD virtual machines:

'clearing house' with ip 192.168.56.2(em1),
'boss' with ip 192.168.56.3(em1),
'ops' with ip 192.168.56.4(em1)

Installation Instructions. BBN_Emulab/freebsdInstall

All above NICs 'em1' are bridged to the host machine's NIC eth1.

Now we need(required by the Emulab installation process) a network for above servers to access internet. So we do,

'clearing house' with ip 128.89.80.197(em0),
'boss' with ip 128.89.80.195(em0),
'ops' with ip 128.89.80.196(em0)

All above NICs 'em0' are bridged to the host machine's NIC eth0.

At this point, we still need a mechanism to allow those virtual machines to access the host's hard disk so that virtual machines can access the software they need. I plan to use the 'shared folder' function provided by VirtualBox.(Doesn't work with FreeBSD guest!!!). Other plans: use third virtual NIC to build a 'host-only' network.

"host" with ip 192.168.57.1(vboxnet0),
'clearing house' with ip 192.168.57.2(em2),
'boss' with ip 192.168.57.3(em2),
'ops' with ip 192.168.57.4(em2)

Tip 1: You have two ways to change the IP address permanently in FreeBSD.

1) Use command 'sysinstall' --> 'Configure' --> 'Networking' --> 'Interfaces', then change the IP.
2) Edit /etc/rc.conf, add 'ifconfig_em1="inet 192.168.57.2 netmask 255.255.255.0"' for example.

Finally, you need to do '/etc/rc.d/netif restart' to take the change into effect (You have to reboot if it failed to make change).

Tip 2: use 'scp root@192.168.57.1:/home/czhang/protogeni/software/testbed.tgz /root/protogeni' inside vm to copy stuff from host.



  1. Build Private BBN Emulab and federate with Utah Clearing House -- Completed (13 July, 2009)


A). Build 'Ops' server

Reference:

http://users.emulab.net/trac/emulab/wiki/install/ops-emusoft.html

Step 1: Get the required software from http://users.emulab.net/trac/emulab/wiki/install/required_software.html and cvs-public.flux.utah.edu On host:

# cd /home/czhang/protogeni/software
# wget http://www.emulab.net/downloads/FreeBSD-6.3-20080901.tar.gz
# wget http://www.emulab.net/downloads/ports-20080901.tar.gz
# cvs -d cvs-public.flux.utah.edu:/flux/CVS checkout testbed
# tar zcvf testbed.tgz testbed //make it easier to copy to VM later

Step 2: Make some required folders on 'Ops'

# mkdir /users /proj /groups

Step 3: Create a folder on 'Ops' to put the required software.

# mkdir /root/protogeni

Step 4: Copy required software from host.

# scp root@192.168.57.1:/home/czhang/protogeni/software/testbed.tgz /root/protogeni
# scp root@192.168.57.1:/home/czhang/protogeni/software/FreeBSD-6.3-20080901.tar.gz /root/protogeni
# scp root@192.168.57.1:/home/czhang/protogeni/software/ports-20080901.tar.gz /root/protogeni  

Step 5: Install perl and python on 'Ops'

# cd /root/protogeni
# tar xvf FreeBSD-6.3-20080901.tar.gz //it produces folder 'packages'
# cd /root/protogeni/packages
# pkg_add perl-5.8.8_1.tbz
# ln -s /usr/local/bin/perl5.8.8 /usr/bin/perl
# pkg_add python25-2.5.1_1.tbz

Step 6: Install 'ops' packages

# cd /root/protogeni
# pkg_add emulab-ops-2.1.tbz

The output is here. wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt

Step 7: Prepare 'defs-bbn-emulab' file.

# mkdir -p /usr/testbed/src

# cd /usr/testbed/src
# tar xvf /root/protogeni/testbed.tgz

# cd /usr/testbed/src/testbed
# cp defs-example-privatecnet defs-bbn-emulab
# vi defs-bbn-emulab

My 'defs-bbn-emulab' wiki:BBN_Emulab:defs-bbn-emulab.

Step 8: Run 'configure'.

# mkdir -p /usr/testbed/obj/testbed
# cd /usr/testbed/obj/testbed
# /usr/testbed/src/testbed/configure --with-TBDEFS=/usr/testbed/src/testbed/defs-bbn-emulab

The output is here. wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt

Step 9: Run the ops installation script

# cd /usr/testbed/obj/testbed/install
# /usr/bin/perl ops-install -p /root/protogeni/packages

The output is here. wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt

Step 10: Install Emulab software from source

# cd /usr/testbed/obj/testbed
# /usr/local/bin/gmake ops-install

The output is here. wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt


B). Build 'Boss' server and protogeni support

Reference:

http://users.emulab.net/trac/emulab/wiki/install/boss-emusoft.html
https://www.protogeni.net/trac/protogeni/wiki/ProtoGENIInstall

Step 1: Create a folder in 'Boss' VM to put the required software.

# mkdir /root/protogeni

Step 2: Copy required software from host.

# scp root@192.168.57.1:/home/czhang/protogeni/software/testbed.tgz /root/protogeni
# scp root@192.168.57.1:/home/czhang/protogeni/software/FreeBSD-6.3-20080901.tar.gz /root/protogeni
# scp root@192.168.57.1:/home/czhang/protogeni/software/ports-20080901.tar.gz /root/protogeni

Step 3: Install perl and python on 'Boss'.

# cd /root/protogeni
# tar xvf FreeBSD-6.3-20080901.tar.gz //it produces folder 'packages'
# cd /root/protogeni/packages
# pkg_add perl-5.8.8_1.tbz
# ln -s /usr/local/bin/perl5.8.8 /usr/bin/perl
# pkg_add python25-2.5.1_1.tbz
# pkg_add rsync-2.6.9_2.tbz

Step 4: Prepare 'defs-bbn-emulab' file.

# mkdir -p /usr/testbed/src

# cd /usr/testbed/src
# tar xvf /root/protogeni/testbed.tgz

# scp root@192.168.57.4:/usr/testbed/src/testbed/defs-bbn-emulab /usr/testbed/src/testbed/   //copy from 'Ops'

My 'defs-bbn-emulab' wiki:BBN_Emulab:defs-bbn-emulab.

Tips: I choose to not enable ProtoGENI support when installing the regular Emulab. Enable it when installing ProtoGENI.

Step 5: Install 'boss' packages.

# cd /root/protogeni
# pkg_add emulab-boss-2.1.tbz

The output is here. wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt

Step 6: Run 'configure' and apply named patch.

# mkdir -p /usr/testbed/obj/testbed
# cd /usr/testbed/obj/testbed
# /usr/testbed/src/testbed/configure --with-TBDEFS=/usr/testbed/src/testbed/defs-bbn-emulab
# patch -p0 < /usr/testbed/src/testbed/named/rc.named.6.3.patch

The output is here. wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt

Step 7: Run the boss installation script.

# cd /usr/testbed/obj/testbed/install
# /usr/bin/perl boss-install -p /root/protogeni/packages

The output is here. wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt

At this point, we have a functional Emulab website up and runing, here is the screenshot,

Step 8: Deal with 'named' and 'dhcpd' configuration.

For 'named':

The configuration file is '/etc/namedb/named.conf' and other zone files in the same folder.
The command to restart it is '/etc/rc.d/named restart'.

During the last step, I encountered a error with named setup. By temporarily ignore that error, I redid the last step and this time I successfully installed it. Then I checked /etc/namedb and found named configuration file:'named.conf'(created by the boss installation script with two 'views', one is for the internal network and the other one for the external network), 'bbn.com.db.head' and 'bbn.com.internal.db.head'. As the Utah Emulab document said, run /usr/testbed/sbin/named_setup should produces two zone files 'bbn.com.db' and 'bbn.com.internal.db'. Then I ran it but it did not produce the zone files. So I manually edited those two files. The file is here.wiki:BBN_Emulab:bbn.com.db and wiki:BBN_Emulab:bbn.com.internal.db The named.conf is here. wiki:BBN_Emulab:named.conf

For 'dhcpd':

The configuration file is '/usr/local/etc/dhcpd.conf'
The way to modify it is: edit '/usr/local/etc/dhcpd.conf.template' first, then run '/usr/testbed/sbin/dhcpd_makeconf -i -r'
The command to start and stop it is '/usr/local/etc/rc.d/2.dhcpd.sh start/stop' 2.dhcpd.sh is a dhcpd wrapper provided by Utah

When you done with the installation, the first thing to do is to add a option 'deny unknown-clients' to the dhcpd.conf. Because your dhcpd may cause conflict with other existing dhcp servers.

Step 9: Try to install protogeni(Failed!!! at this point)

At this point, I did not go further with the testbed configuration including switch configuration and adding node to the testbed. I'd like to try if I could enable the protogeni support at this stage (Follow the wiki page https://www.protogeni.net/trac/protogeni/wiki/ProtoGENIInstall ).(However, this failed to install protogeni with the below error message:)

It seems caused by the fact that I did not customize the disk images and put related information into the database. So I plan to go ahead to do it.


C). Customize Memory Filesystem and continue to install protogeni (Do not "add node" and "customize the generic image" yet)

Reference:

http://users.emulab.net/trac/emulab/wiki/install/mfs.html

Step 1: Download tftpboot MFS tarball to host and copy to 'Boss'(I want to keep a copy on host for late use).

On Host:

# cd /home/czhang/protogeni/software
# wget http://www.emulab.net/downloads/tftpboot-latest.tar.gz

On 'Boss'

# scp root@192.168.57.1:/home/czhang/protogeni/software/tftpboot-latest.tar.gz /root/protogeni
# cd /tftpboot
# tar xvf /root/protogeni/tftpboot-latest.tar.gz

Step 2: Follow the 'README' extracted from tftpboot-latest.tar.gz to customize the mfs and add related information into database.

At this point, I tried to run 'sudo /usr/testbed/sbin/protogeni/initsite' again and was able to get federated with Utah Clearing House successfully!!

This is the output screenshot,



D). Add a experiment node and customize generic images

Reference:

http://users.emulab.net/trac/emulab/wiki/install/switches-db.html
http://users.emulab.net/trac/emulab/wiki/install/new_nodes.html
http://users.emulab.net/trac/emulab/wiki/install/customize_images.html

Step 1: Initial Switch Setup.

# mysql tbdb
mysql> insert into node_types set class='switch', type='cisco2950-ios';  //Just assume our NEC switches are Cisco 2950
mysql> insert into nodes set node_id='cisco1',phys_nodeid='cisco1',type='cisco2950-ios',role='ctrlswitch';
mysql> insert into nodes set node_id='cisco2',phys_nodeid='cisco2',type='cisco2950-ios',role='testswitch';
mysql> insert into switch_stack_types set stack_id='Control', stack_type='generic', supports_private=0, single_domain=0, min_vlan=20, max_vlan=64;
mysql> insert into switch_stack_types set stack_id='Experiment', stack_type='generic', supports_private=0, single_domain=0, min_vlan=20, max_vlan=64;
mysql> insert into switch_stacks values ('cisco1','Control',1),('cisco2','Experiment',1);

At this point, we need to configure a vlan as IP interface for control switch and assign IP 192.168.56.5 to it. Then put lines '192.168.56.5 cisco1' and '192.168.56.6 cisco2' in file Boss:/etc/hosts.

As of now, we could ping from boss to both switches.

Detailed cable connection figure will be provided here.

Step 2: Add a node.

Step 3: Customize generic images



  1. Build BBN Clearinghouse (Optional, not complete)

Reference: http://www.protogeni.net/trac/protogeni/wiki/InstallingAClearingHouse

Step 1: Create a folder in 'Clearinghouse' to put the required software.

# mkdir /root/protogeni

Step 2: Copy required software from host.

# scp root@192.168.57.1:/home/czhang/protogeni/software/testbed.tgz /root/protogeni
# scp root@192.168.57.1:/home/czhang/protogeni/software/FreeBSD-6.3-20080901.tar.gz /root/protogeni
# scp root@192.168.57.1:/home/czhang/protogeni/software/ports-20080901.tar.gz /root/protogeni  

Step 3: Install perl and python on 'Clearinghouse'

# cd /root/protogeni
# tar xvf FreeBSD-6.3-20080901.tar.gz //it produces folder 'packages'
# cd /root/protogeni/packages
# pkg_add perl-5.8.8_1.tbz
# ln -s /usr/local/bin/perl5.8.8 /usr/bin/perl
# pkg_add python25-2.5.1_1.tbz

Step 4: Prepare 'defs-clrhouse' file.

# mkdir -p /usr/testbed/src

# cd /usr/testbed/src
# tar xvf /root/protogeni/testbed.tgz




# cd /usr/testbed/src/testbed/protogeni/install
# cp defs-clrhouse defs-bbn-clrhouse
# vi defs-bbn-clrhouse

My 'defs-bbn-clrhouse' wiki:BBN_Emulab:defs-bbn-clrhouse.

Step 5: Do configure.

# mkdir -p /usr/testbed/obj/testbed
# cd /usr/testbed/obj/testbed
# /usr/testbed/src/testbed/configure --with-TBDEFS=/usr/testbed/src/testbed/protogeni/install/defs-clrhouse

The output is here. wiki:BBN_Emulab:BBN_Emulab_commandoutput.txt

Problem: Configure did not produce 'clrhouse-install' script as indicated in the document from Utah(Solved! Do 'cvs up' and re-configure).

Attachments (15)

Download all attachments as: .zip