Changes between Initial Version and Version 1 of GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopoQuagga/DesignSetup

Timestamp:

06/16/16 17:16:19 (8 years ago)

Author:

lnevers@bbn.com

Comment:

--

GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopoQuagga/DesignSetup

@@ +1 @@
+[[PageOutline]]
+
+'''[wiki:GENIExperimenter/Tutorials/SystematicExprCaseStudy#ATutorialonSystematicExperimentalDesign A Tutorial on Systematic Experimental Design]'''
+
+'''[wiki:GENIExperimenter/Tutorials/SystematicExprCaseStudy/InstallSoftware Step I: Single Node]'''
+
+'''[wiki:GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo Step II: Small Topology]'''
+
+ 
+{{{
+#!html
+
+<div style="text-align:center; width:495px; margin-left:auto; margin-right:auto;">
+<img id="Image-Maps_5201305222028436" src="http://groups.geni.net/geni/attachment/wiki/GENIExperimenter/Tutorials/Graphics/Setup.jpg?format=raw" usemap="#Image-Maps_5201305222028436" border="0" width="495" height="138" alt="" />
+<map id="_Image-Maps_5201305222028436" name="Image-Maps_5201305222028436">
+<area shape="rect" coords="18,18,135,110" href="http://groups.geni.net/geni/wiki/GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/DesignSetup" alt="" title=""    />
+<area shape="rect" coords="180,18,297,111" href="http://groups.geni.net/geni/wiki/GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/Execute" alt="" title=""    />
+<area shape="rect" coords="344,17,460,110" href="http://groups.geni.net/geni/wiki/GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/Finish" alt="" title=""    />
+<area shape="rect" coords="493,136,495,138" href="http://www.image-maps.com/index.php?aff=mapped_users_5201305222028436" alt="Image Map" title="Image Map" />
+</map>
+<!-- Image map text links - End - -->
+
+</div>
+}}}
+
+
+
+= 1. Software Configuration =
+
+Once we have installed the software on the VM, we may need to configure the software, and automate the configuration process for every node used in an experiment. For this tutorial, since we are setting a virtual routing environment, we will need to configure the routing protocol (e.g., OSPF protocol) for `XORP`. Again, let's start with the smallest reasonable topology, which is a four-router virtual topology as shown below. The reason that we chose the four-router topology is that we need at least one pair of routers that are not directly connected to each other and each router must have at least two virtual interfaces. 
+
+[[Image(GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/DesignSetup:4node-config.png, 50%)]]
+
+
+
+
+== a. Create Virtual Network ==
+
+As we have mentioned, a four-router topology is the smallest topology to configure and verify the `OSPF` routing. In this part, we illustrate how to create a virtual network with four virtual routers. 
+
+=== i. Create VMs and Virtual Links ===
+
+Create four VMs from the same InstaGENI aggregate and add the links. Modify each VM's name, set the Sliver Type to `emulab-xen`, and load the custom image by following the steps in the [wiki:GENIExperimenter/Tutorials/SystematicExprCaseStudy/InstallSoftware#a4.VerifyInstaGENICustomImage Load Custom Image] section. You may use the the value `https://www.instageni.clemson.edu/image_metadata.php?uuid=5417bcef-224e-11e4-aa5a-000000000000` in the `Disk Image` option for each router in this topology.
+
+[[Image(GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/DesignSetup:SmallTopo.jpg, 60%)]]
+
+The network configuration address assignment is given below:
+
+{{{
+ router-1 (.1) <-----> router-2 (.2) on 192.168.1.0/24
+ router-2 (.1) <-----> router-3 (.2) on 192.168.2.0/24
+ router-3 (.1) <-----> router-4 (.2) on 192.168.3.0/24
+ router-4 (.1) <-----> router-1 (.2) on 192.168.4.0/24
+ router-2 (.1) <-----> router-4 (.2) on 192.168.5.0/24
+}}}
+
+
+=== ii. Configure Virtual Interfaces ===
+
+Configure the virtual interfaces' IP addresses of each link as outlined in the network configuration address assignment list above:
+
+ [[Image(GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/DesignSetup:ConfigIF.jpg, 60%)]]
+
+ (1) Click on the link to be configured [[BR]]
+ (2) Enter Address for one interface [[BR]]
+ (3) Scroll down and enter address for the other interface part of the link [[BR]]
+ (4) Repeat for each of the list shown in the  network configuration address assignment list[[BR]]
+
+
+=== iii. Save the Current RSpec ===
+
+For version control purposes, it is important to save the resource request specification (RSpec) at each step. Thus, once we have defined the topology above, we select the ''' ''Download'' ''' button to save an intermediate version of your RSpec.
+
+[[Image(SaveRSpec.jpg,60%)]]
+
+{{{
+#!html
+<table border="0" cellpadding="0" cellspacing="0"><tr><td><img src="http://trac.gpolab.bbn.com/gcf/raw-attachment/wiki/Graphics/stock_notes.png" width="40" height="40" alt="Note"></td><td> The download command does not allow naming it simply uses the slice name to generate a RSpec file.  We have save a copy of this RSpec, which you may choose to download, available at <a href="http://www.gpolab.bbn.com/exp/sysexpr/rspec-for-wiki/4node-v1.rspec">here</a> </td></tr></table>
+  </li>
+</ol>
+</td>
+</tr>
+</table>
+}}}
+
+You may now reserve your resources by scrolling to the bottom of the `Add Resources` page ans clicking on the ''' ''Reserve Resources'' ''' button:
+
+[[Image(ReserveResources.jpg)]]
+
+
+== b. Manually Configure OSPF for XORP ==
+
+In this tutorial we configure each VM as a virtual router running the `OSPF` routing protocol.  Examples of `XORP` configurations can be found at [http://www.xorp.org/getting_started.html#configuring XORPExamples]. For each VM running as a software router, the `OSPF` configuration file will be named `ospfd.conf` and is stored in `/etc/xorp/`. We provide sample `OSPF` configuration file that can be downloaded to each router.
+
+Download the sample OSPF configuration to router-1: 
+{{{
+sudo mkdir /etc/xorp
+cd /etc/xorp
+sudo wget http://groups.geni.net/geni/raw-attachment/wiki/GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/DesignSetup/r1-ospf.conf 
+sudo mv r1-ospf.conf ospfd.conf
+}}}
+
+The interface information in the sample files may be different from the routers interfaces, you must check that the interface information is properly mapped in the `ospfd.conf` on each router before you start `xorp`.   
+
+There are several key configuration items in the `OSPF` configuration file for `router-1`. 
+
+__Interface Information:__
+{{{
+interface eth1 {
+    description: "virtual interface-eth1"
+    disable: false
+    discard: false
+    vif eth1 {
+        disable: false
+        address 192.168.4.2 {
+                 prefix-length: 24
+                 disable: false
+        }
+    }
+  }
+interface eth2 {
+......
+}
+}}} 
+
+__OSPF Metrics:__
+{{{
+protocols {
+    ospf4 {
+        router-id: 192.168.1.1
+        area 0.0.0.0 {
+            area-type:"normal"
+            interface eth1 {
+                vif eth1 {
+                  address 192.168.4.2 {
+                    interface-cost 10
+                    hello-interval 5
+                    router-dead-interval 10
+                    disable: false
+                  }
+                }
+            }
+            interface eth2 {
+            ......
+            }
+      }
+   }
+}
+}}}
+
+To simplify the configuration process, we assume the `interface-cost` and the `hello-interval` are both set to 10 seconds for every virtual interface. The main information needed from each node is the virtual interface's `address` and its `prefix-length`, as well as the router's `router-id` that is required for the `protocol` section in the configuration file. 
+
+Now download the configuration on the remaining routers and verify that the interfaces addresses are properly mapped in the `ospfd.conf` file. 
+
+Download the sample OSPF configuration to router-2 and check interfaces address mappings: 
+{{{
+sudo mkdir /etc/xorp
+cd /etc/xorp
+sudo wget http://groups.geni.net/geni/raw-attachment/wiki/GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/DesignSetup/r2-ospf.conf 
+sudo mv r2-ospf.conf ospfd.conf
+}}}
+Download the sample OSPF configuration to router-3 and check interfaces address mappings: 
+{{{
+sudo mkdir /etc/xorp
+cd /etc/xorp
+sudo wget http://groups.geni.net/geni/raw-attachment/wiki/GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/DesignSetup/r3-ospf.conf 
+sudo mv r3-ospf.conf ospfd.conf
+}}}
+
+Download the sample OSPF configuration to router-4 and check interfaces address mappings: 
+{{{
+sudo mkdir /etc/xorp
+cd /etc/xorp
+sudo wget http://groups.geni.net/geni/raw-attachment/wiki/GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/DesignSetup/r4-ospf.conf 
+sudo mv r4-ospf.conf ospfd.conf
+}}}
+
+
+== c. Start `XORP` and Verify Routing ==
+
+=== i. Start `XORP` Process ===
+
+First, we must add the `xorp` user group. Login to each router and run the following command:
+{{{
+sudo groupadd xorp
+}}}
+Start `XORP` by specifying the routing protocol configuration file. 
+{{{
+lnevers@router-1:/etc/xorp$ cd /usr/local/xorp/sbin/
+lnevers@router-1:/usr/local/xorp/sbin$ sudo ./xorp_rtrmgr -b /etc/xorp/ospfd.conf -l /tmp/xorp_rtrmgr_log -d
+Not doing pidfile...
+lnevers@router-1:/usr/local/xorp/sbin$ 
+}}}
+
+Verify the `XORP` process is running:
+{{{
+lnevers@router-1:/usr/local/xorp/sbin$ ps -ef | grep xorp
+root      9658     1  0 06:06 pts/0    00:00:00 xorp_fea
+root      9659     1  0 06:06 pts/0    00:00:00 xorp_rib
+root      9660     1  0 06:06 pts/0    00:00:00 xorp_policy
+root      9661     1  0 06:06 pts/0    00:00:00 xorp_ospfv2
+root      9662     1  0 06:06 ?        00:00:00 ./xorp_rtrmgr -b /etc/xorp/ospfd.conf -l /tmp/xorp_rtrmgr_log -d
+lnevers@router-1:/usr/local/xorp/sbin$ 
+}}}
+
+Now you have routing running on all four router! 
+
+If you need to stop `XORP`, use the kill command as shown below to stop each the 5 xorp processes:
+
+{{{
+ps -ef | grep xorp_ | /usr/bin/awk '{ if ( $1 == "root" ) {print "sudo kill -9 " $2}}' | sh
+}}}
+
+=== ii. Verify Routing ===
+
+Login to `router-1` and view the `OSPF` routing table:
+
+{{{
+xuanliu@router-1:/usr/local/xorp/sbin$ sudo ./xorpsh
+Welcome to XORP v1.8.5 on router-1.xuan-wiki-prep.ch-geni-net.instageni.stanford.edu
+Version tag: 00000000  Build Date: 2014-06-05 17:11 64-bit
+root@router-1.xuan-wiki-prep.ch-geni-net.instageni.stanford.edu> show route table ipv4 unicast ospf
+192.168.2.0/24  [ospf(110)/20]
+                > to 192.168.1.2 via eth2/eth2
+192.168.3.0/24  [ospf(110)/20]
+                > to 192.168.4.1 via eth1/eth1
+192.168.5.0/24  [ospf(110)/20]
+                > to 192.168.1.2 via eth2/eth2
+}}}
+
+
+Login to `router-3` and view the `OSPF` routing table:
+{{{
+xuanliu@router-3:/usr/local/xorp/sbin$ sudo ./xorpsh
+Welcome to XORP v1.8.5 on router-3.xuan-wiki-prep.ch-geni-net.instageni.stanford.edu
+Version tag: 00000000  Build Date: 2014-06-05 17:11 64-bit
+root@router-3.xuan-wiki-prep.ch-geni-net.instageni.stanford.edu> show route table ipv4 unicast ospf
+192.168.1.0/24  [ospf(110)/20]
+                > to 192.168.2.1 via eth2/eth2
+192.168.4.0/24  [ospf(110)/20]
+                > to 192.168.3.1 via eth1/eth1
+192.168.5.0/24  [ospf(110)/20]
+                > to 192.168.2.1 via eth2/eth2
+root@router-3.xuan-wiki-prep.ch-geni-net.instageni.stanford.edu>
+}}}
+
+Since `router-1` and `router-3` are not directly connected, by `OSPF` routing, we should be able to run `ping` and `traceroute` between them. 
+
+{{{
+xuanliu@router-3:/usr/local/xorp/sbin$ ping 192.168.1.1
+PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data.
+64 bytes from 192.168.1.1: icmp_req=1 ttl=63 time=2.15 ms
+64 bytes from 192.168.1.1: icmp_req=2 ttl=63 time=1.45 ms
+64 bytes from 192.168.1.1: icmp_req=3 ttl=63 time=1.41 ms
+^C
+--- 192.168.1.1 ping statistics ---
+3 packets transmitted, 3 received, 0% packet loss, time 2003ms
+rtt min/avg/max/mdev = 1.411/1.673/2.152/0.342 ms
+xuanliu@router-3:/usr/local/xorp/sbin$ traceroute 192.168.1.1
+traceroute to 192.168.1.1 (192.168.1.1), 30 hops max, 60 byte packets
+ 1  router-2-lan1 (192.168.2.1)  0.652 ms  0.595 ms  0.563 ms
+ 2  router-1-lan0 (192.168.1.1)  1.146 ms  1.106 ms  1.076 ms
+xuanliu@router-3:/usr/local/xorp/sbin$
+}}}
+
+== d. Automate `OSPF` Configuration ==
+
+By understanding the syntax of the `OSPF` configuration file for `XORP`, we are able to automate the `OSPF` configuration for every virtual router in a topology, assuming these virtual routers are identical. You can download the automation scripts from [http://www.gpolab.bbn.com/exp/sysexpr/xorp_autostart.tar.gz XORPAutoStart]. 
+
+=== i. Test the Automation ===
+
+To test it, you will need to login to each router (e.g., `router-3`) and run the `start-xorp.sh`. You will see the output similar to the following:
+
+{{{
+dhcp89-69-127:~ xliu$ ssh -p 30781  -i /Users/xliu/.ssh/id_rsa xuanliu@pc3.instageni.stanford.edu
+Welcome to Ubuntu 12.04 LTS (GNU/Linux 3.2.0-56-generic x86_64)
+
+ * Documentation:  https://help.ubuntu.com/
+Last login: Mon Jun 30 14:56:51 2014 from 128.89.69.127
+xuanliu@router-3:~$ cd /local
+xuanliu@router-3:/local$ sudo wget http://www.gpolab.bbn.com/exp/sysexpr/xorp_autostart.tar.gz
+xuanliu@router-3:/local$ sudo tar -zxvf xorp_autostart.tar.gz
+xuanliu@router-3:/local/xorp_autostart$ /bin/bash start-xorp.sh
+router-3.xuan-wiki-prep.ch-geni-net.instageni.stanford.edu
+2014-07-01 06:56:28 AM
+XORP dir exist
+xorp is already added to the group
+xorp is running, stop it first
+XORP is starting ......
+Not doing pidfile...
+}}}
+
+The key procedure is to obtain the virtual interface's `address` and its `prefix` of the subnet information. 
+
+In the install script `start-xorp.sh`, the Following components are the key parts to generate `ospfd.conf`. 
+
+ * Get Hostname and virtual interface information, and store it into a file (`vm_info.txt`)
+{{{
+hostname | sudo tee vm_info.txt > /dev/null
+/sbin/ifconfig | egrep 'eth|inet addr' | sudo tee -a vm_info.txt > /dev/null
+}}}  
+  A sample `vm_info.txt` is available at [http://groups.geni.net/geni/attachment/wiki/GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/DesignSetup/vm_info.txt.example SampleVMInfo]
+
+ * Create `ospfd.conf` by running an `awk` script, available from [http://groups.geni.net/geni/raw-attachment/wiki/GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/DesignSetup/ospfd-conf-gen.awk here]
+{{{
+sudo /usr/bin/awk -f $ABS_PATH/ospfd-conf-gen.awk $vm_info_file "$timestamp" 24 | sudo tee $ABS_PATH/ospfd.conf > /dev/null
+}}}
+  A sample `ospfd.conf` is available at [http://groups.geni.net/geni/attachment/wiki/GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/DesignSetup/ospfd.conf.example SampleOSPFConf]
+ 
+ * Start `XORP` Process
+{{{
+$sudo ./xorp_rtrmgr -b /etc/xorp/ospfd.conf -l /tmp/xorp_rtrmgr_log -d
+}}}
+ 
+The complete shell script to wrap up the above procedures is available at [http://groups.geni.net/geni/raw-attachment/wiki/GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/DesignSetup/start-xorp.sh StartXORP]
+
+
+{{{
+#!html
+<table border="0" cellpadding="0" cellspacing="0"><tr><td><img src="http://trac.gpolab.bbn.com/gcf/raw-attachment/wiki/Graphics/stock_notes.png" width="40" height="40" alt="Note"></td><td> Please go to <a href="http://groups.geni.net/geni/wiki/HowTo/WriteInstallScript"> HowToWriteInstallScript</a> for details on how to write an install script. </td></tr></table>
+  </li></ol></td></tr></table>
+}}}
+
+Next we are going to install the automate script as an install script at the postboot stage. 
+
+{{{
+#!html
+<table border="0" cellpadding="0" cellspacing="0"><tr><td><img src="http://groups.geni.net/geni/attachment/wiki/GENIExperimenter/Tutorials/Graphics/atention_92.jpg?format=raw" width='40' height='40'/></td><td>If you plan to use the same slice to do the next step, please delete your existing resources in your slice at this time, otherwise you can create a new slice to continue.</td></tr></table>
+  </li>
+</ol>
+</td>
+</tr>
+</table>
+}}}
+
+== e. Install Script ==
+
+Launch Flack for your slice, and import the RSpec `4node-v1.rspec` we just saved. 
+
+Click '''Import''' and select '''Import from file''' from the pull-down list, then select `4node-v1.rspec` from the location where you have just saved to, and confirm it. You will see the four-node topology on your canvas, and each node has been configured to have the custom image. 
+
+[[Image(GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/DesignSetup:import-rspec-from-file.png)]]
+
+Now, we are ready to add an install script by editing the node property. For each node, click [[Image(GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/DesignSetup:info.jpg)]] to see the node information. By clicking [[Image(GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/DesignSetup:add-service-btn.png)]], we are able to add a service that usually refers to an install script. In this tutorial, we will add an install script called `xorp_autostart.tar.gz`, by entering the link ''http://www.gpolab.bbn.com/exp/sysexpr/xorp_autostart.tar.gz'', and the install path is set as ''/local''. By clicking [[Image(GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/DesignSetup:add-exe-serv-btn.png)]], we can add the commend that execute the install script. In the execute box, we enter `/bin/bash /local/xorp_autostart/start-xorp.sh`, and the shell type is `sh`.
+
+[[Image(GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/DesignSetup:node-install-service.png)]]
+
+Once we added the service and the execute service in Flack, apply the changes and submit the VM reservation request, the OSPF configuration file will be automatically created and the XORP will be started at `router-1` once it is initialized. 
+
+Repeat the same procedures for the other three routers (i.e., `router-2`, `router-3`, `router-4`), and then '''submit''' the request. When all resources are ready, '''Save''' the RSpec as `4node-v2.rspec` at this stage, we will load this RSpec in the next step to get the same setup. (You can download a sample RSpec for `4node-v2.rspec` at [http://www.gpolab.bbn.com/exp/sysexpr/rspec-for-wiki/4node-v2.rspec here])
+
+Setting up the install script will automate installation or configuration process at every node's postboot stage, and we do not need to login to every node to run `start-xorp.sh` manually. 
+
+= 2. Impact to Request RSpec =
+
+Whenever we make a change to node information or link information in Flack, the user request RSpec is modified accordingly. In other words, we can rely on Flack to easily modify the request RSpec.
+
+Now let's take a look at how the steps [wiki:GENIExperimenter/Tutorials/SystematicExprCaseStudy/InstallSoftware#a4.VerifyInstaGENICustomImage Load a Custom Image] and [wiki:GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/DesignSetup#e.InstallScript Adding Install Scripts] will modify the request RSpec. For instance, before adding the custom image and install scripts, the RSpec describing `router-1` was like:
+
+{{{
+<node client_id="router-1" component_manager_id="urn:publicid:IDN+instageni.stanford.edu+authority+cm" component_id="urn:publicid:IDN+instageni.stanford.edu+node+pc3" component_name="pc3" exclusive="false">
+    <vnode name="pcvm3-3" xmlns="http://www.protogeni.net/resources/rspec/ext/emulab/1"/>
+    <location latitude="37.43" longitude="-122.17" country="US"/>
+    <interface client_id="xen-0:if0">
+      <ip address="192.168.1.1" netmask="255.255.255.0" type="ipv4"/>
+      <flack:interface_info addressBound="true" bound="false"/>
+    </interface>
+    <interface client_id="xen-0:if1">
+      <ip address="192.168.4.2" netmask="255.255.255.0" type="ipv4"/>
+      <flack:interface_info addressBound="true" bound="false"/>
+    </interface>
+    <flack:node_info x="150" y="407" unbound="true"/>
+  </node>
+}}}
+
+First, when we added the custom image (see [wiki:GENIExperimenter/Tutorials/SystematicExprCaseStudy/InstallSoftware#a4.VerifyInstaGENICustomImage Load a Custom Image]), a new tag `<sliver>` will be added to `router-1`'s RSpec, which is 
+
+{{{
+  <sliver_type name="emulab-xen">
+      <disk_image url="https://www.instageni.clemson.edu/image_metadata.php?uuid=5417bcef-224e-11e4-aa5a-000000000000"/>
+  </sliver_type>
+}}}
+
+
+Secondly, when we added the install scripts (see [wiki:GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/DesignSetup#e.InstallScript Adding Install Scripts]), a new tag `<service>` will be added to `router-1`'s RSpec, which is
+{{{
+   <services>
+      <execute command="/bin/bash /local/xorp_autostart/start-xorp.sh" shell="sh"/>
+      <install install_path="/local" url="http://www.gpolab.bbn.com/exp/sysexpr/xorp_autostart.tar.gz"/>
+   </services>
+}}}
+
+This indicates that we can add custom image and postboot scripts by directly modifying the request RSpec, this is helpful when we have a larger topology.
+
+
+
+
+= 3. Script to Capture Routing Table Periodically =
+
+
+In this tutorial, one metric we would like to evaluate is how often the routing table will be updated when there is a link/node failure. To emulate a link failure, we can disable the virtual interfaces, and to emulate a virtual router failure, we can stop the XORP process at the virtual router. A short script (downloaded from [http://www.gpolab.bbn.com/exp/sysexpr/xorp_run.tar.gz here] is available to be loaded as an install script. 
+
+Delete any existing resources or create a new slice, '''LOAD''' the RSpec `4node-v2.rspec`, and add a new install service to every node in Flack. Now, the service portion for each node becomes to 
+{{{
+    <services>
+      <execute command="/bin/bash /local/xorp_autostart/start-xorp.sh" shell="sh"/>
+      <install install_path="/local" url="http://www.gpolab.bbn.com/exp/sysexpr/xorp_autostart.tar.gz"/>
+      <install install_path="/local" url="http://www.gpolab.bbn.com/exp/sysexpr/xorp_run.tar.gz"/>
+    </services>
+}}}
+
+After adding the new install script to every node, send the resource reservation request. When all nodes are ready to login,  we save the new modified RSpec as `4node-v3.rspec`. (You can download a sample RSpec for `4node-v3.rspec` at [http://www.gpolab.bbn.com/exp/sysexpr/rspec-for-wiki/4node-v3.rspec here])
+
+In the next step, we will add two end hosts to current four-node topology, so now we can delete existing resources from the current slice. 
+
+= 4. Add End Hosts and Install `iperf` =
+
+
+
+Once we have verified the routing functions for the virtual router, we add end hosts to validate the end-to-end communication.  As the figure shows below, we create the same four-node topology by loading `4node-v3.rspec` in Flack, and create two more VMs as a client and a server, and add two virtual links.  The links between end-hosts and edge routers are configured as below:
+{{{
+client ----- router-1: 192.168.10.0/24
+server ----- router-3: 192.168.20.0/24
+client: 192.168.10.11
+server: 192.168.20.10
+}}}
+
+
+
+For client and server,  we select regular `Ubuntu 12.04` OS in the node information, so we need to install `iperf` separately. In this example, there are only two hosts need iperf installed, so one option is to login to each host, and manually install it by running
+
+{{{
+sudo apt-get install iperf
+}}}
+
+
+Or, we can also write a script to install `iperf` to the client and server, load it as an install script by following the same steps as we just did for XORP. The install scripts for client and server can be downloaded from [http://www.gpolab.bbn.com/exp/sysexpr/install_script.tar.gz here]. An sample RSpec with iperf install script and execute command can be downloaded at [http://www.gpolab.bbn.com/exp/sysexpr/rspec-for-wiki/4node-v4.rspec `4node-v4.rspec`]. 
+
+[[Image(GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/DesignSetup:4node-add-endhosts.png)]]
+
+'''SAVE''' the new RSpec to `4node-v4.rspec`, when all nodes have been reserved. 
+
+
+
+'''[wiki:GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo Small Topology: Introduction]'''
+
+'''[wiki:GENIExperimenter/Tutorials/SystematicExprCaseStudy/SmallTopo/Execute Small Topology: Execute]'''