1 | |
---|
2 | OSPF experiment using Quagga software and virtual network of OpenVZ nodes + EGRE tunnels. |
---|
3 | |
---|
4 | 1. Create an emulab experiment using "ospf_setup.ns" setup file. |
---|
5 | 2. When Emulab swaps in experiments, it leaves lots of free space on the HDD of the created nodes. |
---|
6 | Create partitions on this free space to store virtual machines. |
---|
7 | To do so, use following commands on all the machines. |
---|
8 | a. parted |
---|
9 | b. print free |
---|
10 | (Notice the last line it prints the free space. For e.g. a run of this on an emulab machine is) |
---|
11 | -------------------------------------------------------------------------------------------------------- |
---|
12 | (parted) print free |
---|
13 | Model: SEAGATE ST3146707LC (scsi) |
---|
14 | Disk /dev/sda: 147GB |
---|
15 | Sector size (logical/physical): 512B/512B |
---|
16 | Partition Table: msdos |
---|
17 | |
---|
18 | Number Start End Size Type File system Flags |
---|
19 | 1 32.3kB 3216MB 3216MB primary sun-ufs |
---|
20 | 2 3216MB 6432MB 3216MB primary ext3 boot |
---|
21 | 3 6432MB 6564MB 132MB primary linux-swap |
---|
22 | 6564MB 147GB 140GB Free Space |
---|
23 | -------------------------------------------------------------------------------------------------------- |
---|
24 | |
---|
25 | We will create new partition from 6564MB till the end of the disk. |
---|
26 | |
---|
27 | c. mkpart |
---|
28 | (answer the questions as shown below) |
---|
29 | -------------------------------------------------------------------------------------------------------- |
---|
30 | (parted) mkpart |
---|
31 | Partition type? primary/extended? extended |
---|
32 | Start? 6564MB |
---|
33 | End? -1 |
---|
34 | -------------------------------------------------------------------------------------------------------- |
---|
35 | Please note that -1 denotes end of the disk for parted. |
---|
36 | |
---|
37 | d. mkpart (Yes...once again...although answer change this time as shown below) |
---|
38 | -------------------------------------------------------------------------------------------------------- |
---|
39 | (parted) mkpart |
---|
40 | Partition type? [logical]? |
---|
41 | File system type? [ext2]? |
---|
42 | Start? 6564MB |
---|
43 | End? -1 |
---|
44 | (parted) |
---|
45 | -------------------------------------------------------------------------------------------------------- |
---|
46 | e. quit |
---|
47 | f. fdisk -l -u |
---|
48 | Output would be something like below |
---|
49 | -------------------------------------------------------------------------------------------------------- |
---|
50 | [root@node0 yogesh]# fdisk -l -u |
---|
51 | |
---|
52 | Disk /dev/sda: 146.8 GB, 146815733760 bytes |
---|
53 | 255 heads, 63 sectors/track, 17849 cylinders, total 286749480 sectors |
---|
54 | Units = sectors of 1 * 512 = 512 bytes |
---|
55 | |
---|
56 | Device Boot Start End Blocks Id System |
---|
57 | /dev/sda1 63 6281414 3140676 a5 FreeBSD |
---|
58 | /dev/sda2 * 6281415 12562829 3140707+ 83 Linux |
---|
59 | /dev/sda3 12562830 12819869 128520 82 Linux swap / Solaris |
---|
60 | /dev/sda4 12819870 286749479 136964805 f W95 Ext'd (LBA) |
---|
61 | /dev/sda5 12819933 286749479 136964773+ 83 Linux |
---|
62 | |
---|
63 | Disk /dev/sdb: 146.8 GB, 146815733760 bytes |
---|
64 | 255 heads, 63 sectors/track, 17849 cylinders, total 286749480 sectors |
---|
65 | Units = sectors of 1 * 512 = 512 bytes |
---|
66 | |
---|
67 | Device Boot Start End Blocks Id System |
---|
68 | /dev/sdb1 63 286744184 143372061 83 Linux |
---|
69 | [root@node0 yogesh]# |
---|
70 | -------------------------------------------------------------------------------------------------------- |
---|
71 | g. Formate the new partition in ext3 FS. Use command "mkfs -t ext3 new_partition" |
---|
72 | For e.g. in this case: |
---|
73 | [root@node0 yogesh]# mkfs -t ext3 /dev/sda5 |
---|
74 | h. Create a new directory to mount this partition "mkdir /vz" |
---|
75 | i. Edit /etc/fstab so that this directory will be mounted automatically. |
---|
76 | for example, in this case add following command to /etc/fstab at the end |
---|
77 | |
---|
78 | /dev/sda5 /vz ext3 defaults 0 0 |
---|
79 | |
---|
80 | j. Mount the new partition using 'mount -a' |
---|
81 | |
---|
82 | 3. Now install the OpenVZ kernel with EGRE module. Execute following on all four nodes. |
---|
83 | a. Copy the source code of the kernel to /vz/ using following command. |
---|
84 | git clone git://143.215.129.52/home/yogesh/git_repository/vn-with-openvz |
---|
85 | Enter the kernel directory. |
---|
86 | b. Compile the kernel using "time make" |
---|
87 | c. Compile modules using "time make modules" |
---|
88 | d. Install modules using "time make modules_install" |
---|
89 | e. Install kernel using "make install" |
---|
90 | f. To boot using newly installed kernel, modify the "default" entry to new kernel in the file "/boot/grub/grub.conf" |
---|
91 | (Usually this means changing the "default to 0) |
---|
92 | g Please make sure the following steps are performed before rebooting into OpenVZ kernel. |
---|
93 | |
---|
94 | sysctl |
---|
95 | There are a number of kernel parameters that should be set for OpenVZ to work correctly. These parameters are stored in /etc/sysctl.conf file. Here are the relevant portions of the file; please edit accordingly. |
---|
96 | -------------------------------------------------------------------------------------------------------- |
---|
97 | # On Hardware Node we generally need |
---|
98 | # packet forwarding enabled and proxy arp disabled |
---|
99 | net.ipv4.ip_forward = 1 |
---|
100 | net.ipv4.conf.default.proxy_arp = 0 |
---|
101 | |
---|
102 | # Enables source route verification |
---|
103 | net.ipv4.conf.all.rp_filter = 1 |
---|
104 | |
---|
105 | # Enables the magic-sysrq key |
---|
106 | kernel.sysrq = 1 |
---|
107 | |
---|
108 | # We do not want all our interfaces to send redirects |
---|
109 | net.ipv4.conf.default.send_redirects = 1 |
---|
110 | net.ipv4.conf.all.send_redirects = 0 |
---|
111 | -------------------------------------------------------------------------------------------------------- |
---|
112 | |
---|
113 | SELinux |
---|
114 | SELinux should be disabled. To that effect, put the following line to /etc/sysconfig/selinux: |
---|
115 | -------------------------------------------------------------------------------------------------------- |
---|
116 | SELINUX=disabled |
---|
117 | -------------------------------------------------------------------------------------------------------- |
---|
118 | h. reboot |
---|
119 | i. Verify that you are running the new kernel using "uname -a" |
---|
120 | |
---|
121 | 4. Install utilities on all the machines by executing following |
---|
122 | a. cd /etc/yum.repos.d && wget http://download.openvz.org/openvz.repo && rpm --import http://download.openvz.org/RPM-GPG-Key-OpenVZ && cd |
---|
123 | b. yum -y install vzctl vzquota bridge-utils |
---|
124 | c. service vz start |
---|
125 | d. yum -y install vzpkg vzyum vzrpm43-python vzrpm44-python vzctl-lib |
---|
126 | e. yum -y install vztmpl-fedora-core-5 |
---|
127 | f. wget http://download.openvz.org/template/precreated/fedora-core-5-i386-minimal.tar.gz |
---|
128 | g. mv fedora-core-5-i386-minimal.tar.gz /vz/template/cache/fedora-core-5-i386-minimal.tar.gz |
---|
129 | h. modprobe gre |
---|
130 | i. In the checked out directory, there is the iproute source package which is required to be installed to create the |
---|
131 | gre tunnels. Compile it using "make". After compilation enter directory ip by "cd ip/" |
---|
132 | |
---|
133 | Setting up the Virtual network of OpenVZ nodes |
---|
134 | ------------------------------------------------------------------------------------- |
---|
135 | |
---|
136 | Execute following on all four nodes |
---|
137 | |
---|
138 | 1. vzctl create 101 --ostemplate fedora-core-5-minimal |
---|
139 | 2. for CAP in net_admin net_raw sys_admin; do vzctl set 101 --capability ${CAP}:on --save ; done |
---|
140 | 3. vzctl start 101 |
---|
141 | 4. vzyum 101 -y install tcpdump quagga emacs telnet |
---|
142 | |
---|
143 | Node0 - physical machine with 4 interfaces i.e. machine connected to all the others |
---|
144 | ------------------------------------------------------------------------------------- |
---|
145 | Execute following commands |
---|
146 | |
---|
147 | 1. vzctl set 101 --netif_add eth0,00:43:34:56:78:0A,veth101.0,00:43:34:56:88:0B --save |
---|
148 | 2. vzctl set 101 --netif_add eth1,00:43:34:56:78:1A,veth101.1,00:43:34:56:88:1B --save |
---|
149 | 3. vzctl set 101 --netif_add eth2,00:43:34:56:78:2A,veth101.2,00:43:34:56:88:2B --save |
---|
150 | 4. vzctl enter 101 |
---|
151 | 5. ifconfig eth0 13.0.0.59/24 |
---|
152 | 6. ifconfig eth1 11.0.0.53/24 |
---|
153 | 7. ifconfig eth2 7.0.0.37/24 |
---|
154 | 8. ifconfig lo up |
---|
155 | 4. vzctl exit 101 |
---|
156 | 9. brctl addbr br0 |
---|
157 | 10. brctl addbr br1 |
---|
158 | 11. brctl addbr br2 |
---|
159 | 12. brctl addif br0 veth101.0 |
---|
160 | 13. brctl addif br1 veth101.1 |
---|
161 | 14. brctl addif br2 veth101.2 |
---|
162 | 15. cd /users/yogesh/iproute-2.6.20-070313/ip/ |
---|
163 | 16. ./ip tunnel add egre0 mode gre type eth local 10.1.2.2 remote 10.1.2.3 key 5 |
---|
164 | 17. ./ip tunnel add egre1 mode gre type eth local 10.1.3.2 remote 10.1.3.3 key 6 |
---|
165 | 18. ./ip tunnel add egre2 mode gre type eth local 10.1.1.2 remote 10.1.1.3 key 7 |
---|
166 | 19. brctl addif br0 egre0 |
---|
167 | 20. brctl addif br1 egre1 |
---|
168 | 21. brctl addif br2 egre2 |
---|
169 | 22. ./ip link set egre0 up |
---|
170 | 23. ./ip link set egre1 up |
---|
171 | 24. ./ip link set egre2 up |
---|
172 | 25. ifconfig br0 up |
---|
173 | 26. ifconfig br1 up |
---|
174 | 27. ifconfig br2 up |
---|
175 | |
---|
176 | Node1 |
---|
177 | ----- |
---|
178 | |
---|
179 | 1. vzctl set 101 --netif_add eth0,00:47:34:56:78:0A,veth101.0,00:47:34:56:88:0B --save |
---|
180 | 2. vzctl enter 101 |
---|
181 | 3. ifconfig eth0 13.0.0.57/24 |
---|
182 | 4. ifconfig lo up |
---|
183 | 4. vzctl exit 101 |
---|
184 | 5. brctl addbr br0 |
---|
185 | 5. brctl addif br0 veth101.0 |
---|
186 | 6. cd /users/yogesh/iproute-2.6.20-070313/ip/ |
---|
187 | 7. ./ip tunnel add egre0 mode gre type eth local 10.1.2.3 remote 10.1.2.2 key 5 |
---|
188 | 8. brctl addif br0 egre0 |
---|
189 | 9. ./ip link set egre0 up |
---|
190 | 10. ifconfig br0 up |
---|
191 | |
---|
192 | Node2 |
---|
193 | ----- |
---|
194 | |
---|
195 | 1. vzctl set 101 --netif_add eth0,00:53:34:56:78:0A,veth101.0,00:53:34:56:88:0B --save |
---|
196 | 2. vzctl enter 101 |
---|
197 | 3. ifconfig eth0 11.0.0.47/24 |
---|
198 | 4. ifconfig lo up |
---|
199 | 4. vzctl exit 101 |
---|
200 | 9. brctl addbr br0 |
---|
201 | 12. brctl addif br0 veth101.0 |
---|
202 | 15. cd /users/yogesh/iproute-2.6.20-070313/ip/ |
---|
203 | 17. ./ip tunnel add egre0 mode gre type eth local 10.1.3.3 remote 10.1.3.2 key 6 |
---|
204 | 19. brctl addif br0 egre0 |
---|
205 | 9. ./ip link set egre0 up |
---|
206 | 10. ifconfig br0 up |
---|
207 | |
---|
208 | |
---|
209 | Node3 |
---|
210 | ------ |
---|
211 | |
---|
212 | 1. vzctl set 101 --netif_add eth0,00:53:34:56:78:0A,veth101.0,00:53:34:56:88:0B --save |
---|
213 | 2. vzctl enter 101 |
---|
214 | 3. ifconfig eth0 7.0.0.31/24 |
---|
215 | 4. ifconfig lo up |
---|
216 | 4. exit 101 |
---|
217 | 9. brctl addbr br0 |
---|
218 | 12. brctl addif br0 veth101.0 |
---|
219 | 15. cd /users/yogesh/iproute-2.6.20-070313/ip/ |
---|
220 | 18. ./ip tunnel add egre0 mode gre type eth local 10.1.1.3 remote 10.1.1.2 key 7 |
---|
221 | 19. brctl addif br0 egre0 |
---|
222 | 9. ./ip link set egre0 up |
---|
223 | 10. ifconfig br0 up |
---|
224 | |
---|
225 | Now on all the nodes execute following |
---|
226 | ----------------------------------------- |
---|
227 | |
---|
228 | 0. vzyum 101 install quagga |
---|
229 | 1. vzctl enter 101 |
---|
230 | 2. Add like "password zebra" to file /etc/quagga/zebra.conf |
---|
231 | 3. service zebra start |
---|
232 | 4. add line "127.0.0.1 localhost" to file /etc/hosts |
---|
233 | 5. touch /etc/sysconfig/network |
---|
234 | 6. mv /etc/quagga/ospfd.conf.sample /etc/quagga/ospfd.conf |
---|
235 | 7. service ospfd start |
---|
236 | |
---|
237 | Node0: |
---|
238 | ------ |
---|
239 | |
---|
240 | 5. enter zebra "telnet localhost 2601" (password is zebra as set above) |
---|
241 | 6. enable |
---|
242 | 7. configure terminal |
---|
243 | 8. interface eth0 |
---|
244 | 9. ip address 13.0.0.59/24 |
---|
245 | 10. exit |
---|
246 | 11. interface eth1 |
---|
247 | 12. ip address 11.0.0.53/24 |
---|
248 | 13. exit |
---|
249 | 14. interface eth2 |
---|
250 | 15. ip address 7.0.0.37/24 |
---|
251 | 16. exit |
---|
252 | 17. write memory |
---|
253 | 18. exit |
---|
254 | 19. exit |
---|
255 | 20. telnet localhost 2604 |
---|
256 | 21. enable |
---|
257 | 22. configure terminal |
---|
258 | 23. router ospf |
---|
259 | 24. network 13.0.0.0/24 area 13 |
---|
260 | 25. network 11.0.0.0/24 area 11 |
---|
261 | 26. network 7.0.0.0/24 area 7 |
---|
262 | 27. write memory |
---|
263 | |
---|
264 | Node1: |
---|
265 | ------ |
---|
266 | |
---|
267 | 5. enter zebra "telnet localhost 2601" |
---|
268 | 6. enable |
---|
269 | 7. configure terminal |
---|
270 | 8. interface eth0 |
---|
271 | 9. ip address 13.0.0.57/24 |
---|
272 | 10. exit |
---|
273 | 17. write memory |
---|
274 | 18. exit |
---|
275 | 19. exit |
---|
276 | 20. telnet localhost 2604 |
---|
277 | 21. enable |
---|
278 | 22. configure terminal |
---|
279 | 23. router ospf |
---|
280 | 24. network 13.0.0.0/24 area 13 |
---|
281 | 25. write memory |
---|
282 | |
---|
283 | Node2: |
---|
284 | ------ |
---|
285 | |
---|
286 | 5. enter zebra "telnet localhost 2601" |
---|
287 | 6. enable |
---|
288 | 7. configure terminal |
---|
289 | 8. interface eth0 |
---|
290 | 9. ip address 11.0.0.47/24 |
---|
291 | 10. exit |
---|
292 | 17. write memory |
---|
293 | 18. exit |
---|
294 | 19. exit |
---|
295 | 20. telnet localhost 2604 |
---|
296 | 21. enable |
---|
297 | 22. configure terminal |
---|
298 | 23. router ospf |
---|
299 | 24. network 11.0.0.0/24 area 11 |
---|
300 | 25. write memory |
---|
301 | |
---|
302 | Node3: |
---|
303 | ------ |
---|
304 | |
---|
305 | 5. enter zebra "telnet localhost 2601" |
---|
306 | 6. enable |
---|
307 | 7. configure terminal |
---|
308 | 8. interface eth0 |
---|
309 | 9. ip address 7.0.0.31/24 |
---|
310 | 10. exit |
---|
311 | 17. write memory |
---|
312 | 18. exit |
---|
313 | 19. exit |
---|
314 | 20. telnet localhost 2604 |
---|
315 | 21. enable |
---|
316 | 22. configure terminal |
---|
317 | 23. router ospf |
---|
318 | 24. network 7.0.0.0/24 area 7 |
---|
319 | 25. write memory |
---|