- Configuring dataplane interfaces on MyPLC PlanetLab nodes
Configuring dataplane interfaces on MyPLC PlanetLab nodes
This page discusses techniques for managing secondary interfaces on MyPLC-controlled PlanetLab nodes, including installation and usage of the BBN developed plifconfig tool, and tips for handling secondary interface labeling. For upgrade instructions specific to your plficonfig version, please see the INSTALL file included in the source distribution.
Purpose of the plifconfig
Tool
plifconfig is a management tool for network interfaces on PlanetLab nodes. The tool can be used to be add new interfaces to a node, remove interfaces from a node, or show information about interfaces on a given node.
Secondary Interface IP Configuration Using plifconfig
Variables
These variables will be used throughout this document as placeholders inside of installation and usage instructions. As you work through this document, replace the variables with values that are specific to your setup.
Variable | Description | Notes |
<plifconfig-version> | The target plifconfig version | Example: plifconfig-2.0 |
<python-version> | The version of python installed on your node | Examples: python2.5, python2.6 |
<myplc-node> | FQDN of a target MyPLC-based PlanetLab node | |
<version-step> | The step between versions for a patch file | Example: 2.1_to_2.2 |
Set Up the Environment
Obtain Scripts and Patches
Download the current version of the plifconfig
package from http://www.gpolab.bbn.com/local-sw/. Copy the tarball onto your MyPLC machine, and unpackage it in your home directory:
cd ~ tar xvzf /path/to/<plifconfig-version>.tar.gz
Instructions below assume that you downloaded the package to your home directory on your MyPLC machine.
Add VLAN Tagtype to PlanetLab Database on MyPLC Machine
To create VLAN subinterfaces, you need to add a new tag type to your PlanetLab database. You should only do this if you have not already added the VLAN tagtype.
- Confirm that you have not added the VLAN tag type yet:
sudo ~/<plifconfig-version>/myplc/add_vlan_tagtype_to_plc_db -- -n echo $?
If 1 is printed out by the echo command then the VLAN tagtype already exists, and you should jump to applying the patch; otherwise, continue with this section.
- Run the
add_vlan_tagtype_to_plc_db
script:sudo ~/<plifconfig-version>/myplc/add_vlan_tagtype_to_plc_db
- Confirm tagtype was added:
sudo ~/<plifconfig-version>/myplc/add_vlan_tagtype_to_plc_db -- -n echo $?
If 1 is printed out by the echo command then the VLAN tagtype was successfully added to the PlanetLab database.
Apply the Patch to plnet.py
File on All Nodes
Make sure that you have a backup of the PlanetLab distributed originial plnet.py
.
- This may exist in
~/plnet.py.orig
if you followed previous instructions here. - If you have not patched the file before, then create a backup of the original file on your nodes.
cp /usr/lib/<python-version>/site-packages/plnet.py ~/plnet.py.orig
- If you cannot find a copy of the original file on your MyPLC-based PlanetLab nodes, then you can also find one on the MyPLC machine under
/usr/lib/<python-version>/site-packages/plnet.py
. Save a copy of this file to~/plnet.py.orig
on your nodes.
Option 1: Apply the Full Patch the Original File
If you are unsure which version of plifconfig you are running, or if you have not installed plifconfig before, you should use this option.
- From your MyPLC machine, copy the original
plnet.py.orig
file over from your MyPLC-based PlanetLab node.scp root@<myplc-node>:~/plnet.py.orig ~/plnet.py
- On your MyPLC machine, apply the patch:
patch ~/plnet.py < ~/<plifconfig-version>/nodes/plnet_full.patch
- If the patch succeeded, copy the newly patched file from your patch machine over to your MyPLC-based PlanetLab node:
scp ~/plnet.py root@<myplc-node>:~/usr/lib/<python-version>/site-packages/
- If the patch did not work please let us know at gpo-infra@geni.net.
- Login to the MyPLC-based PlanetLab node as root and restart NodeManager:
service nm restart
Option 2: Patch a Previously Patched Version of plnet.py
If you know that you have the second-most recent version of plifconfig installed and a <version-step> patch is included with this plifconfig version, then you can use this option.
If you log into your MyPLC machine and run
sudo plifconfig --version
then you can compare the version to the first number in <version-step>. Note that the <version-step> can be found by looking at ~/<plifconfig-version>/nodes/plnet_<version-step>.patch
.
- From your MyPLC machine, copy the
plnet.py
file over from your MyPLC-based PlanetLab node.scp root@<myplc-node>:/usr/lib/<python-version>/site-packages/plnet.py ~/
- On your MyPLC machine, apply the patch:
patch ~/plnet.py < ~/<plifconfig-version>/nodes/plnet_<version-step>.patch
- If the patch succeeded, copy the newly patched file from your patch machine over to your MyPLC-based PlanetLab node:
scp ~/plnet.py root@<myplc-node>:~/usr/lib/<python-version>/site-packages/
- If the patch did not work please let us know at gpo-infra@geni.net.
- Login to the MyPLC-based PlanetLab node as root and restart NodeManager:
service nm restart
Setup the plifconfig
Tool
- Setup the plifconfig.py library on your MyPLC machine:
cd ~/<plifconfig-version>/myplc/ sudo cp plifconfig.py /usr/local/lib/ sudo chown root /usr/local/lib/plifconfig.py sudo chmod 444 /usr/local/lib/plifconfig.py
- Setup the plifconfig wrapper script on your MyPLC machine:
cd ~/<plifconfig-version>/myplc/ sudo cp plifconfig /usr/local/sbin/ sudo chown root /usr/local/sbin/plifconfig sudo chmod 544 /usr/local/sbin/plifconfig
You should now be finished setting up the environment for plifconfig.
Using the plifconfig
Tool
Getting Started
plifconfig
currently supports 4 basic subcommands:
- add: add an interface with statically defined values (DHCP not yet supported)
- del: delete an interface
- show: show detailed information about an interface
- list: list all interfaces
The tool has a help option (-h or --help) for plifconfig and all of its subcommands for more information on usage.
[you@myplc] $ sudo plifconfig -h [you@myplc] $ sudo plifconfig add -h [you@myplc] $ sudo plifconfig del -h [you@myplc] $ sudo plifconfig show -h [you@myplc] $ sudo plifconfig list -h
Typical plifconfig
Workflows
Adding an Interface to a Node
If you have more than one NIC attached to your host, you will need to add it to the PlanetLab database on your MyPLC machine for it to be usable. In the example below, I will add eth1
to node2.fqdn.com
. Note that you can add interfaces without IP addresses by specifying an IP address of 0.0.0.0
and a netmask of 255.255.255.255
.
Options in this example:
Option | Description | Notes |
-n | Name of node to modify | |
-i | IP address | |
-m | Netmask | |
-d | Device name | |
-g | Gateway IP address | You do not need to specify a gateway if the node is not on a routable network |
--mac | MAC address | This should only be specified for base devices (e.g. not for subinterfaces like aliases or VLAN interfaces) |
[you@myplc] $ sudo plifconfig list Node: node1.fqdn.com IP: 125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0 IP: 1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1 IP: 10.2.3.3 HWaddr: Device: eth1:1234 Node: node2.fqdn.com IP: 125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0 [you@myplc] $ sudo plifconfig add -n node2.fqdn.com -i 10.2.3.6 -m 255.255.255.128 -d eth1 -g 10.2.3.1 --mac ab:12:cd:34:ef:99 Interface successfully created [you@myplc] $ sudo plifconfig list Node: node1.fqdn.com IP: 125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0 IP: 1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1 IP: 10.2.3.3 HWaddr: Device: eth1:1234 Node: node2.fqdn.com IP: 125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0 IP: 10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1
Adding an Alias or VLAN Interface to a Node:
You can create an alias or a VLAN interface in the same way that you create normal interfaces. The only difference is that you must pass a device name and a value for the VLAN id or the alias. The device must exist in the PlanetLab database in order to create the alias or VLAN interface.
Adding a VLAN Interface to a Node
Options in this example:
Option | Description | Notes |
-n | Name of node to modify | |
-i | IP address | |
-m | Netmask | |
-d | Device name | |
-g | Gateway IP address | You do not need to specify a gateway if the node is not on a routable network |
--vlan | VLAN ID | Creates a VLAN interface, ALWAYS use in conjunction with -d |
[you@myplc] $ sudo plifconfig list Node: node1.fqdn.com IP: 125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0 IP: 1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1 IP: 10.2.3.3 HWaddr: Device: eth1:1234 Node: node2.fqdn.com IP: 125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0 IP: 10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1 [you@myplc] $ sudo plifconfig add -n node2.fqdn.com -i 10.2.4.7 -m 255.255.255.128 -d eth1 --vlan 123 -g 10.2.4.1 Interface successfully created [you@myplc] $ sudo plifconfig list Node: node1.fqdn.com IP: 125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0 IP: 1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1 IP: 10.2.3.3 HWaddr: Device: eth1:1234 Node: node2.fqdn.com IP: 125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0 IP: 10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1 IP: 10.2.4.7 HWaddr: Device: eth1.123
Adding an Alias to a Node
Options in this example:
Option | Description | Notes |
-n | Name of node to modify | |
-i | IP address | |
-m | Netmask | |
-d | Device name | |
-g | Gateway IP address | You do not need to specify a gateway if the node is not on a routable network |
--alias | Alias name | Creates an interface alias, ALWAYS use in conjunction with -d |
[you@myplc] $ sudo plifconfig list Node: node1.fqdn.com IP: 125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0 IP: 1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1 IP: 10.2.3.3 HWaddr: Device: eth1:1234 Node: node2.fqdn.com IP: 125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0 IP: 10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1 IP: 10.2.4.7 HWaddr: Device: eth1.123 [you@myplc] $ sudo plifconfig add -n node2.fqdn.com -i 10.2.4.8 -m 255.255.255.128 -d eth1 --alias 123 -g 10.2.4.1 Interface successfully created [you@myplc] $ sudo plifconfig list Node: node1.fqdn.com IP: 125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0 IP: 1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1 IP: 10.2.3.3 HWaddr: Device: eth1:1234 Node: node2.fqdn.com IP: 125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0 IP: 10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1 IP: 10.2.4.7 HWaddr: Device: eth1.123 IP: 10.2.4.8 HWaddr: Device: eth1:123
Adding a Alias of a VLAN Interface to a Node
Options in this example:
Option | Description | Notes |
-n | Name of node to modify | |
-i | IP address | |
-m | Netmask | |
-d | Device name | |
-g | Gateway IP address | You do not need to specify a gateway if the node is not on a routable network |
--vlan | VLAN ID | Creates a VLAN interface, ALWAYS use in conjunction with -d |
--alias | Alias name | Creates an interface alias, ALWAYS use in conjunction with -d |
[you@myplc] $ sudo plifconfig list Node: node1.fqdn.com IP: 125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0 IP: 1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1 IP: 10.2.3.3 HWaddr: Device: eth1:1234 Node: node2.fqdn.com IP: 125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0 IP: 10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1 IP: 10.2.4.7 HWaddr: Device: eth1.123 IP: 10.2.4.8 HWaddr: Device: eth1:123 [you@myplc] $ sudo plifconfig add -n node2.fqdn.com -i 10.2.4.9 -m 255.255.255.128 -d eth1 --vlan 123 --alias 123 -g 10.2.4.1 Interface successfully created [you@myplc] $ sudo plifconfig list Node: node1.fqdn.com IP: 125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0 IP: 1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1 IP: 10.2.3.3 HWaddr: Device: eth1:1234 Node: node2.fqdn.com IP: 125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0 IP: 10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1 IP: 10.2.4.7 HWaddr: Device: eth1.123 IP: 10.2.4.8 HWaddr: Device: eth1:123 IP: 10.2.4.9 HWaddr: Device: eth1.123:123
Viewing Interface Details
Options in this example:
Option | Description | Notes |
-n | Name of node to modify | |
-d | Device name | |
--alias | Alias name | Creates an interface alias, ALWAYS use in conjunction with -d |
To look up more information about a specific node, you can use the show
subcommand. All of this information can also be found on the node or on the web interface, but the show
subcommand is useful for quickly debugging issues from a central location. Note that when querying by device, you must specify the full set of {device, vlan, alias} of the device. For example, to see information on eth1, specify -d eth1
. To see information on eth1.123:3, specify -d eth1 --vlan 123 --alias 3
. To see information on eth1:3, specify -d eth1 --alias 3
.
[you@myplc] $ sudo plifconfig show -n node1.fqdn.com -d eth1 --alias 1234 Interface details for query: node = node1.fqdn.com, device = eth1:1234 Node: node1.fqdn.com Device: eth1:1234 HWaddr: Is Primary? False Bandwidth limit: None IP Version: ipv4 Method: static IP: 10.2.3.3 Bcast: 10.2.3.63 Mask: 255.255.255.192 NetID: 10.2.3.0 GW: 10.2.3.1 DNS1: 125.125.125.244 DNS2:
Deleting Interfaces
If you need to remove an interface you can use the del
subcommand. Note that when querying by device, you must specify the full set of {device, vlan, alias} of the device. For example, to see information on eth1, specify -d eth1
. To see information on eth1.123:3, specify -d eth1 --vlan 123 --alias 3
. To see information on eth1:3, specify -d eth1 --alias 3
.
Options in this example:
Option | Description | Notes |
-n | Name of node to modify | |
-d | Device name | |
--alias | Alias name | Creates an interface alias, ALWAYS use in conjunction with -d |
[you@myplc] $ sudo plifconfig list Node: node1.fqdn.com IP: 125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0 IP: 1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1 IP: 10.2.3.3 HWaddr: Device: eth1:1234 Node: node2.fqdn.com IP: 125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0 IP: 10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1 IP: 10.2.4.7 HWaddr: Device: eth1.123 [you@myplc] $ sudo plifconfig del -n node1.fqdn.com -d eth1 --alias 1234 Interface successfully deleted [you@myplc] $ sudo plifconfig list Node: node1.fqdn.com IP: 125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0 IP: 1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1 Node: node2.fqdn.com IP: 125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0 IP: 10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1 IP: 10.2.4.7 HWaddr: Device: eth1.123
plifconfig
Best Practices
- Always fully specify the interface you are operating on. There are three
plifconfig
flags,-d
,--vlan
, and--alias
, which are used to uniquely identify a given device and subinterface. While it may be possible for the OS to fill in the-d
flag correctly, you should always specify it when you invokeplifconfig
to ensure that you are operating on the correct device. Whenever you are viewing or modifying information for a subinterface, always specify that subinterface fully.
- Use MAC addresses when defining physical interfaces. Whenever you configure a physical interface (e.g.
eth1
) inplifconfig
, specify the MAC address. This will ensure that device names do not change across node reboots. However, do not specify MAC addresses when defining virtual subinterfaces (whether VLAN- or alias-based) --- a known bug causes trouble when the same MAC address is used on multiple defined interfaces.
- Be careful when adding or deleting interfaces.
plifconfig
does not ask for confirmation when adding or deleting interfaces. For example,plifconfig
won't prevent you from deleting your primary interface. If you do this, you might not be able to change the interfaces on the node without doing a physical reboot of the node.
- Manage interfaces from the MyPLC machine, not from the PlanetLab node. If you try to manage interfaces on the node using traditional tools like
ifconfig
andifcfg-ethX
startup scripts, it will clash with the PlanetLab way of creating interfaces. MyPLC wants to manage the interfaces actively, so use eitherplifconfig
or the MyPLC web interface.
- Use the fast sliver creation change described here. PlanetLab nodes will create new slivers and new interfaces at the same time; thus, if you are only creating slivers every 15 minutes, then you will only be able to add or delete interfaces every 15 minutes.
plifconfig
Known Bugs
- When using
plifconfig
, you can only specify a mac address once when adding interfaces. Although aliases and VLAN interfaces share the same mac address as their associated ethX parent interface, if you specify that mac address for all of these interfaces then unexpected behavior can occur on the node. For the time being, it is recommended to specify a mac address for the ethX interface, and to not specify mac addresses for VLAN interfaces or aliases.
- Currently,
plifconfig
allows you to delete an interface that has subinterfaces attached to it. For example, if I have eth1, eth1.123, and eth1:3,plifconfig
does not prevent you from deleting eth1. Similarly, plifconfig does not prevent you from adding a subinterface to a physical interface that is not already in the database.
plifconfig
FAQ
- Q: Can I create multiple interfaces with the same IP address?
A: Yes, but not on the same node. The only exception to this rule is having multiple interfaces with an IP address of 0.0.0.0.
- Q: Interfaces are not showing up inside of my slices. How can I fix that?
A: One suggested fix which may or may not apply to you is listed here.
Another suggestion is outlined below:- Log into your MyPLC's web interface
- Click "Slices" in the left-hand navigation bar
- Click on the name of the slice of your choice in the table on the slices page
- On the page for your slice:
- Expand the "tags" view
- Select "ip_addresses" from the "Choose Tag" dropdown
- Enter 0.0.0.0 into the text box as the value of the tag
- Make sure either "All Nodes" or your specific node is selected to apply the tag to
- Hit the "Set Tag" button to confirm
- Log into the root context of your PlanetLab node and run:
service nm restart
- Log into the slice and confirm that the interfaces are visible
- Q: How do I modify an interface that I have already added?
A: There is currently no way to do this throughplifconfig
other than deleting and adding the interface again. You can still modify the interface's properties through the "interface" and "interface tags" pages of the MyPLC web interface.
- Q: Can I set the bandwidth limit of an interface through
plifconfig
?
A: Not currently. Please contact gpo-infra@geni.net if this feature would be useful in your environment.
- Q: Does
plifconfig
support creation of DHCP interfaces?
A: Not currently. Please contact gpo-infra@geni.net if this feature would be useful in your environment.
- Q: Does
plifconfig
support creation of IPv6 interfaces?
A: Not currently. Please contact gpo-infra@geni.net if this feature would be useful in your environment.
Interface numbering gotchas
PlanetLab nodes boot using a two-stage process. When multiple interface cards are present, the first-stage boot and second-stage boot may not number the interfaces consistently. This typically causes the node to be unusable.
We don't know how to control which interface the first-stage boot will use as eth0. However, it is easy to use the MyPLC interface to control which MAC address should be assigned as each interface in the second-stage boot. This section describes how we force MyPLC to use the same interface allocation in second-stage boot as it did in first-stage, allowing the node to be used even if these settings do not natively match.
Step 1: Boot into first-stage with the control interface connected
Before booting the node, physically connect the control interface cable to the interface which first-stage boot will choose as eth0
. On our lab nodes (1U Dell R300s), first-stage boot always chooses the leftmost interface on the leftmost expansion NIC to be eth0
(or the leftmost interface on the motherboard if there are no expansion cards). In your environment, you may need to find this interface by guesswork: if the node can successfully enter first-stage boot and seems to reach a server, you guessed right.
Step 2: Use the first-stage boot logs to identify the MAC address of eth0
The MyPLC gathers logs from each node's first-stage boot. These logs will contain the MAC address of first-stage eth0
on your node. To find this, login to MyPLC, and do:
sudo find /var/log/bm/raw | grep <node_hostname> | sort | tail -1 | xargs sudo grep macs
That command should report output similar to this:
$ sudo find /var/log/bm/raw | grep <node_hostname> | sort | tail -1 | xargs sudo grep macs 19:48:22(UTC) net:InitInterfaces macs = {'00:00:00:00:00:00': 'lo', 'aa:bb:cc:dd:ee:ff': 'eth0'}
Use the reported MAC address:
<eth0_mac>
:aa:bb:cc:dd:ee:ff
Step 3: Configure MyPLC to use the correct MAC in stage 2
Use the PlanetLab UI to force the second-stage boot to use the leftmost interface on the leftmost expansion card:
- Browse to the MyPLC web interface and login as an administrator
- Click "My Site Nodes" in the left menubar
- Click "
<node_hostname>
" in the table of nodes - Expand the "One interface" menu
- Click the IP of the primary interface
- Set MAC to
<eth0_mac>
- Click "Update"
- Click "Back to Node"
- Set MAC to
- Click "Update Node"
- Click the IP of the primary interface
- Reboot the node
Step 4: Modify udev to fix the live interface configuration
Sometimes, Fedora gets confused by eth0 switching identities, as a result of which eth1 or eth2 (if you have that many interfaces) does not appear, and is replaced by a weird devNNNNN
interface. This appears to be fixable by modifying udev's configuration:
- Login to the node as root
- Edit
/etc/udev/rules.d/70-persistent-net.rules
- Find a configuration section containing the comment
(custom name provided by external tool)
, e.g.:# PCI device 0x8086:0x10c9 (igb) (custom name provided by external tool) SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:1b:21:4b:3f:ac", ATTR{type}=="1", NAME="eth0"
Make sure that there is only one such section, and that its MAC address matches<eth0_mac>
- Delete all SUBSYSTEM comment-and-line blocks except this one, and save the file
- Reboot the node