wiki:GpoLab/MyplcNodeDataplaneInterfaces

Version 5 (modified by chaos@bbn.com, 13 years ago) (diff)

--

Configuring dataplane interfaces on MyPLC PlanetLab nodes

This page discusses how to configure secondary interfaces on MyPLC-controlled PlanetLab nodes.

Secondary interface IP configuration using plifconfig

Variables

Variable Description Notes
<plifconfig-version> The target plifconfig version Example: plifconfig-2.0
<python-version> The version of python installed on your node Examples: python2.5, python2.6
<myplc-node> FQDN of a target MyPLC-based PlanetLab node

Set Up the Environment for VLAN Interfaces

Obtain Scripts and Patches

The latest plifconfig tarball can be obtained from gpo-infra@geni.net. Copy the tarball onto your MyPLC machine, and unpackage it in your home directory:

cd ~
tar xvzf /path/to/<plifconfig-version>.tar.gz

Instructions below assume that you downloaded the package to your home directory on your MyPLC machine.

Add VLAN Tagtype to PlanetLab Database on MyPLC Machine

To create VLAN subinterfaces, you need to add a new tag type to your PlanetLab database.

  • Run the add_vlan_tagtype_to_plc_db script:
    sudo ~/<plifconfig-version>/myplc/add_vlan_tagtype_to_plc_db
    
  • Confirm tagtype was added:
    sudo ~/<plifconfig-version>/myplc/add_vlan_tagtype_to_plc_db -- -n
    echo $?
    

If 1 is printed out by the echo command then the VLAN tagtype was successfully added to the PlanetLab database.

Apply the Patch to plnet.py File on All Nodes

In order to create VLAN interfaces on a node, you need to patch your plnet.py file on each PlanetLab node associated with your MyPLC. The following instructions assume that your MyPLC machine has patch installed, but the individual PlanetLab nodes do not.

  • Log into your MyPLC-based PlanetLab node as root and make a backup of the existing plnet.py file
    cp /usr/lib/<python-version>/site-packages/plnet.py ~/plnet.py.orig
    
  • From your MyPLC machine, copy the plnet.py file over from your MyPLC-based PlanetLab node.
    scp root@<myplc-node>:/usr/lib/<python-version>/site-packages/plnet.py ~/
    
  • On your MyPLC machine, apply the patch:
    patch ~/plnet.py < ~/<plifconfig-version>/nodes/plnet.patch
    
  • If the patch succeeded, copy the newly patched file from your patch machine over to your MyPLC-based PlanetLab node:
    scp ~/plnet.py root@<myplc-node>:~/usr/lib/<python-version>/site-packages/
    
  • Login to the MyPLC-based PlanetLab node as root and restart NodeManager:
    service nm restart
    

What if the patch failed?

If the patch did not work please let us know at gpo-infra@geni.net.

The plifconfig Tool

Purpose of the plifconfig Tool

plifconfig is a management tool for network interfaces on PlanetLab nodes.
The tool can be used to be add new interfaces to a node, remove interfaces
from a node, or show information about interfaces on a given node.

Setup the plifconfig Tool

  • Setup the plifconfig.py library on your MyPLC machine:
    cd ~/plifconfig-<version>/myplc/
    sudo cp plifconfig.py /usr/local/lib/
    sudo chown root /usr/local/lib/plifconfig.py
    sudo chmod 444 /usr/local/lib/plifconfig.py
    
  • Setup the plifconfig wrapper script on your MyPLC machine:
    cd ~/plifconfig-<version>/myplc/
    sudo cp plifconfig /usr/local/sbin/
    sudo chown root /usr/local/sbin/plifconfig
    sudo chmod 544 /usr/local/sbin/plifconfig
    

Using the plifconfig Tool

plifconfig currently supports 4 basic subcommands:

  • add: add an interface with statically defined values (DHCP not yet supported)
  • del: delete an interface
  • show: show detailed information about an interface
  • list: list all interfaces

The tool has a help option (-h or --help) for plifconfig and all of its subcommands for more information on usage.

[you@myplc] $ sudo plifconfig -h
[you@myplc] $ sudo plifconfig add -h
[you@myplc] $ sudo plifconfig del -h
[you@myplc] $ sudo plifconfig show -h
[you@myplc] $ sudo plifconfig list -h

Typical plifconfig Workflows

Adding an Interface to a Node

If you have more than one NIC attached to your host, you will need to add it to the PlanetLab database on your MyPLC machine for it to be usable. In the example below, I will add eth1 to node2.fqdn.com. Note that you can add interfaces without IP addresses by specifying an IP address of 0.0.0.0 and a netmask of 255.255.255.255.

[you@myplc] $ sudo plifconfig list

Node: node1.fqdn.com
	IP:  125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0
	IP:         1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1
	IP:        10.2.3.3 HWaddr:                   Device: eth1:1234

Node: node2.fqdn.com
	IP:   125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0       

[you@myplc] $ sudo plifconfig add -n node2.fqdn.com -i 10.2.3.6 -m 255.255.255.128 -d eth1 -g 10.2.3.1 --mac ab:12:cd:34:ef:99

Interface successfully created

[you@myplc] $ sudo plifconfig list

Node: node1.fqdn.com
	IP:  125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0
	IP:         1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1
	IP:        10.2.3.3 HWaddr:                   Device: eth1:1234

Node: node2.fqdn.com
	IP:   125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0
       	IP:        10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1
Adding an Alias or VLAN Interface to a Node:

You can create an alias or a VLAN interface in the same way that you create normal interfaces. The only difference is that you must pass a device name and a value for the VLAN id or the alias. The device must exist in the PlanetLab database in order to create the alias or VLAN interface.

Adding a VLAN Interface to a Node
[you@myplc] $ sudo plifconfig list

Node: node1.fqdn.com
	IP:  125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0
	IP:         1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1
	IP:        10.2.3.3 HWaddr:                   Device: eth1:1234

Node: node2.fqdn.com
	IP:   125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0
       	IP:        10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1       

[you@myplc] $ sudo plifconfig add -n node2.fqdn.com -i 10.2.4.7 -m 255.255.255.128 -d eth1 --vlan 123 -g 10.2.4.1

Interface successfully created

[you@myplc] $ sudo plifconfig list

Node: node1.fqdn.com
	IP:  125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0
	IP:         1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1
	IP:        10.2.3.3 HWaddr:                   Device: eth1:1234

Node: node2.fqdn.com
	IP:   125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0   
       	IP:        10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1
       	IP:        10.2.4.7 HWaddr:                   Device: eth1.123
Adding an Alias to a Node
[you@myplc] $ sudo plifconfig list

Node: node1.fqdn.com
	IP:  125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0
	IP:         1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1
	IP:        10.2.3.3 HWaddr:                   Device: eth1:1234

Node: node2.fqdn.com
	IP:   125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0
       	IP:        10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1
       	IP:        10.2.4.7 HWaddr:                   Device: eth1.123      

[you@myplc] $ sudo plifconfig add -n node2.fqdn.com -i 10.2.4.8 -m 255.255.255.128 -d eth1 --alias 123 -g 10.2.4.1

Interface successfully created

[you@myplc] $ sudo plifconfig list

Node: node1.fqdn.com
	IP:  125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0
	IP:         1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1
	IP:        10.2.3.3 HWaddr:                   Device: eth1:1234

Node: node2.fqdn.com
	IP:   125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0   
       	IP:        10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1
       	IP:        10.2.4.7 HWaddr:                   Device: eth1.123
       	IP:        10.2.4.8 HWaddr:                   Device: eth1:123
Adding a Alias of a VLAN Interface to a Node
[you@myplc] $ sudo plifconfig list

Node: node1.fqdn.com
	IP:  125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0
	IP:         1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1
	IP:        10.2.3.3 HWaddr:                   Device: eth1:1234

Node: node2.fqdn.com
	IP:   125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0
       	IP:        10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1
       	IP:        10.2.4.7 HWaddr:                   Device: eth1.123
       	IP:        10.2.4.8 HWaddr:                   Device: eth1:123    

[you@myplc] $ sudo plifconfig add -n node2.fqdn.com -i 10.2.4.9 -m 255.255.255.128 -d eth1 --vlan 123 --alias 123 -g 10.2.4.1

Interface successfully created

[you@myplc] $ sudo plifconfig list

Node: node1.fqdn.com
	IP:  125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0
	IP:         1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1
	IP:        10.2.3.3 HWaddr:                   Device: eth1:1234

Node: node2.fqdn.com
	IP:   125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0   
       	IP:        10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1
       	IP:        10.2.4.7 HWaddr:                   Device: eth1.123
       	IP:        10.2.4.8 HWaddr:                   Device: eth1:123
       	IP:        10.2.4.9 HWaddr:                   Device: eth1.123:123
Viewing Interface Details

To look up more information about a specific node, you can use the show subcommand. All of this information can also be found on the node or on the web interface, but the show subcommand is useful for quickly debugging issues from a central location. Note that when querying by device, you must specify the full set of {device, vlan, alias} of the device. For example, to see information on eth1, specify -d eth1 . To see information on eth1.123:3, specify -d eth1 --vlan 123 --alias 3 . To see information on eth1:3, specify -d eth1 --alias 3 .

[you@myplc] $ sudo plifconfig show -n node1.fqdn.com -d eth1 --alias 1234 
Interface details for query: node = node1.fqdn.com, device = eth1:1234
Node: node1.fqdn.com
Device: eth1:1234
HWaddr: 
Is Primary? False
Bandwidth limit: None
IP Version: ipv4
Method: static

IP:           10.2.3.3
Bcast:       10.2.3.63 Mask: 255.255.255.192
NetID:        10.2.3.0 GW:          10.2.3.1
DNS1:  125.125.125.244 DNS2:
Deleting Interfaces

If you need to remove an interface you can use the del subcommand. Note that when querying by device, you must specify the full set of {device, vlan, alias} of the device. For example, to see information on eth1, specify -d eth1 . To see information on eth1.123:3, specify -d eth1 --vlan 123 --alias 3 . To see information on eth1:3, specify -d eth1 --alias 3 .

[you@myplc] $ sudo plifconfig list

Node: node1.fqdn.com
	IP:  125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0
	IP:         1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1
	IP:        10.2.3.3 HWaddr:                   Device: eth1:1234

Node: node2.fqdn.com
	IP:   125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0
       	IP:        10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1
       	IP:        10.2.4.7 HWaddr:                   Device: eth1.123

[you@myplc] $ sudo plifconfig del -n node1.fqdn.com -d eth1 --alias 1234

Interface successfully deleted

[you@myplc] $ sudo plifconfig list

Node: node1.fqdn.com
	IP:  125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0
	IP:         1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1

Node: node2.fqdn.com
	IP:   125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0
       	IP:        10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1
       	IP:        10.2.4.7 HWaddr:                   Device: eth1.123

plifconfig Best Practices

  • Always fully specify the interface you are operating on. There are three plifconfig flags, -d, --vlan, and --alias, which are used to uniquely identify a given device and subinterface. While it may be possible for the OS to fill in the -d flag correctly, you should always specify it when you invoke plifconfig to ensure that you are operating on the correct device. Whenever you are viewing or modifying information for a subinterface, always specify that subinterface fully.
  • Use MAC addresses when defining physical interfaces. Whenever you configure a physical interface (e.g. eth1) in plifconfig, specify the MAC address. This will ensure that device names do not change across node reboots. However, do not specify MAC addresses when defining virtual subinterfaces (whether VLAN- or alias-based) --- a known bug causes trouble when the same MAC address is used on multiple defined interfaces.
  • Be careful when adding or deleting interfaces. plifconfig does not ask for confirmation when adding or deleting interfaces. For example, plifconfig won't prevent you from deleting your primary interface. If you do this, you might not be able to change the interfaces on the node without doing a physical reboot of the node.
  • Manage interfaces from the MyPLC machine, not from the PlanetLab node. If you try to manage interfaces on the node using traditional tools like ifconfig and ifcfg-ethX startup scripts, it will clash with the PlanetLab way of creating interfaces. MyPLC wants to manage the interfaces actively, so use either plifconfig or the MyPLC web interface.
  • Use the fast sliver creation change described here. PlanetLab nodes will create new slivers and new interfaces at the same time; thus, if you are only creating slivers every 15 minutes, then you will only be able to add or delete interfaces every 15 minutes.

plifconfig Known Bugs

  • When using plifconfig, you can only specify a mac address once when adding interfaces. Although aliases and VLAN interfaces share the same mac address as their associated ethX parent interface, if you specify that mac address for all of these interfaces then unexpected behavior can occur on the node. For the time being, it is recommended to specify a mac address for the ethX interface, and to not specify mac addresses for VLAN interfaces or aliases.
  • Currently, plifconfig allows you to delete an interface that has subinterfaces attached to it. For example, if I have eth1, eth1.123, and eth1:3, plifconfig does not prevent you from deleting eth1. Similarly, plifconfig does not prevent you from adding a subinterface to a physical interface that is not already in the database.

plifconfig FAQ

  1. Q: Can I create multiple interfaces with the same IP address?
    A: Yes, but not on the same node. The only exception to this rule is having multiple interfaces with an IP address of 0.0.0.0.
  1. Q: Interfaces are not showing up inside of my slices. How can I fix that?
    A: One suggested fix which may or may not apply to you is listed here.
    Another suggestion is outlined below:
    • Log into your MyPLC's web interface
    • Click "Slices" in the left-hand navigation bar
    • Click on the name of the slice of your choice in the table on the slices page
      • On the page for your slice:
      • Expand the "tags" view
      • Select "ip_addresses" from the "Choose Tag" dropdown
      • Enter 0.0.0.0 into the text box as the value of the tag
      • Make sure either "All Nodes" or your specific node is selected to apply the tag to
      • Hit the "Set Tag" button to confirm
    • Log into the root context of your PlanetLab node and run:
      service nm restart
      
    • Log into the slice and confirm that the interfaces are visible
  1. Q: How do I modify an interface that I have already added?
    A: There is currently no way to do this through plifconfig other than deleting and adding the interface again. You can still modify the interface's properties through the "interface" and "interface tags" pages of the MyPLC web interface.
  1. Q: Can I set the bandwidth limit of an interface through plifconfig?
    A: Not currently. Please contact gpo-infra@geni.net if this feature would be useful in your environment.
  1. Q: Does plifconfig support creation of DHCP interfaces?
    A: Not currently. Please contact gpo-infra@geni.net if this feature would be useful in your environment.
  1. Q: Does plifconfig support creation of IPv6 interfaces?
    A: Not currently. Please contact gpo-infra@geni.net if this feature would be useful in your environment.

Interface numbering gotchas

PlanetLab nodes boot using a two-stage process. When multiple interface cards are present, the first-stage boot and second-stage boot may not number the interfaces consistently. This typically causes the node to be unusable.

We don't know how to control which interface the first-stage boot will use as eth0. However, it is easy to use the MyPLC interface to control which MAC address should be assigned as each interface in the second-stage boot. This section describes how we force MyPLC to use the same interface allocation in second-stage boot as it did in first-stage, allowing the node to be used even if these settings do not natively match.

Step 1: Boot into first-stage with the control interface connected

Before booting the node, physically connect the control interface cable to the interface which first-stage boot will choose as eth0. On our lab nodes (1U Dell R300s), first-stage boot always chooses the leftmost interface on the leftmost expansion NIC to be eth0 (or the leftmost interface on the motherboard if there are no expansion cards). In your environment, you may need to find this interface by guesswork: if the node can successfully enter first-stage boot and seems to reach a server, you guessed right.

Step 2: Use the first-stage boot logs to identify the MAC address of eth0

The MyPLC gathers logs from each node's first-stage boot. These logs will contain the MAC address of first-stage eth0 on your node. To find this, login to MyPLC, and do:

sudo find /var/log/bm/raw | grep <node_hostname> | sort | tail -1 | xargs sudo grep macs

That command should report output similar to this:

$ sudo find /var/log/bm/raw | grep <node_hostname> | sort | tail -1 | xargs sudo grep macs
19:48:22(UTC) net:InitInterfaces macs = {'00:00:00:00:00:00': 'lo', 'aa:bb:cc:dd:ee:ff': 'eth0'}

Use the reported MAC address:

  • <eth0_mac>: aa:bb:cc:dd:ee:ff

Step 3: Configure MyPLC to use the correct MAC in stage 2

Use the PlanetLab UI to force the second-stage boot to use the leftmost interface on the leftmost expansion card:

  • Browse to the MyPLC web interface and login as an administrator
  • Click "My Site Nodes" in the left menubar
  • Click "<node_hostname>" in the table of nodes
  • Expand the "One interface" menu
    • Click the IP of the primary interface
      • Set MAC to <eth0_mac>
      • Click "Update"
      • Click "Back to Node"
    • Click "Update Node"
  • Reboot the node

Step 4: Modify udev to fix the live interface configuration

Sometimes, Fedora gets confused by eth0 switching identities, as a result of which eth1 or eth2 (if you have that many interfaces) does not appear, and is replaced by a weird devNNNNN interface. This appears to be fixable by modifying udev's configuration:

  • Login to the node as root
  • Edit /etc/udev/rules.d/70-persistent-net.rules
  • Find a configuration section containing the comment (custom name provided by external tool), e.g.:
    # PCI device 0x8086:0x10c9 (igb) (custom name provided by external tool)
    SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:1b:21:4b:3f:ac", ATTR{type}=="1", NAME="eth0"
    
    Make sure that there is only one such section, and that its MAC address matches <eth0_mac>
  • Delete all SUBSYSTEM comment-and-line blocks except this one, and save the file
  • Reboot the node