wiki:GpoLab/MyplcNodeDataplaneInterfaces

Version 1 (modified by chaos@bbn.com, 8 years ago) (diff)

--

Configuring dataplane interfaces on MyPLC PlanetLab nodes

This page discusses how to configure secondary interfaces on MyPLC-controlled PlanetLab nodes.

Interface numbering gotchas

PlanetLab nodes boot using a two-stage process. When multiple interface cards are present, the first-stage boot and second-stage boot may not number the interfaces consistently. This typically causes the node to be unusable.

We don't know how to control which interface the first-stage boot will use as eth0. However, it is easy to use the MyPLC interface to control which MAC address should be assigned as each interface in the second-stage boot. This section describes how we force MyPLC to use the same interface allocation in second-stage boot as it did in first-stage, allowing the node to be used even if these settings do not natively match.

Step 1: Boot into first-stage with the control interface connected

Before booting the node, physically connect the control interface cable to the interface which first-stage boot will choose as eth0. On our lab nodes (1U Dell R300s), first-stage boot always chooses the leftmost interface on the leftmost expansion NIC to be eth0 (or the leftmost interface on the motherboard if there are no expansion cards). In your environment, you may need to find this interface by guesswork: if the node can successfully enter first-stage boot and seems to reach a server, you guessed right.

Step 2: Use the first-stage boot logs to identify the MAC address of eth0

The MyPLC gathers logs from each node's first-stage boot. These logs will contain the MAC address of first-stage eth0 on your node. To find this, login to MyPLC, and do:

sudo find /var/log/bm/raw | grep <node_hostname> | sort | tail -1 | xargs sudo grep macs

That command should report output similar to this:

$ sudo find /var/log/bm/raw | grep <node_hostname> | sort | tail -1 | xargs sudo grep macs
19:48:22(UTC) net:InitInterfaces macs = {'00:00:00:00:00:00': 'lo', 'aa:bb:cc:dd:ee:ff': 'eth0'}

Use the reported MAC address:

  • <eth0_mac>: aa:bb:cc:dd:ee:ff

Step 3: Configure MyPLC to use the correct MAC in stage 2

Use the PlanetLab UI to force the second-stage boot to use the leftmost interface on the leftmost expansion card:

  • Browse to the MyPLC web interface and login as an administrator
  • Click "My Site Nodes" in the left menubar
  • Click "<node_hostname>" in the table of nodes
  • Expand the "One interface" menu
    • Click the IP of the primary interface
      • Set MAC to <eth0_mac>
      • Click "Update"
      • Click "Back to Node"
    • Click "Update Node"
  • Reboot the node

Step 4: Modify udev to fix the live interface configuration

Sometimes, Fedora gets confused by eth0 switching identities, as a result of which eth1 or eth2 (if you have that many interfaces) does not appear, and is replaced by a weird devNNNNN interface. This appears to be fixable by modifying udev's configuration:

  • Login to the node as root
  • Edit /etc/udev/rules.d/70-persistent-net.rules
  • Find a configuration section containing the comment (custom name provided by external tool), e.g.:
    # PCI device 0x8086:0x10c9 (igb) (custom name provided by external tool)
    SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:1b:21:4b:3f:ac", ATTR{type}=="1", NAME="eth0"
    
    Make sure that there is only one such section, and that its MAC address matches <eth0_mac>
  • Delete all SUBSYSTEM comment-and-line blocks except this one, and save the file
  • Reboot the node