Changes between Initial Version and Version 1 of GpoLab/MyplcNodeDataplaneInterfaces


Ignore:
Timestamp:
04/20/11 16:57:56 (10 years ago)
Author:
chaos@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • GpoLab/MyplcNodeDataplaneInterfaces

    v1 v1  
     1[[PageOutline]]
     2
     3= Configuring dataplane interfaces on MyPLC PlanetLab nodes =
     4
     5This page discusses how to configure secondary interfaces on MyPLC-controlled PlanetLab nodes.
     6
     7== Interface numbering gotchas ==
     8
     9PlanetLab nodes boot using a two-stage process.  When multiple interface cards are present, the first-stage boot and second-stage boot may not number the interfaces consistently.  This typically causes the node to be unusable.
     10
     11We don't know how to control which interface the first-stage boot will use as eth0.  However, it is easy to use the MyPLC interface to control which MAC address should be assigned as each interface in the second-stage boot.  This section describes how we force MyPLC to use the same interface allocation in second-stage boot as it did in first-stage, allowing the node to be used even if these settings do not natively match.
     12
     13=== Step 1: Boot into first-stage with the control interface connected ===
     14
     15Before booting the node, physically connect the control interface cable to the interface which first-stage boot will choose as `eth0`.  On our lab nodes (1U Dell R300s), first-stage boot always chooses the leftmost interface on the leftmost expansion NIC to be `eth0` (or the leftmost interface on the motherboard if there are no expansion cards).  In your environment, you may need to find this interface by guesswork: if the node can successfully enter first-stage boot and seems to reach a server, you guessed right.
     16
     17=== Step 2: Use the first-stage boot logs to identify the MAC address of eth0 ===
     18
     19The MyPLC gathers logs from each node's first-stage boot.  These logs will contain the MAC address of first-stage `eth0` on your node.  To find this, login to MyPLC, and do:
     20{{{
     21sudo find /var/log/bm/raw | grep <node_hostname> | sort | tail -1 | xargs sudo grep macs
     22}}}
     23That command should report output similar to this:
     24{{{
     25$ sudo find /var/log/bm/raw | grep <node_hostname> | sort | tail -1 | xargs sudo grep macs
     2619:48:22(UTC) net:InitInterfaces macs = {'00:00:00:00:00:00': 'lo', 'aa:bb:cc:dd:ee:ff': 'eth0'}
     27}}}
     28
     29Use the reported MAC address:
     30 * `<eth0_mac>`: `aa:bb:cc:dd:ee:ff`
     31
     32=== Step 3: Configure MyPLC to use the correct MAC in stage 2 ===
     33
     34Use the PlanetLab UI to force the second-stage boot to use the leftmost interface on the leftmost expansion card:
     35 * Browse to the MyPLC web interface and login as an administrator
     36 * Click "My Site Nodes" in the left menubar
     37 * Click "`<node_hostname>`" in the table of nodes
     38 * Expand the "One interface" menu
     39   * Click the IP of the primary interface
     40     * Set MAC to `<eth0_mac>`
     41     * Click "Update"
     42     * Click "Back to Node"
     43   * Click "Update Node"
     44 * Reboot the node
     45
     46=== Step 4: Modify udev to fix the live interface configuration ===
     47
     48Sometimes, Fedora gets confused by eth0 switching identities, as a result of which eth1 or eth2 (if you have that many interfaces) does not appear, and is replaced by a weird `devNNNNN` interface.  This appears to be fixable by modifying udev's configuration:
     49
     50 * Login to the node as root
     51 * Edit `/etc/udev/rules.d/70-persistent-net.rules`
     52 * Find a configuration section containing the comment `(custom name provided by external tool)`, e.g.:
     53{{{
     54# PCI device 0x8086:0x10c9 (igb) (custom name provided by external tool)
     55SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:1b:21:4b:3f:ac", ATTR{type}=="1", NAME="eth0"
     56}}}
     57 Make sure that there is only one such section, and that its MAC address matches `<eth0_mac>`
     58 * Delete all SUBSYSTEM comment-and-line blocks except this one, and save the file
     59 * Reboot the node
     60