Changes between Initial Version and Version 1 of GpoLab/MyplcNodeDataplaneInterfaces

Timestamp:

04/20/11 16:57:56 (13 years ago)

Author:

chaos@bbn.com

Comment:

--

GpoLab/MyplcNodeDataplaneInterfaces

@@ +1 @@
+[[PageOutline]]
+
+= Configuring dataplane interfaces on MyPLC PlanetLab nodes =
+
+This page discusses how to configure secondary interfaces on MyPLC-controlled PlanetLab nodes.
+
+== Interface numbering gotchas == 
+
+PlanetLab nodes boot using a two-stage process.  When multiple interface cards are present, the first-stage boot and second-stage boot may not number the interfaces consistently.  This typically causes the node to be unusable.
+
+We don't know how to control which interface the first-stage boot will use as eth0.  However, it is easy to use the MyPLC interface to control which MAC address should be assigned as each interface in the second-stage boot.  This section describes how we force MyPLC to use the same interface allocation in second-stage boot as it did in first-stage, allowing the node to be used even if these settings do not natively match. 
+
+=== Step 1: Boot into first-stage with the control interface connected ===
+
+Before booting the node, physically connect the control interface cable to the interface which first-stage boot will choose as `eth0`.  On our lab nodes (1U Dell R300s), first-stage boot always chooses the leftmost interface on the leftmost expansion NIC to be `eth0` (or the leftmost interface on the motherboard if there are no expansion cards).  In your environment, you may need to find this interface by guesswork: if the node can successfully enter first-stage boot and seems to reach a server, you guessed right.
+
+=== Step 2: Use the first-stage boot logs to identify the MAC address of eth0 ===
+
+The MyPLC gathers logs from each node's first-stage boot.  These logs will contain the MAC address of first-stage `eth0` on your node.  To find this, login to MyPLC, and do:
+{{{
+sudo find /var/log/bm/raw | grep <node_hostname> | sort | tail -1 | xargs sudo grep macs
+}}}
+That command should report output similar to this:
+{{{
+$ sudo find /var/log/bm/raw | grep <node_hostname> | sort | tail -1 | xargs sudo grep macs
+19:48:22(UTC) net:InitInterfaces macs = {'00:00:00:00:00:00': 'lo', 'aa:bb:cc:dd:ee:ff': 'eth0'}
+}}}
+
+Use the reported MAC address:
+ * `<eth0_mac>`: `aa:bb:cc:dd:ee:ff`
+
+=== Step 3: Configure MyPLC to use the correct MAC in stage 2 ===
+
+Use the PlanetLab UI to force the second-stage boot to use the leftmost interface on the leftmost expansion card:
+ * Browse to the MyPLC web interface and login as an administrator
+ * Click "My Site Nodes" in the left menubar
+ * Click "`<node_hostname>`" in the table of nodes
+ * Expand the "One interface" menu
+   * Click the IP of the primary interface
+     * Set MAC to `<eth0_mac>`
+     * Click "Update"
+     * Click "Back to Node" 
+   * Click "Update Node" 
+ * Reboot the node 
+
+=== Step 4: Modify udev to fix the live interface configuration ===
+
+Sometimes, Fedora gets confused by eth0 switching identities, as a result of which eth1 or eth2 (if you have that many interfaces) does not appear, and is replaced by a weird `devNNNNN` interface.  This appears to be fixable by modifying udev's configuration: 
+
+ * Login to the node as root
+ * Edit `/etc/udev/rules.d/70-persistent-net.rules`
+ * Find a configuration section containing the comment `(custom name provided by external tool)`, e.g.:
+{{{
+# PCI device 0x8086:0x10c9 (igb) (custom name provided by external tool)
+SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:1b:21:4b:3f:ac", ATTR{type}=="1", NAME="eth0"
+}}}
+ Make sure that there is only one such section, and that its MAC address matches `<eth0_mac>`
+ * Delete all SUBSYSTEM comment-and-line blocks except this one, and save the file
+ * Reboot the node 
+