[[PageOutline]] = Configuring dataplane interfaces on MyPLC PlanetLab nodes = This page discusses techniques for managing secondary interfaces on MyPLC-controlled PlanetLab nodes, including installation and usage of the BBN developed plifconfig tool, and tips for handling secondary interface labeling. For upgrade instructions specific to your plficonfig version, please see the INSTALL file included in the source distribution. == Purpose of the `plifconfig` Tool == plifconfig is a management tool for network interfaces on !PlanetLab nodes. The tool can be used to be add new interfaces to a node, remove interfaces from a node, or show information about interfaces on a given node. == Secondary Interface IP Configuration Using `plifconfig` == === Variables === These variables will be used throughout this document as placeholders inside of installation and usage instructions. As you work through this document, replace the variables with values that are specific to your setup. || '''Variable''' || '''Description''' || '''Notes''' || || || The target plifconfig version || Example: plifconfig-2.0 || || || The version of python installed on your node || Examples: python2.5, python2.6 || || || FQDN of a target MyPLC-based !PlanetLab node || || || || The step between versions for a patch file || Example: 2.1_to_2.2 || === Set Up the Environment === ==== Obtain Scripts and Patches ==== Download the current version of the `plifconfig` package from [http://www.gpolab.bbn.com/local-sw/]. Copy the tarball onto your MyPLC machine, and unpackage it in your home directory: {{{ cd ~ tar xvzf /path/to/.tar.gz }}} ''Instructions below assume that you downloaded the package to your home directory on your MyPLC machine.'' ==== Add VLAN Tagtype to !PlanetLab Database on MyPLC Machine ==== To create VLAN subinterfaces, you need to add a new tag type to your !PlanetLab database. You should only do this if you have not already added the VLAN tagtype. * Confirm that you have not added the VLAN tag type yet: {{{ sudo ~//myplc/add_vlan_tagtype_to_plc_db -- -n echo $? }}} If '''1''' is printed out by the echo command then the VLAN tagtype already exists, and you should jump to [#ApplythePatchtoplnet.pyFileonAllNodes applying the patch]; otherwise, continue with this section. * Run the `add_vlan_tagtype_to_plc_db` script: {{{ sudo ~//myplc/add_vlan_tagtype_to_plc_db }}} * Confirm tagtype was added: {{{ sudo ~//myplc/add_vlan_tagtype_to_plc_db -- -n echo $? }}} If '''1''' is printed out by the echo command then the VLAN tagtype was successfully added to the !PlanetLab database. ==== Apply the Patch to `plnet.py` File on All Nodes ==== Make sure that you have a backup of the PlanetLab distributed originial `plnet.py`. * This may exist in `~/plnet.py.orig` if you followed previous instructions here. * If you have not patched the file before, then create a backup of the original file on your nodes. {{{ cp /usr/lib//site-packages/plnet.py ~/plnet.py.orig }}} * If you cannot find a copy of the original file on your MyPLC-based PlanetLab nodes, then you can also find one on the MyPLC machine under `/usr/lib//site-packages/plnet.py`. Save a copy of this file to `~/plnet.py.orig` on your nodes. ===== Option 1: Apply the Full Patch the Original File ===== #FullPatch If you are unsure which version of plifconfig you are running, or if you have not installed plifconfig before, you should use this option. * From your MyPLC machine, copy the original `plnet.py.orig` file over from your MyPLC-based !PlanetLab node. {{{ scp root@:~/plnet.py.orig ~/plnet.py }}} * On your MyPLC machine, apply the patch: {{{ patch ~/plnet.py < ~//nodes/plnet_full.patch }}} * If the patch succeeded, copy the newly patched file from your patch machine over to your MyPLC-based !PlanetLab node: {{{ scp ~/plnet.py root@:~/usr/lib//site-packages/ }}} * If the patch did not work please let us know at gpo-infra@geni.net. * Login to the MyPLC-based !PlanetLab node as root and restart !NodeManager: {{{ service nm restart }}} ===== Option 2: Patch a Previously Patched Version of plnet.py ===== #StepPatch If you know that you have the second-most recent version of plifconfig installed and a patch is included with this plifconfig version, then you can use this option. If you log into your MyPLC machine and run {{{ sudo plifconfig --version }}} then you can compare the version to the first number in . Note that the can be found by looking at `~//nodes/plnet_.patch`. * From your MyPLC machine, copy the `plnet.py` file over from your MyPLC-based !PlanetLab node. {{{ scp root@:/usr/lib//site-packages/plnet.py ~/ }}} * On your MyPLC machine, apply the patch: {{{ patch ~/plnet.py < ~//nodes/plnet_.patch }}} * If the patch succeeded, copy the newly patched file from your patch machine over to your MyPLC-based !PlanetLab node: {{{ scp ~/plnet.py root@:~/usr/lib//site-packages/ }}} * If the patch did not work please let us know at gpo-infra@geni.net. * Login to the MyPLC-based !PlanetLab node as root and restart !NodeManager: {{{ service nm restart }}} ==== Setup the `plifconfig` Tool ==== * Setup the plifconfig.py library on your MyPLC machine: {{{ cd ~//myplc/ sudo cp plifconfig.py /usr/local/lib/ sudo chown root /usr/local/lib/plifconfig.py sudo chmod 444 /usr/local/lib/plifconfig.py }}} * Setup the plifconfig wrapper script on your MyPLC machine: {{{ cd ~//myplc/ sudo cp plifconfig /usr/local/sbin/ sudo chown root /usr/local/sbin/plifconfig sudo chmod 544 /usr/local/sbin/plifconfig }}} You should now be finished setting up the environment for plifconfig. === Using the `plifconfig` Tool === ==== Getting Started ==== `plifconfig` currently supports 4 basic subcommands: * add: add an interface with statically defined values (DHCP not yet supported) * del: delete an interface * show: show detailed information about an interface * list: list all interfaces The tool has a help option (-h or --help) for plifconfig and all of its subcommands for more information on usage. {{{ [you@myplc] $ sudo plifconfig -h [you@myplc] $ sudo plifconfig add -h [you@myplc] $ sudo plifconfig del -h [you@myplc] $ sudo plifconfig show -h [you@myplc] $ sudo plifconfig list -h }}} ==== Typical `plifconfig` Workflows ==== ===== Adding an Interface to a Node ===== If you have more than one NIC attached to your host, you will need to add it to the !PlanetLab database on your MyPLC machine for it to be usable. In the example below, I will add `eth1` to `node2.fqdn.com`. Note that you can add interfaces without IP addresses by specifying an IP address of `0.0.0.0` and a netmask of `255.255.255.255`. Options in this example: ||'''Option'''||'''Description''' ||'''Notes''' || || -n || Name of node to modify || || || -i || IP address || || || -m || Netmask || || || -d || Device name || || || -g || Gateway IP address || You do not need to specify a gateway if the node is not on a routable network || || --mac || MAC address || This should only be specified for base devices (e.g. '''not''' for subinterfaces like aliases or VLAN interfaces) || {{{ [you@myplc] $ sudo plifconfig list Node: node1.fqdn.com IP: 125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0 IP: 1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1 IP: 10.2.3.3 HWaddr: Device: eth1:1234 Node: node2.fqdn.com IP: 125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0 [you@myplc] $ sudo plifconfig add -n node2.fqdn.com -i 10.2.3.6 -m 255.255.255.128 -d eth1 -g 10.2.3.1 --mac ab:12:cd:34:ef:99 Interface successfully created [you@myplc] $ sudo plifconfig list Node: node1.fqdn.com IP: 125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0 IP: 1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1 IP: 10.2.3.3 HWaddr: Device: eth1:1234 Node: node2.fqdn.com IP: 125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0 IP: 10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1 }}} ===== Adding an Alias or VLAN Interface to a Node: ===== You can create an alias or a VLAN interface in the same way that you create normal interfaces. The only difference is that you must pass a device name and a value for the VLAN id or the alias. The device must exist in the !PlanetLab database in order to create the alias or VLAN interface. ====== Adding a VLAN Interface to a Node ====== Options in this example: ||'''Option'''||'''Description''' ||'''Notes''' || || -n || Name of node to modify || || || -i || IP address || || || -m || Netmask || || || -d || Device name || || || -g || Gateway IP address || You do not need to specify a gateway if the node is not on a routable network || || --vlan || VLAN ID || Creates a VLAN interface, ALWAYS use in conjunction with -d || {{{ [you@myplc] $ sudo plifconfig list Node: node1.fqdn.com IP: 125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0 IP: 1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1 IP: 10.2.3.3 HWaddr: Device: eth1:1234 Node: node2.fqdn.com IP: 125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0 IP: 10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1 [you@myplc] $ sudo plifconfig add -n node2.fqdn.com -i 10.2.4.7 -m 255.255.255.128 -d eth1 --vlan 123 -g 10.2.4.1 Interface successfully created [you@myplc] $ sudo plifconfig list Node: node1.fqdn.com IP: 125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0 IP: 1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1 IP: 10.2.3.3 HWaddr: Device: eth1:1234 Node: node2.fqdn.com IP: 125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0 IP: 10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1 IP: 10.2.4.7 HWaddr: Device: eth1.123 }}} ====== Adding an Alias to a Node ====== Options in this example: ||'''Option'''||'''Description''' ||'''Notes''' || || -n || Name of node to modify || || || -i || IP address || || || -m || Netmask || || || -d || Device name || || || -g || Gateway IP address || You do not need to specify a gateway if the node is not on a routable network || || --alias || Alias name || Creates an interface alias, ALWAYS use in conjunction with -d || {{{ [you@myplc] $ sudo plifconfig list Node: node1.fqdn.com IP: 125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0 IP: 1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1 IP: 10.2.3.3 HWaddr: Device: eth1:1234 Node: node2.fqdn.com IP: 125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0 IP: 10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1 IP: 10.2.4.7 HWaddr: Device: eth1.123 [you@myplc] $ sudo plifconfig add -n node2.fqdn.com -i 10.2.4.8 -m 255.255.255.128 -d eth1 --alias 123 -g 10.2.4.1 Interface successfully created [you@myplc] $ sudo plifconfig list Node: node1.fqdn.com IP: 125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0 IP: 1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1 IP: 10.2.3.3 HWaddr: Device: eth1:1234 Node: node2.fqdn.com IP: 125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0 IP: 10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1 IP: 10.2.4.7 HWaddr: Device: eth1.123 IP: 10.2.4.8 HWaddr: Device: eth1:123 }}} ====== Adding a Alias of a VLAN Interface to a Node ====== Options in this example: ||'''Option'''||'''Description''' ||'''Notes''' || || -n || Name of node to modify || || || -i || IP address || || || -m || Netmask || || || -d || Device name || || || -g || Gateway IP address || You do not need to specify a gateway if the node is not on a routable network || || --vlan || VLAN ID || Creates a VLAN interface, ALWAYS use in conjunction with -d || || --alias || Alias name || Creates an interface alias, ALWAYS use in conjunction with -d || {{{ [you@myplc] $ sudo plifconfig list Node: node1.fqdn.com IP: 125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0 IP: 1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1 IP: 10.2.3.3 HWaddr: Device: eth1:1234 Node: node2.fqdn.com IP: 125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0 IP: 10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1 IP: 10.2.4.7 HWaddr: Device: eth1.123 IP: 10.2.4.8 HWaddr: Device: eth1:123 [you@myplc] $ sudo plifconfig add -n node2.fqdn.com -i 10.2.4.9 -m 255.255.255.128 -d eth1 --vlan 123 --alias 123 -g 10.2.4.1 Interface successfully created [you@myplc] $ sudo plifconfig list Node: node1.fqdn.com IP: 125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0 IP: 1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1 IP: 10.2.3.3 HWaddr: Device: eth1:1234 Node: node2.fqdn.com IP: 125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0 IP: 10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1 IP: 10.2.4.7 HWaddr: Device: eth1.123 IP: 10.2.4.8 HWaddr: Device: eth1:123 IP: 10.2.4.9 HWaddr: Device: eth1.123:123 }}} ===== Viewing Interface Details ===== Options in this example: ||'''Option'''||'''Description''' ||'''Notes''' || || -n || Name of node to modify || || || -d || Device name || || || --alias || Alias name || Creates an interface alias, ALWAYS use in conjunction with -d || To look up more information about a specific node, you can use the `show` subcommand. All of this information can also be found on the node or on the web interface, but the `show` subcommand is useful for quickly debugging issues from a central location. Note that when querying by device, you must specify the full set of {device, vlan, alias} of the device. For example, to see information on eth1, specify {{{ -d eth1 }}}. To see information on eth1.123:3, specify {{{ -d eth1 --vlan 123 --alias 3 }}}. To see information on eth1:3, specify {{{ -d eth1 --alias 3 }}}. {{{ [you@myplc] $ sudo plifconfig show -n node1.fqdn.com -d eth1 --alias 1234 Interface details for query: node = node1.fqdn.com, device = eth1:1234 Node: node1.fqdn.com Device: eth1:1234 HWaddr: Is Primary? False Bandwidth limit: None IP Version: ipv4 Method: static IP: 10.2.3.3 Bcast: 10.2.3.63 Mask: 255.255.255.192 NetID: 10.2.3.0 GW: 10.2.3.1 DNS1: 125.125.125.244 DNS2: }}} ===== Deleting Interfaces ===== If you need to remove an interface you can use the `del` subcommand. Note that when querying by device, you must specify the full set of {device, vlan, alias} of the device. For example, to see information on eth1, specify {{{ -d eth1 }}}. To see information on eth1.123:3, specify {{{ -d eth1 --vlan 123 --alias 3 }}}. To see information on eth1:3, specify {{{ -d eth1 --alias 3 }}}. Options in this example: ||'''Option'''||'''Description''' ||'''Notes''' || || -n || Name of node to modify || || || -d || Device name || || || --alias || Alias name || Creates an interface alias, ALWAYS use in conjunction with -d || {{{ [you@myplc] $ sudo plifconfig list Node: node1.fqdn.com IP: 125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0 IP: 1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1 IP: 10.2.3.3 HWaddr: Device: eth1:1234 Node: node2.fqdn.com IP: 125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0 IP: 10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1 IP: 10.2.4.7 HWaddr: Device: eth1.123 [you@myplc] $ sudo plifconfig del -n node1.fqdn.com -d eth1 --alias 1234 Interface successfully deleted [you@myplc] $ sudo plifconfig list Node: node1.fqdn.com IP: 125.125.125.88 HWaddr: ab:12:cd:34:ef:77 Device: eth0 IP: 1.2.3.5 HWaddr: ab:12:cd:34:ef:56 Device: eth1 Node: node2.fqdn.com IP: 125.125.125.3 HWaddr: ab:12:cd:34:ef:11 Device: eth0 IP: 10.2.3.6 HWaddr: ab:12:cd:34:ef:99 Device: eth1 IP: 10.2.4.7 HWaddr: Device: eth1.123 }}} ==== `plifconfig` Best Practices ==== * '''Always fully specify the interface you are operating on.''' There are three `plifconfig` flags, `-d`, `--vlan`, and `--alias`, which are used to uniquely identify a given device and subinterface. While it may be possible for the OS to fill in the `-d` flag correctly, you should always specify it when you invoke `plifconfig` to ensure that you are operating on the correct device. Whenever you are viewing or modifying information for a subinterface, always specify that subinterface fully. * '''Use MAC addresses when defining physical interfaces.''' Whenever you configure a physical interface (e.g. `eth1`) in `plifconfig`, specify the MAC address. This will ensure that device names do not change across node reboots. However, '''do not specify MAC addresses when defining virtual subinterfaces''' (whether VLAN- or alias-based) --- a [#plifconfigKnownBugs known bug] causes trouble when the same MAC address is used on multiple defined interfaces. * '''Be careful when adding or deleting interfaces.''' `plifconfig` does not ask for confirmation when adding or deleting interfaces. For example, `plifconfig` won't prevent you from deleting your primary interface. If you do this, you might not be able to change the interfaces on the node without doing a ''physical reboot'' of the node. * '''Manage interfaces from the MyPLC machine, not from the !PlanetLab node.''' If you try to manage interfaces on the node using traditional tools like `ifconfig` and `ifcfg-ethX` startup scripts, it will clash with the !PlanetLab way of creating interfaces. MyPLC wants to manage the interfaces actively, so use either `plifconfig` or the MyPLC web interface. * '''Use the fast sliver creation change described [http://groups.geni.net/geni/wiki/GpoLab/MyplcReferenceImplementation#SetupFastSliverCreation here].''' !PlanetLab nodes will create new slivers and new interfaces at the same time; thus, if you are only creating slivers every 15 minutes, then you will only be able to add or delete interfaces every 15 minutes. ==== `plifconfig` Known Bugs ==== * When using `plifconfig`, you can only specify a mac address once when adding interfaces. Although aliases and VLAN interfaces share the same mac address as their associated ethX parent interface, if you specify that mac address for all of these interfaces then unexpected behavior can occur on the node. For the time being, it is recommended to specify a mac address for the ethX interface, and to not specify mac addresses for VLAN interfaces or aliases. * Currently, `plifconfig` allows you to delete an interface that has subinterfaces attached to it. For example, if I have eth1, eth1.123, and eth1:3, `plifconfig` does not prevent you from deleting eth1. Similarly, plifconfig does not prevent you from adding a subinterface to a physical interface that is not already in the database. ==== `plifconfig` FAQ ==== 1. Q: Can I create multiple interfaces with the same IP address?[[BR]] '''A: Yes, but not on the same node. The only exception to this rule is having multiple interfaces with an IP address of 0.0.0.0.''' 2. Q: Interfaces are not showing up inside of my slices. How can I fix that?[[BR]] '''A: One suggested fix which may or may not apply to you is listed [http://groups.geni.net/geni/wiki/GpoLab/MyplcReferenceImplementation#ImportantNotesonPlanetLabNodeInterfaces here].[[BR]] Another suggestion is outlined below:''' * Log into your MyPLC's web interface * Click "Slices" in the left-hand navigation bar * Click on the name of the slice of your choice in the table on the slices page * On the page for your slice: * Expand the "tags" view * Select "ip_addresses" from the "Choose Tag" dropdown * Enter 0.0.0.0 into the text box as the value of the tag * Make sure either "All Nodes" or your specific node is selected to apply the tag to * Hit the "Set Tag" button to confirm * Log into the root context of your !PlanetLab node and run: {{{ service nm restart }}} * Log into the slice and confirm that the interfaces are visible 3. Q: How do I modify an interface that I have already added?[[BR]] '''A: There is currently no way to do this through `plifconfig` other than deleting and adding the interface again. You can still modify the interface's properties through the "interface" and "interface tags" pages of the MyPLC web interface.''' 4. Q: Can I set the bandwidth limit of an interface through `plifconfig`?[[BR]] '''A: Not currently. Please contact gpo-infra@geni.net if this feature would be useful in your environment.''' 5. Q: Does `plifconfig` support creation of DHCP interfaces?[[BR]] '''A: Not currently. Please contact gpo-infra@geni.net if this feature would be useful in your environment.''' 6. Q: Does `plifconfig` support creation of IPv6 interfaces?[[BR]] '''A: Not currently. Please contact gpo-infra@geni.net if this feature would be useful in your environment.''' == Interface numbering gotchas == PlanetLab nodes boot using a two-stage process. When multiple interface cards are present, the first-stage boot and second-stage boot may not number the interfaces consistently. This typically causes the node to be unusable. We don't know how to control which interface the first-stage boot will use as eth0. However, it is easy to use the MyPLC interface to control which MAC address should be assigned as each interface in the second-stage boot. This section describes how we force MyPLC to use the same interface allocation in second-stage boot as it did in first-stage, allowing the node to be used even if these settings do not natively match. === Step 1: Boot into first-stage with the control interface connected === Before booting the node, physically connect the control interface cable to the interface which first-stage boot will choose as `eth0`. On our lab nodes (1U Dell R300s), first-stage boot always chooses the leftmost interface on the leftmost expansion NIC to be `eth0` (or the leftmost interface on the motherboard if there are no expansion cards). In your environment, you may need to find this interface by guesswork: if the node can successfully enter first-stage boot and seems to reach a server, you guessed right. === Step 2: Use the first-stage boot logs to identify the MAC address of eth0 === The MyPLC gathers logs from each node's first-stage boot. These logs will contain the MAC address of first-stage `eth0` on your node. To find this, login to MyPLC, and do: {{{ sudo find /var/log/bm/raw | grep | sort | tail -1 | xargs sudo grep macs }}} That command should report output similar to this: {{{ $ sudo find /var/log/bm/raw | grep | sort | tail -1 | xargs sudo grep macs 19:48:22(UTC) net:InitInterfaces macs = {'00:00:00:00:00:00': 'lo', 'aa:bb:cc:dd:ee:ff': 'eth0'} }}} Use the reported MAC address: * ``: `aa:bb:cc:dd:ee:ff` === Step 3: Configure MyPLC to use the correct MAC in stage 2 === Use the PlanetLab UI to force the second-stage boot to use the leftmost interface on the leftmost expansion card: * Browse to the MyPLC web interface and login as an administrator * Click "My Site Nodes" in the left menubar * Click "``" in the table of nodes * Expand the "One interface" menu * Click the IP of the primary interface * Set MAC to `` * Click "Update" * Click "Back to Node" * Click "Update Node" * Reboot the node === Step 4: Modify udev to fix the live interface configuration === Sometimes, Fedora gets confused by eth0 switching identities, as a result of which eth1 or eth2 (if you have that many interfaces) does not appear, and is replaced by a weird `devNNNNN` interface. This appears to be fixable by modifying udev's configuration: * Login to the node as root * Edit `/etc/udev/rules.d/70-persistent-net.rules` * Find a configuration section containing the comment `(custom name provided by external tool)`, e.g.: {{{ # PCI device 0x8086:0x10c9 (igb) (custom name provided by external tool) SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="00:1b:21:4b:3f:ac", ATTR{type}=="1", NAME="eth0" }}} Make sure that there is only one such section, and that its MAC address matches `` * Delete all SUBSYSTEM comment-and-line blocks except this one, and save the file * Reboot the node