wiki:GENIRacksHome/InstageniRacks/AcceptanceTestStatus/IG-MON-3

Version 34 (modified by lnevers@bbn.com, 11 years ago) (diff)

--

IG-MON-3: GENI Active Experiment Inspection Test

This page captures status for the test case IG-MON-3, which verifies the ability to start, stop, and terminate experiments by experimenters and administrators. For overall status see the InstaGENI Acceptance Test Status page.

Last update: 2013-03-06

Test Status

Step State Tickets Notes
1 Color(green,Pass)?
2 Color(green,Pass)?
3 Color(green,Pass)?
4 Color(green,Pass)? 2013-02-28(35, 26) blocked on resolution of MAC reporting issue
5 Color(lightgreen,Pass: most criteria)? 2013-02-28 (26, 31) terminated experiments display format is a little hard to use, but contains the requested information
6 blocked on availability of OpenFlow functionality
7 ready to test non-OpenFlow functionality
8 ready to test non-OpenFlow functionality


State Legend Description
Color(green,Pass)? Test completed and met all criteria
Color(#98FB98,Pass: most criteria)? Test completed and met most criteria. Exceptions documented
Color(red,Fail)? Test completed and failed to meet criteria.
Color(yellow,Complete)? Test completed but will require re-execution due to expected changes
Color(orange,Blocked)? Blocked by ticketed issue(s).
Color(#63B8FF,In Progress)? Currently under test.


Test Plan Steps

  • An experimenter from the GPO starts up experiments to ensure there is data to look at:
    • An experimenter runs an experiment containing at least one rack OpenVZ VM, and terminates it.
    • An experimenter runs an experiment containing at least one rack OpenVZ VM, and leaves it running.
  • A site administrator uses available system and experiment data sources to determine current experimental state, including:
    • How many VMs are running and which experimenters own them
    • How many physical hosts are in use by experiments, and which experimenters own them
    • How many VMs were terminated within the past day, and which experimenters owned them
    • What OpenFlow controllers the data plane switch, the rack FlowVisor, and the rack FOAM are communicating with
  • A site administrator examines the switches and other rack data sources, and determines:
    • What MAC addresses are currently visible on the data plane switch and what experiments do they belong to?
    • For some experiment which was terminated within the past day, what data plane and control MAC and IP addresses did the experiment use?
    • For some experimental data path which is actively sending traffic on the data plane switch, do changes in interface counters show approximately the expected amount of traffic into and out of the switch?

Step 1. Start a VM experiment and leave it running

As experimenter create a 2 VMs experiment in the GPO InstaGENI rack. RSpec used:

<?xml version="1.0" encoding="UTF-8"?>
<!-- 
This example RSpec shows how to request 2 VMs at the GPO InstaGENI Rack. 
-->
<rspec type="request" 
	xmlns="http://www.geni.net/resources/rspec/3" 
	xmlns:flack="http://www.protogeni.net/resources/rspec/ext/flack/1" 
	xmlns:planetlab="http://www.planet-lab.org/resources/sfa/ext/planetlab/1" 
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
	xsi:schemaLocation="http://www.geni.net/resources/rspec/3 
	http://www.geni.net/resources/rspec/3/request.xsd">  

  <node client_id="VM-1" component_manager_id="urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm" exclusive="false">
    <sliver_type name="emulab-openvz"/>
    <interface client_id="VM-1:if0">
      <ip address="192.168.1.1" netmask="255.255.255.0" type="ipv4"/>
    </interface>
  </node>
  <node client_id="VM-2" component_manager_id="urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm" exclusive="false">
    <sliver_type name="emulab-openvz"/>
    <interface client_id="VM-2:if0">
      <ip address="192.168.1.2" netmask="255.255.255.0" type="ipv4"/>
    </interface>
  </node>
  <link client_id="lan0">
    <component_manager name="urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm"/>
    <interface_ref client_id="VM-1:if0"/>
    <interface_ref client_id="VM-2:if0"/>
    <link_type name="lan"/>
  </link>
</rspec>

Create a slice and sliver:

$ omni.py createslice EG-MON-3
INFO:omni:Loading config file /home/lnevers/.gcf/omni_config
INFO:omni:Using control framework pg
INFO:omni:Created slice with Name EG-MON-3, URN urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+EG-MON-3, Expiration 2013-03-07 15:09:24+00:00
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed createslice:
  Options as run:
		framework: pg

  Args: createslice EG-MON-3
  Result Summary: Created slice with Name EG-MON-3, URN urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+EG-MON-3, Expiration 2013-03-07 15:09:24+00:00
INFO:omni: ============================================================

$ omni.py createsliver -a ig-gpo EG-MON-3  ./instageni-2vm-at-gpo.rspec 
INFO:omni:Loading config file /home/lnevers/.gcf/omni_config
INFO:omni:Using control framework pg
INFO:omni:Substituting AM nickname ig-gpo with URL https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN
INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+EG-MON-3 expires on 2013-03-07 15:09:24 UTC
INFO:omni:Substituting AM nickname ig-gpo with URL https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN
INFO:omni:Substituting AM nickname ig-gpo with URL https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN
INFO:omni:Creating sliver(s) from rspec file ./instageni-2vm-at-gpo.rspec for slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+EG-MON-3
INFO:omni: (PG log url - look here for details on any failures: https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=388177e82b203e84aa3694dda7bf441d)
INFO:omni:Got return from CreateSliver for slice EG-MON-3 at https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0:
INFO:omni:<!-- Reserved resources for:
	Slice: EG-MON-3
	at AM:
	URN: unspecified_AM_URN
	URL: https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0
 -->
INFO:omni:<rspec xmlns="http://www.geni.net/resources/rspec/3" xmlns:flack="http://www.protogeni.net/resources/rspec/ext/flack/1" xmlns:planetlab="http://www.planet-lab.org/resources/sfa/ext/planetlab/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" type="manifest" xsi:schemaLocation="http://www.geni.net/resources/rspec/3   http://www.geni.net/resources/rspec/3/manifest.xsd">  

  <node client_id="VM-1" component_manager_id="urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm" exclusive="false" component_id="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc2" sliver_id="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2048">
    <sliver_type name="emulab-openvz"/>
    <interface client_id="VM-1:if0" component_id="urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc2:lo0" sliver_id="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2052" mac_address="02104639f347">
      <ip address="192.168.1.1" netmask="255.255.255.0" type="ipv4"/>
    </interface>
  <rs:vnode xmlns:rs="http://www.protogeni.net/resources/rspec/ext/emulab/1" name="pcvm2-1"/><host name="VM-1.EG-MON-3.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com"/><services><login authentication="ssh-keys" hostname="pc2.instageni.gpolab.bbn.com" port="31290" username="lnevers"/></services></node>
  <node client_id="VM-2" component_manager_id="urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm" exclusive="false" component_id="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc2" sliver_id="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2049">
    <sliver_type name="emulab-openvz"/>
    <interface client_id="VM-2:if0" component_id="urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc2:lo0" sliver_id="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2053" mac_address="020063c74361">
      <ip address="192.168.1.2" netmask="255.255.255.0" type="ipv4"/>
    </interface>
  <rs:vnode xmlns:rs="http://www.protogeni.net/resources/rspec/ext/emulab/1" name="pcvm2-6"/><host name="VM-2.EG-MON-3.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com"/><services><login authentication="ssh-keys" hostname="pc2.instageni.gpolab.bbn.com" port="31291" username="lnevers"/></services></node>
  <link client_id="lan0" sliver_id="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2051">
    <component_manager name="urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm"/>
    <interface_ref client_id="VM-1:if0" component_id="urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc2:lo0" sliver_id="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2052"/>
    <interface_ref client_id="VM-2:if0" component_id="urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc2:lo0" sliver_id="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2053"/>
    <link_type name="lan"/>
  </link>
</rspec>
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed createsliver:
  Options as run:
		aggregate: ['ig-gpo']
		framework: pg
  Args: createsliver EG-MON-3 ./instageni-2vm-at-gpo.rspec
  Result Summary: Got Reserved resources RSpec from instageni-gpolab-bbn-com-protogeniv2 
INFO:omni: ============================================================

Determine login information for the VM resources assigned and that the VMs are "ready" for use:

$ readyToLogin.py -a ig-gpo EG-MON-3                               
<...>
VM-2's geni_status is: ready (am_status:ready) 
User lnevers logins to VM-2 using:
	xterm -e ssh -p 31291  -i /home/lnevers/.ssh/id_rsa lnevers@pc2.instageni.gpolab.bbn.com &

VM-1's geni_status is: ready (am_status:ready) 
User lnevers logins to VM-1 using:
	xterm -e ssh -p 31290  -i /home/lnevers/.ssh/id_rsa lnevers@pc2.instageni.gpolab.bbn.com &

Login to VM-1, verify hostname, address assignment and initial ARP table entries:

$ ssh -p 31290  -i /home/lnevers/.ssh/id_rsa lnevers@pc2.instageni.gpolab.bbn.com 

[lnevers@VM-1 ~]$ /sbin/ifconfig mv1.12
mv1.12    Link encap:Ethernet  HWaddr 02:10:46:39:F3:47  
          inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::10:46ff:fe39:f347/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:6 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:552 (552.0 b)  TX bytes:0 (0.0 b)

[lnevers@VM-1 ~]$ /sbin/arp -a
pc2.instageni.gpolab.bbn.com (192.1.242.141) at 10:60:4b:9b:82:14 [ether] on eth999
boss.instageni.gpolab.bbn.com (192.1.242.132) at 02:9e:26:f1:52:99 [ether] on eth999

Login to VM-2, verify hostname, address assignment, MAC address for data plane interface, and initial ARP table entries:

$ ssh -p 31291  -i /home/lnevers/.ssh/id_rsa lnevers@pc2.instageni.gpolab.bbn.com 
[lnevers@VM-2 ~]$ /sbin/ifconfig mv6.13
mv6.13    Link encap:Ethernet  HWaddr 02:00:63:C7:43:61  
          inet addr:192.168.1.2  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::63ff:fec7:4361/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:6 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

[lnevers@VM-2 ~]$ /sbin/arp -a
pc2.instageni.gpolab.bbn.com (192.1.242.141) at 10:60:4b:9b:82:14 [ether] on eth999
boss.instageni.gpolab.bbn.com (192.1.242.132) at 02:9e:26:f1:52:99 [ether] on eth999

Install iperf on both VMs and exchange TCP traffic from VM-1 client to VM-2 server over the data plane interface and look at ARP table entries:

[lnevers@VM-1 ~]$ /usr/bin/iperf -c 192.168.1.2 -t 60
------------------------------------------------------------
Client connecting to 192.168.1.2, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.1 port 55185 connected with 192.168.1.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec   687 MBytes  96.0 Mbits/sec

[lnevers@VM-1 ~]$  /sbin/arp -a
pc2.instageni.gpolab.bbn.com (192.1.242.141) at 10:60:4b:9b:82:14 [ether] on eth999
boss.instageni.gpolab.bbn.com (192.1.242.132) at 02:9e:26:f1:52:99 [ether] on eth999
VM-2-lan0 (192.168.1.2) at 02:00:63:c7:43:61 [ether] on mv1.12

Show ARP entries on VM2 (iperf server):

[lnevers@VM-2 ~]$ /sbin/arp -a
pc2.instageni.gpolab.bbn.com (192.1.242.141) at 10:60:4b:9b:82:14 [ether] on eth999
boss.instageni.gpolab.bbn.com (192.1.242.132) at 02:9e:26:f1:52:99 [ether] on eth999
VM-1-lan0 (192.168.1.1) at 02:10:46:39:f3:47 [ether] on mv6.13

Leave experiment running.

Step 2. Start a bare metal node experiment and leave it running

As experimenter create a 2 Raw PCs experiment in the GPO InstaGENI rack. RSpec used:

<?xml version="1.0" encoding="UTF-8"?>
<rspec type="request" 
	xmlns="http://www.geni.net/resources/rspec/3" 
	xmlns:flack="http://www.protogeni.net/resources/rspec/ext/flack/1" 
	xmlns:planetlab="http://www.planet-lab.org/resources/sfa/ext/planetlab/1" 
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
	xsi:schemaLocation="http://www.geni.net/resources/rspec/3 
	http://www.geni.net/resources/rspec/3/request.xsd">  

  <node client_id="PC-1" component_id="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc5"          
        component_manager_id="urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm" exclusive="true">
    <sliver_type name="raw-pc"/>
    <interface client_id="PC-1:if0">
      <ip address="192.168.1.1" netmask="255.255.255.0" type="ipv4"/>
    </interface>
  </node>

  <node client_id="PC-2" component_id="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc4"  
        component_manager_id="urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm" exclusive="true">
    <sliver_type name="raw-pc"/>
    <interface client_id="PC-2:if0">
      <ip address="192.168.1.2" netmask="255.255.255.0" type="ipv4"/>
    </interface>
  </node>
  <link client_id="lan0">
    <component_manager name="urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm"/>
    <interface_ref client_id="PC-1:if0"/>
    <interface_ref client_id="PC-2:if0"/>
    <link_type name="lan"/>
  </link>
</rspec>

Create a slice and sliver:

$ omni.py createslice IG-MON-3
INFO:omni:Loading config file /home/lnevers/.gcf/omni_config
INFO:omni:Using control framework pg
INFO:omni:Created slice with Name IG-MON-3, URN urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3, Expiration 2013-03-07 15:47:53+00:00
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed createslice:
  Options as run:
		framework: pg

  Args: createslice IG-MON-3
  Result Summary: Created slice with Name IG-MON-3, URN urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3, Expiration 2013-03-07 15:47:53+00:00
INFO:omni: ============================================================

$ omni.py createsliver -a ig-gpo IG-MON-3 ./instageni-2rawpc-at-gpo.rspec 
INFO:omni:Loading config file /home/lnevers/.gcf/omni_config
INFO:omni:Using control framework pg
INFO:omni:Substituting AM nickname ig-gpo with URL https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN
INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3 expires on 2013-03-07 15:47:53 UTC
INFO:omni:Substituting AM nickname ig-gpo with URL https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN
INFO:omni:Substituting AM nickname ig-gpo with URL https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN
INFO:omni:Creating sliver(s) from rspec file ./instageni-2rawpc-at-gpo.rspec for slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3
INFO:omni: (PG log url - look here for details on any failures: https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=deb52735b826a50320d5274e8e4c6c8e)
INFO:omni:Got return from CreateSliver for slice IG-MON-3 at https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0:
INFO:omni:<!-- Reserved resources for:
	Slice: IG-MON-3
	at AM:
	URN: unspecified_AM_URN
	URL: https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0
 -->
INFO:omni:<rspec xmlns="http://www.geni.net/resources/rspec/3" xmlns:flack="http://www.protogeni.net/resources/rspec/ext/flack/1" xmlns:planetlab="http://www.planet-lab.org/resources/sfa/ext/planetlab/1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" type="manifest" xsi:schemaLocation="http://www.geni.net/resources/rspec/3   http://www.geni.net/resources/rspec/3/manifest.xsd">  

  <node client_id="PC-1" component_id="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc5" component_manager_id="urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm" exclusive="true" sliver_id="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2042">
    <sliver_type name="raw-pc"/>
    <interface client_id="PC-1:if0" component_id="urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc5:eth1" sliver_id="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2046" mac_address="10604B9C476A">
      <ip address="192.168.1.1" netmask="255.255.255.0" type="ipv4"/>
    </interface>
  <rs:vnode xmlns:rs="http://www.protogeni.net/resources/rspec/ext/emulab/1" name="pc5"/><host name="PC-1.IG-MON-3.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com"/><services><login authentication="ssh-keys" hostname="pc5.instageni.gpolab.bbn.com" port="22" username="lnevers"/></services></node>

  <node client_id="PC-2" component_id="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc4" component_manager_id="urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm" exclusive="true" sliver_id="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2043">
    <sliver_type name="raw-pc"/>
    <interface client_id="PC-2:if0" component_id="urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc4:eth1" sliver_id="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2047" mac_address="10604B9600D6">
      <ip address="192.168.1.2" netmask="255.255.255.0" type="ipv4"/>
    </interface>
  <rs:vnode xmlns:rs="http://www.protogeni.net/resources/rspec/ext/emulab/1" name="pc4"/><host name="PC-2.IG-MON-3.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com"/><services><login authentication="ssh-keys" hostname="pc4.instageni.gpolab.bbn.com" port="22" username="lnevers"/></services></node>
  <link client_id="lan0" sliver_id="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2045" vlantag="258">
    <component_manager name="urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm"/>
    <interface_ref client_id="PC-1:if0" component_id="urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc5:eth1" sliver_id="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2046"/>
    <interface_ref client_id="PC-2:if0" component_id="urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc4:eth1" sliver_id="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2047"/>
    <link_type name="lan"/>
  </link>
</rspec>
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed createsliver:
  Options as run:
		aggregate: ['ig-gpo']
		framework: pg
  Args: createsliver IG-MON-3 ./instageni-2rawpc-at-gpo.rspec
  Result Summary: Got Reserved resources RSpec from instageni-gpolab-bbn-com-protogeniv2 
INFO:omni: ============================================================

Determine login information for the sliver resources:

$ readyToLogin.py -a ig-gpo IG-MON-3                   
<...>
PC-1's geni_status is: ready (am_status:ready) 
User lnevers logins to PC-1 using:
	xterm -e ssh -i /home/lnevers/.ssh/id_rsa lnevers@pc5.instageni.gpolab.bbn.com &

PC-2's geni_status is: ready (am_status:ready) 
User lnevers logins to PC-2 using:
	xterm -e ssh -i /home/lnevers/.ssh/id_rsa lnevers@pc4.instageni.gpolab.bbn.com &

Login to PC-1, verify hostname, address assignment and initial ARP table entries:

$ ssh -i /home/lnevers/.ssh/id_rsa lnevers@pc5.instageni.gpolab.bbn.com
[lnevers@pc-1 ~]$ /sbin/ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 10:60:4B:9C:47:6A  
          inet addr:192.168.1.1  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::1260:4bff:fe9c:476a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:492 (492.0 b)
          Interrupt:37 Memory:f2000000-f2012800 

[lnevers@pc-1 ~]$ /sbin/arp -a
ops.instageni.gpolab.bbn.com (192.1.242.133) at 02:07:2d:51:f4:6a [ether] on eth0
boss.instageni.gpolab.bbn.com (192.1.242.132) at 02:9e:26:f1:52:99 [ether] on eth0
? (192.1.242.129) at cc:ef:48:7a:7a:a9 [ether] on eth0

Login to PC-2, verify hostname, address assignment and initial ARP table entries:

$ ssh -i /home/lnevers/.ssh/id_rsa lnevers@pc4.instageni.gpolab.bbn.com 
[lnevers@pc-2 ~]$ /sbin/ifconfig eth1
eth1      Link encap:Ethernet  HWaddr 10:60:4B:96:00:D6  
          inet addr:192.168.1.2  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::1260:4bff:fe96:d6/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 b)  TX bytes:492 (492.0 b)
          Interrupt:37 Memory:f2000000-f2012800 

[lnevers@pc-2 ~]$ /sbin/arp -a
ops.instageni.gpolab.bbn.com (192.1.242.133) at 02:07:2d:51:f4:6a [ether] on eth0
boss.instageni.gpolab.bbn.com (192.1.242.132) at 02:9e:26:f1:52:99 [ether] on eth0
? (192.1.242.129) at cc:ef:48:7a:7a:a9 [ether] on eth0

Install iperf on both PCs and exchange TCP traffic from PC-1 client to PC-2 server over the data plane interface and look at ARP table entries:

[lnevers@pc-1 ~]$ /usr/bin/iperf -c 192.168.1.2 -i 1
------------------------------------------------------------
Client connecting to 192.168.1.2, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.1.1 port 58436 connected with 192.168.1.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 1.0 sec   113 MBytes   947 Mbits/sec
[  3]  1.0- 2.0 sec   112 MBytes   940 Mbits/sec
[  3]  2.0- 3.0 sec   112 MBytes   943 Mbits/sec
[  3]  3.0- 4.0 sec   112 MBytes   942 Mbits/sec
[  3]  4.0- 5.0 sec   112 MBytes   940 Mbits/sec

Leave sliver running.

Step 3. Start an OpenFlow experiment and leave it running

As experimenter create a 2 VMs OpenFlow experiment in the GPO InstaGENI rack. RSpec used:

<?xml version="1.0" encoding="UTF-8"?>
<rspec xmlns="http://www.geni.net/resources/rspec/3"
       xmlns:xs="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:sharedvlan="http://www.protogeni.net/resources/rspec/ext/shared-vlan/1"
       xs:schemaLocation="http://www.geni.net/resources/rspec/3
           http://www.geni.net/resources/rspec/3/request.xsd
           http://www.protogeni.net/resources/rspec/ext/shared-vlan/1
           http://www.protogeni.net/resources/rspec/ext/shared-vlan/1/request.xsd"
       type="request">

  <node client_id="gpo-ig" exclusive="false">
    <sliver_type name="emulab-openvz" />
    <interface client_id="gpo-ig:if0">
      <ip address="10.42.18.43" netmask="255.255.255.0" type="ipv4" />
    </interface>
  </node>

  <node client_id="gpo-ig2" exclusive="false">
    <sliver_type name="emulab-openvz" />
    <interface client_id="gpo-ig2:if0">
      <ip address="10.42.18.42" netmask="255.255.255.0" type="ipv4" />
    </interface>
  </node>

  <link client_id="openflow-mesoscale-0">
    <interface_ref client_id="gpo-ig:if0" />
    <sharedvlan:link_shared_vlan name="mesoscale-openflow" />
  </link>

  <link client_id="openflow-mesoscale-1">
    <interface_ref client_id="gpo-ig2:if0" />
    <sharedvlan:link_shared_vlan name="mesoscale-openflow" />
  </link>
</rspec>

Create a sliver and sliver for the 2 VMs on shared OpenFlow VLAN 1750 in the GPO InstaGENI rack:

$ omni.py createslice IG-MON-3-OF
INFO:omni:Loading config file /home/lnevers/.gcf/omni_config
INFO:omni:Using control framework pg
INFO:omni:Created slice with Name IG-MON-3-OF, URN urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF, Expiration 2013-03-07 16:26:04+00:00
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed createslice:
  Options as run:
		framework: pg

  Args: createslice IG-MON-3-OF
  Result Summary: Created slice with Name IG-MON-3-OF, URN urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF, Expiration 2013-03-07 16:26:04+00:00
INFO:omni: ============================================================


$ omni.py createsliver -a ig-gpo IG-MON-3-OF ./instageni-2vm-vlan1750-at-gpo.rspec 
INFO:omni:Loading config file /home/lnevers/.gcf/omni_config
INFO:omni:Using control framework pg
INFO:omni:Substituting AM nickname ig-gpo with URL https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN
INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF expires on 2013-03-07 16:26:04 UTC
INFO:omni:Substituting AM nickname ig-gpo with URL https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN
INFO:omni:Substituting AM nickname ig-gpo with URL https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN
INFO:omni:Creating sliver(s) from rspec file ./instageni-2vm-vlan1750-at-gpo.rspec for slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF
INFO:omni: (PG log url - look here for details on any failures: https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=8af1230117ca9b383b6510da21d7abc7)
INFO:omni:Got return from CreateSliver for slice IG-MON-3-OF at https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0:
INFO:omni:<!-- Reserved resources for:
	Slice: IG-MON-3-OF
	at AM:
	URN: unspecified_AM_URN
	URL: https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0
 -->
INFO:omni:<rspec xmlns="http://www.geni.net/resources/rspec/3" xmlns:xs="http://www.w3.org/2001/XMLSchema-instance" xmlns:sharedvlan="http://www.protogeni.net/resources/rspec/ext/shared-vlan/1" xs:schemaLocation="http://www.geni.net/resources/rspec/3            http://www.geni.net/resources/rspec/3/manifest.xsd            http://www.protogeni.net/resources/rspec/ext/shared-vlan/1            http://www.protogeni.net/resources/rspec/ext/shared-vlan/1/request.xsd" type="manifest">

  <node client_id="gpo-ig" exclusive="false" component_id="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc2" component_manager_id="urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm" sliver_id="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2065">
    <sliver_type name="emulab-openvz"/>
    <interface client_id="gpo-ig:if0" component_id="urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc2:eth1" sliver_id="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2069" mac_address="02e922fceb01">
      <ip address="10.42.18.43" netmask="255.255.255.0" type="ipv4"/>
    </interface>
  <rs:vnode xmlns:rs="http://www.protogeni.net/resources/rspec/ext/emulab/1" name="pcvm2-9"/><host name="gpo-ig.IG-MON-3-OF.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com"/><services><login authentication="ssh-keys" hostname="pc2.instageni.gpolab.bbn.com" port="32058" username="lnevers"/></services></node>

  <node client_id="gpo-ig2" exclusive="false" component_id="urn:publicid:IDN+instageni.gpolab.bbn.com+node+pc2" component_manager_id="urn:publicid:IDN+instageni.gpolab.bbn.com+authority+cm" sliver_id="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2066">
    <sliver_type name="emulab-openvz"/>
    <interface client_id="gpo-ig2:if0" component_id="urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc2:eth2" sliver_id="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2071" mac_address="0240e5291a6f">
      <ip address="10.42.18.42" netmask="255.255.255.0" type="ipv4"/>
    </interface>
  <rs:vnode xmlns:rs="http://www.protogeni.net/resources/rspec/ext/emulab/1" name="pcvm2-10"/><host name="gpo-ig2.IG-MON-3-OF.pgeni-gpolab-bbn-com.instageni.gpolab.bbn.com"/><services><login authentication="ssh-keys" hostname="pc2.instageni.gpolab.bbn.com" port="32059" username="lnevers"/></services></node>

  <link xmlns:sharedvlan="http://www.protogeni.net/resources/rspec/ext/shared-vlan/1" client_id="openflow-mesoscale-0" sliver_id="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2068">
    <interface_ref client_id="gpo-ig:if0" component_id="urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc2:eth1" sliver_id="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2069"/>
    <sharedvlan:link_shared_vlan name="mesoscale-openflow"/>
  </link>

  <link xmlns:sharedvlan="http://www.protogeni.net/resources/rspec/ext/shared-vlan/1" client_id="openflow-mesoscale-1" sliver_id="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2070">
    <interface_ref client_id="gpo-ig2:if0" component_id="urn:publicid:IDN+instageni.gpolab.bbn.com+interface+pc2:eth2" sliver_id="urn:publicid:IDN+instageni.gpolab.bbn.com+sliver+2071"/>
    <sharedvlan:link_shared_vlan name="mesoscale-openflow"/>
  </link>
</rspec>
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed createsliver:
  Options as run:
		aggregate: ['ig-gpo']
		framework: pg
  Args: createsliver IG-MON-3-OF ./instageni-2vm-vlan1750-at-gpo.rspec
  Result Summary: Got Reserved resources RSpec from instageni-gpolab-bbn-com-protogeniv2 
INFO:omni: ============================================================

A FOAM sliver is needed to allow the traffic exchange. Using this FOAM RSpec:

<?xml version="1.0" encoding="UTF-8"?>
<rspec xmlns="http://www.geni.net/resources/rspec/3"
       xmlns:xs="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:openflow="http://www.geni.net/resources/rspec/ext/openflow/3"
       xs:schemaLocation="http://www.geni.net/resources/rspec/3
           http://www.geni.net/resources/rspec/3/request.xsd
           http://www.geni.net/resources/rspec/ext/openflow/3
           http://www.geni.net/resources/rspec/ext/openflow/3/of-resv.xsd"
       type="request">

  <openflow:sliver description=" InstaGENI OpenFlow" email="lnevers@bbn.com">
    <openflow:controller url="tcp:mallorea.gpolab.bbn.com:33018" type="primary" />

    <openflow:group name="bbn-instageni-1750">
      <openflow:datapath component_id="urn:publicid:IDN+openflow:foam:foam.instageni.gpolab.bbn.com+datapath+06:d6:84:34:97:c6:c9:00" 
	component_manager_id="urn:publicid:IDN+openflow:foam:foam.instageni.gpolab.bbn.com+authority+am"  />
    </openflow:group>

    <openflow:match>

      <openflow:use-group name="bbn-instageni-1750" />

      <openflow:packet>
        <openflow:dl_type value="0x800,0x806"/>
        <openflow:nw_dst value="10.42.18.0/24"/>
        <openflow:nw_src value="10.42.18.0/24"/>
      </openflow:packet>
    </openflow:match>

  </openflow:sliver>
</rspec>

Create a sliver at the GPO InstaGENI rack FOAM to allow the traffic exchange:

$ omni.py createsliver -a ig-of-gpo IG-MON-3-OF ./instageni-openflow-at-gpo.rspec     
INFO:omni:Loading config file /home/lnevers/.gcf/omni_config
INFO:omni:Using control framework pg
INFO:omni:Substituting AM nickname ig-of-gpo with URL https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1, URN unspecified_AM_URN
WARNING:omni:You asked to use AM API 2, but the AM(s) you are contacting do not all speak that version.
WARNING:omni:At the URLs you are contacting, all your AMs speak AM API v1. 
WARNING:omni:Switching to AM API v1. Next time call Omni with '-V1'.
INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF expires on 2013-03-07 16:26:04 UTC
INFO:omni:Substituting AM nickname ig-of-gpo with URL https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1, URN unspecified_AM_URN
INFO:omni:Substituting AM nickname ig-of-gpo with URL https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1, URN unspecified_AM_URN
INFO:omni:Creating sliver(s) from rspec file ./instageni-openflow-at-gpo.rspec for slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF
INFO:omni:Got return from CreateSliver for slice IG-MON-3-OF at https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1:
INFO:omni:<?xml version="1.0" encoding="UTF-8"?>
INFO:omni:  <!-- Reserved resources for:
	Slice: IG-MON-3-OF
	at AM:
	URN: unspecified_AM_URN
	URL: https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1
 -->
INFO:omni:  
<rspec xmlns="http://www.geni.net/resources/rspec/3"
       xmlns:xs="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:openflow="http://www.geni.net/resources/rspec/ext/openflow/3"
       xs:schemaLocation="http://www.geni.net/resources/rspec/3
           http://www.geni.net/resources/rspec/3/manifest.xsd
           http://www.geni.net/resources/rspec/ext/openflow/3
           http://www.geni.net/resources/rspec/ext/openflow/3/of-resv.xsd"
       type="manifest">

  <openflow:sliver description=" InstaGENI OpenFlow" email="lnevers@bbn.com">
    <openflow:controller url="tcp:mallorea.gpolab.bbn.com:33018" type="primary" />

    <openflow:group name="bbn-instageni-1750">
      <openflow:datapath component_id="urn:publicid:IDN+openflow:foam:foam.instageni.gpolab.bbn.com+datapath+06:d6:84:34:97:c6:c9:00" 
	component_manager_id="urn:publicid:IDN+openflow:foam:foam.instageni.gpolab.bbn.com+authority+am"  />
    </openflow:group>

    <openflow:match>

      <openflow:use-group name="bbn-instageni-1750" />

      <openflow:packet>
        <openflow:dl_type value="0x800,0x806"/>
        <openflow:nw_dst value="10.42.18.0/24"/>
        <openflow:nw_src value="10.42.18.0/24"/>
      </openflow:packet>

    </openflow:match>

  </openflow:sliver>

</rspec>


INFO:omni: ------------------------------------------------------------
INFO:omni: Completed createsliver:

  Options as run:
		aggregate: ['ig-of-gpo']
		api_version: 1
		framework: pg

  Args: createsliver IG-MON-3-OF ./instageni-openflow-at-gpo.rspec

  Result Summary: Your AMs do not all speak requested API v2. At the URLs you are contacting, all your AMs speak AM API v1. Switching to AM API v1. Next time call Omni with '-V1'.
Got Reserved resources RSpec from foam-instageni-gpolab-bbn-com 
INFO:omni: ============================================================

This sliver is auto approved. State can be confirmed with omni command:

$ omni.py sliverstatus -a ig-of-gpo IG-MON-3-OF -V1                               
INFO:omni:Loading config file /home/lnevers/.gcf/omni_config
INFO:omni:Using control framework pg
INFO:omni:Substituting AM nickname ig-of-gpo with URL https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1, URN unspecified_AM_URN
INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF expires on 2013-03-07 16:26:04 UTC
INFO:omni:Substituting AM nickname ig-of-gpo with URL https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1, URN unspecified_AM_URN
INFO:omni:Status of Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF:
INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF at AM https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 has overall SliverStatus: ready
INFO:omni:Sliver status for Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF at AM URL https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1
INFO:omni:{
  "geni_status": "ready", 
  "geni_urn": "urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF:8070b745-84c7-47eb-8e04-63a764997f3a", 
  "foam_pend_reason": null, 
  "foam_expires": "2013-03-07 16:26:04+00:00", 
  "geni_resources": [
    {
      "geni_urn": "urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF:8070b745-84c7-47eb-8e04-63a764997f3a", 
      "geni_error": "", 
      "geni_status": "ready"
    }
  ], 
  "foam_status": "Approved"
}
INFO:omni: ------------------------------------------------------------
INFO:omni: Completed sliverstatus:
  Options as run:
		aggregate: ['ig-of-gpo']
		api_version: 1
		framework: pg
  Args: sliverstatus IG-MON-3-OF
  Result Summary: Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF expires on 2013-03-07 16:26:04 UTC
Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF at AM https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 has overall SliverStatus: ready.
 Returned status of slivers on 1 of 1 possible aggregates. 
INFO:omni: ============================================================

Determine login for assigned nodes:

$ readyToLogin.py -a ig-gpo IG-MON-3-OF                
<...>
gpo-ig's geni_status is: ready (am_status:ready) 
User lnevers logins to gpo-ig using:
	xterm -e ssh -p 32058  -i /home/lnevers/.ssh/id_rsa lnevers@pc2.instageni.gpolab.bbn.com &

gpo-ig2's geni_status is: ready (am_status:ready) 
User lnevers logins to gpo-ig2 using:
	xterm -e ssh -p 32059  -i /home/lnevers/.ssh/id_rsa lnevers@pc2.instageni.gpolab.bbn.com &

Login to VM named 'gpo-ig' and send traffic to remote:

$ ssh -p 32058  -i /home/lnevers/.ssh/id_rsa lnevers@pc2.instageni.gpolab.bbn.com 

[lnevers@ig-gpo ~]$ ping 10.42.18.42 -c 5
PING 10.42.18.42 (10.42.18.42) 56(84) bytes of data.
64 bytes from 10.42.18.42: icmp_req=1 ttl=64 time=6.73 ms
64 bytes from 10.42.18.42: icmp_req=2 ttl=64 time=0.077 ms
64 bytes from 10.42.18.42: icmp_req=3 ttl=64 time=0.075 ms
64 bytes from 10.42.18.42: icmp_req=4 ttl=64 time=0.075 ms
64 bytes from 10.42.18.42: icmp_req=5 ttl=64 time=0.075 ms

--- 10.42.18.42 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4001ms
rtt min/avg/max/mdev = 0.075/1.407/6.735/2.664 ms

Login to VM named 'gpo-ig2' and send traffic to remote:

$ ssh -p 32059  -i /home/lnevers/.ssh/id_rsa lnevers@pc2.instageni.gpolab.bbn.com

[lnevers@ig-gpo2 ~]$ ping 10.42.18.43 -c 5
PING 10.42.18.43 (10.42.18.43) 56(84) bytes of data.
64 bytes from 10.42.18.43: icmp_req=1 ttl=64 time=6.33 ms
64 bytes from 10.42.18.43: icmp_req=2 ttl=64 time=0.075 ms
64 bytes from 10.42.18.43: icmp_req=3 ttl=64 time=0.074 ms
64 bytes from 10.42.18.43: icmp_req=4 ttl=64 time=0.076 ms
64 bytes from 10.42.18.43: icmp_req=5 ttl=64 time=0.074 ms

--- 10.42.18.43 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4000ms
rtt min/avg/max/mdev = 0.074/1.327/6.337/2.505 ms
[lnevers@ig-gpo2 ~]$ 

Leave experiment running

Step 4. Administrator review of running experiments

Three experiment are running from previous steps:

  • EG-MON-3 - a 2 VM experiment
  • IG-MON-3 - a 2 raw pc experiment
  • IG-MON-3-OF - a 2 VM OpenFlow experiment

As administrator determine experiments status, resource assignment, and configuration for the 3 experiments.

Determine list of active experiments, by selecting "Experimentation" pulldown in "red dot" mode, and choosing "Experiment list", which maps to the URL https://boss.instageni.gpolab.bbn.com/showexp_list.php3. A capture of the active experiment page shows:

Each of the 3 experiment created for this test case (EG-MON-3, IG-MON-3, and IG-MON-3-OF) are present in the list.
Select the experiment EG-MON-3 (2 VM experiment) and this shows show all know details for the experiment:

Select the experiment IG-MON-3 (2 raw pc experiment) and this shows show all know details for the experiment:

Select the experiment IG-MON-3-OF (2 VM OpenFlow experiment)and this shows show all know details for the experiment:

For each of the above experiment clicking on the topology diagram on the left hand side panel will result in a details page that includes specific configuration and allocation for the experiment.

Clicking on the topology for the experiment IG-MON-3 (2 raw pc experiment) show:

The above shows node mapping, IP Port allocation, and other useful information.

For the OpenFlow experiment, the

  { 'attributes':
    { 'client_id': 'phys1:if0',
      'component_id': 'urn:publicid:IDN+utah.geniracks.net+interface+pc3:eth1',
      'mac_address': 'e83935b14e8a',
...
  { 'attributes':
    { 'client_id': 'virt1:if0',
      'component_id': 'urn:publicid:IDN+utah.geniracks.net+interface+pc5:eth1',
      'mac_address': '00000a0a0102',
  • So i think it should be possible for the admin interface to know that virtual mac address too.
  • Huh, but also, that mac address reported in sliverstatus is in fact wrong. Let me summarize:
    MAC addrs reported for phys1:0 == 10.10.1.1
      E8:39:35:B1:4E:8A: from /sbin/ifconfig eth1 run on phys1 (authoritative)
      e83935b14e8a:      from sliverstatus as experimenter (correct)
      e8:39:35:b1:4e:8a: from: https://boss.utah.geniracks.net/showexp.php3?experiment=363#details (correct)
    
    MAC addrs reported for virt1:0 == 10.10.1.2
      82:01:0A:0A:01:02: from /sbin/ifconfig mv1.1 run on virt1 (authoritative)
      00000a0a0102:      from sliverstatus as experimenter (incorrect: first four digits are wrong)
      -                : from https://boss.utah.geniracks.net/showexp.php3?experiment=363#details (not reported)
    
    I opened 26 for this issue.
  • Now, use the OpenVZ host itself to view activity:
    • As an admin, login to pc5.utah.geniracks.net
    • Poking around, i was led to a couple of prospective data sources:
      • Logs in /var/emulab
      • The vzctl RPM, containing a number of OpenVZ control commands
    • The latter seems to give a list of running VMs easily:
      vhost1,[/var/emulab],05:00(1)$ sudo vzlist -a
            CTID      NPROC STATUS    IP_ADDR         HOSTNAME
               1         15 running   -               virt1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net
      
    • I also see a command to figure out which container is running a given PID. Suppose i run top and am concerned about an sshd process chewing up all system CPU:
          PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND         
        51817 20001     20   0  116m 3780  872 R 94.4  0.0   0:05.74 sshd             
      
    • Since the user is numeric, i can assume this process is probably running in a container, so find out which one:
      vhost1,[/var/emulab],05:05(0)$ sudo vzpid 51766
      Pid     CTID    Name
      51766   1       sshd
      chaos      51804   51163  0 05:04 pts/0    00:00:00 grep --color=auto ssh
      
    • and then look up the container info as above.
    • The files in /var/emulab give details about how each experiment was created. In particular:
      Information about experiment startup attributes:
        /var/emulab/boot/tmcc.pcvm5-1/
        /var/emulab/boot/tmcc.pcvm5-2/
      
      Logs of experiment progress:
        /var/emulab/logs/tbvnode-pcvm5-1.log
        /var/emulab/logs/tbvnode-pcvm5-2.log
        /var/emulab/logs/tmccproxy.pcvm5-1.log
        /var/emulab/logs/tmccproxy.pcvm5-2.log
      
    • These may be useful for running and terminated experiments if the context IDs are unique.

Results of testing step 4: 2012-05-21

  • Per-host view of current state:
  • Per-experiment view of current state:
    • Browse to https://boss.utah.geniracks.net/genislices.php and find one slice running on the Component Manager:
      ID   HRN                         Created             Expires
      535  bbn-pgeni.ecgtest (ecgtest) 2012-05-21 13:19:28 2012-05-22 10:02:36
      
    • Click (ecgtest) to view the details of that experiment at https://boss.utah.geniracks.net/showexp.php3?experiment=536#details.
    • This shows what nodes it's using, including that its VM has been put on pc3:
      Physical Node Mapping:
      ID              Type         OS              Physical
      --------------- ------------ --------------- ------------
      phys1           dl360        FEDORA15-STD    pc4
      virt1           pcvm         OPENVZ-STD      pcvm3-1 (pc3)
      
    • Here are some other interesting things, all of which are similar to Friday's test:
      IP Port allocation:
      Low             High
      --------------- ------------
      30000           30255
      
      SSHD Port allocation ('ssh -p portnum'):
      ID              Port       SSH command
      --------------- ---------- ----------------------
      
      Physical Lan/Link Mapping:
      ID              Member          IP              MAC                  NodeID
      --------------- --------------- --------------- -------------------- ---------
      phys1-virt1-0   phys1:0         10.10.1.1       e8:39:35:b1:ec:9e    pc4
                                                      1/1 <-> 1/37         procurve2
      phys1-virt1-0   virt1:0         10.10.1.2                            pcvm3-1
      
  • Now, use the OpenVZ host itself to view activity:
    • As an admin, login to pc3.utah.geniracks.net
    • Everything seems similar to when i looked Friday:
      vhost2,[~],13:57(0)$ sudo vzlist -a
            CTID      NPROC STATUS    IP_ADDR         HOSTNAME
               1         19 running   -               virt1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net
      

Results of testing step 4: 2012-05-26

  • Per-host view of current state:
  • Per-experiment view of current state:
    • Browse to https://boss.utah.geniracks.net/genislices.php and find two slices running on the Component Manager:
      ID   HRN                                 Created             Expires
      949  bbn-pgeni.lnubuntu12b (lnubuntu12b) 2012-05-25 09:54:08 2012-05-29 18:00:00
      951  bbn-pgeni.ecgtest (ecgtest)         2012-05-26 05:30:20 2012-06-29 18:00:00
      
    • Click (ecgtest) to view the details of that experiment at https://boss.utah.geniracks.net/showexp.php3?experiment=952#details.
    • This shows what nodes it's using, including that its VM has been put on pc3:
      Physical Node Mapping:
      ID              Type         OS              Physical
      --------------- ------------ --------------- ------------
      phys1           dl360        FEDORA15-STD    pc2
      virt1           pcvm         OPENVZ-STD      pcvm3-1 (pc3)
      
    • Here are some other interesting things:
      IP Port allocation:
      Low             High
      --------------- ------------
      30000           30255
      
      SSHD Port allocation ('ssh -p portnum'):
      ID              Port       SSH command
      --------------- ---------- ----------------------
      
      Physical Lan/Link Mapping:
      ID              Member          IP              MAC                  NodeID
      --------------- --------------- --------------- -------------------- ---------
      phys1-virt1-0   phys1:0         10.10.1.1       e8:39:35:b1:0c:7e    pc2
                                                      1/1 <-> 1/28         procurve2
      phys1-virt1-0   virt1:0         10.10.1.2       00:00:0a:0a:01:02    pcvm3-1
      
  • So, indeed, a MAC address is reported for the virtual node. However, virt1 itself still says:
    mv1.1     Link encap:Ethernet  HWaddr 82:01:0A:0A:01:02  
              inet addr:10.10.1.2  Bcast:10.10.1.255  Mask:255.255.255.0
    
  • Now, use the OpenVZ host itself to view activity:
    • As an admin, login to pc3.utah.geniracks.net
    • Everything seems similar to when i looked Friday:
      vhost2,[~],06:00(0)$ sudo vzlist -a
            CTID      NPROC STATUS    IP_ADDR         HOSTNAME
               1         19 running   -               virt1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net
      

Earlier, i said:

Step 5: get information about terminated experiments

Using:

  • On boss, use AM state, logs, or administrator interfaces to find evidence of the two terminated experiments.
  • Determine how many other experiments were run in the past day.
  • Determine which GENI user created each of the terminated experiments.
  • Determine the mapping of experiments to OpenVZ or exclusive hosts for each of the terminated experiments.
  • Determine the control and dataplane MAC addresses assigned to each VM in each terminated experiment.
  • Determine any IP addresses assigned by InstaGENI to each VM in each terminated experiment.
  • Given a control IP address which InstaGENI had assigned to a now-terminated VM, determine which experiment was given that control IP.
  • Given a data plane IP address which an experimenter had requested for a now-terminated VM, determine which experiment was given that IP.

Verify:

  • A site administrator can get ownership and resource allocation information for recently-terminated experiments which used OpenVZ VMs.
  • A site administrator can get ownership and resource allocation information for recently-terminated experiments which used physical hosts.
  • A site administrator can get information about MAC addresses and IP addresses used by recently-terminated experiments.

Results of testing step 5: 2012-05-21

  • In red dot mode, https://boss.utah.geniracks.net/genihistory.php, i can view lots of previous slivers, of which ecgtest3 and ecgtest2 are among the most recent
  • I can type:
    urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest2
    
    into the search box, and bring up all previous instances of slivers in that slice.
  • Note that this is an exact match, not a regexp:
    urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest
    
    only pulls up ecgtest slivers, not ecgtest2 or ecgtest3. And just searching for ecgtest reports nothing.
  • As promised by the default text in the search box, searching for:
    urn:publicid:IDN+pgeni.gpolab.bbn.com+user+chaos
    
    does appear to get all of my slivers.
  • That UI shows that the following slivers were created in the past 24 hours:
    ID  Slice HRN/URN                                         Creator HRN/URN                                    Created             Destroyed           Manifest
    784 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest   urn:publicid:IDN+pgeni.gpolab.bbn.com+user+chaos   2012-05-21 13:19:41                     manifest
    778 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest3  urn:publicid:IDN+pgeni.gpolab.bbn.com+user+chaos   2012-05-21 12:51:18 2012-05-21 12:56:36 manifest
    772 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest2  urn:publicid:IDN+pgeni.gpolab.bbn.com+user+chaos   2012-05-21 12:17:44 2012-05-21 12:40:30 manifest
    760 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest   urn:publicid:IDN+pgeni.gpolab.bbn.com+user+chaos   2012-05-21 09:05:11 2012-05-21 09:27:04 manifest
    718 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+20vm      urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers 2012-05-21 08:03:37 2012-05-21 10:34:19 manifest
    686 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+15vm      urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers 2012-05-21 07:47:56 2012-05-21 10:52:30 manifest
    654 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+15vm      urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers 2012-05-21 07:32:17 2012-05-21 07:39:03 manifest
    622 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+2vmubuntu urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers 2012-05-21 07:24:50 2012-05-21 07:29:53 manifest
    616 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+2vmubuntu urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers 2012-05-21 07:10:14 2012-05-21 07:23:27 manifest
    
  • That display shows which GENI user created each experiment.
  • The clickable manifests can be used to get the sliver-to-resource mappings. Within each manifest, <rs:vnode /> elements can be used to find the resources used by the experiment. These look like:
    <rs:vnode xmlns:rs="http://www.protogeni.net/resources/rspec/ext/emulab/1" name="pc4"><host name="phys1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net"><services><login authentication="ssh-keys" hostname="pc4.utah.geniracks.net" port="22" username="chaos"></login></services></host></rs:vnode></sliver_type></node>
    <rs:vnode xmlns:rs="http://www.protogeni.net/resources/rspec/ext/emulab/1" name="pcvm3-1"><host name="virt1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net"><services><login authentication="ssh-keys" hostname="pc3.utah.geniracks.net" port="30010" username="chaos"></login></services></host></rs:vnode></sliver_type></node>
    
  • In addition, the manifests contain dataplane IP addresses and MAC addresses for each experiment (though these are wrong or missing for VMs, per 26)
  • Here is all the information i can get this way:
Emulab ID Sliver URN Physical nodes OpenVZ containers Dataplane IPs and MACs
784 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest pc4(phys1) pc3:pcvm3-1(virt1) 10.10.1.1(phys1:e83935b1ec9e) 10.10.1.2(virt1:00000a0a0102)
778 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest3 pc5(phys1) pc4(phys2) 10.10.1.1(phys1:e4115bed1cb6) 10.10.1.2(phys2:e83935b1ec9e)
772 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest2 pc3:pcvm3-1(virt1) pc3:pcvm3-2(virt2) 10.10.1.1(virt1:UNKNOWN) 10.10.1.2(virt2:UNKNOWN)
760 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest pc5:pcvm5-21(virt01) pc5:pcvm5-22(virt02) pc5:pcvm5-23(virt03) pc5:pcvm5-24(virt04) pc5:pcvm5-25(virt05) pc5:pcvm5-26(virt06) pc5:pcvm5-27(virt07) pc5:pcvm5-28(virt08) pc5:pcvm5-29(virt09) pc5:pcvm5-30(virt10) pc1:pcvm1-1(virt11)
718 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+20vm pc5:pcvm5-11(VM-1) pc2:pcvm2-8(VM-2) pc5:pcvm5-16(VM-3) pc5:pcvm5-17(VM-4) pc2:pcvm2-9(VM-5) pc2:pcvm2-10(VM-6) pc5:pcvm5-18(VM-7) pc5:pcvm5-19(VM-8) pc5:pcvm5-20(VM-9) pc5:pcvm5-12(VM-10) pc5:pcvm5-13(VM-11) pc2:pcvm2-1(VM-12) pc2:pcvm2-2(VM-13) pc2:pcvm2-3(VM-14) pc2:pcvm2-4(VM-15) pc5:pcvm5-14(VM-16) pc2:pcvm2-5(VM-17) pc2:pcvm2-6(VM-18) pc2:pcvm2-7(VM-19) pc5:pcvm5-15(VM-20) 10.10.1.1(VM-1:00000a0a0101) 10.10.1.2(VM-2:00000a0a0102) 10.10.1.3(VM-3:00000a0a0103) 10.10.1.4(VM-4:00000a0a0104) 10.10.1.5(VM-5:00000a0a0105) 10.10.1.6(VM-6:00000a0a0106) 10.10.1.7(VM-7:00000a0a0107) 10.10.1.8(VM-8:00000a0a0108) 10.10.1.9(VM-9:00000a0a0109) 10.10.1.10(VM-10:00000a0a010a) 10.10.1.20(VM-11:00000a0a0114) 10.10.1.19(VM-12:00000a0a0113) 10.10.1.11(VM-13:00000a0a010b) 10.10.1.12(VM-14:00000a0a010c) 10.10.1.13(VM-15:00000a0a010d) 10.10.1.14(VM-16:00000a0a010e) 10.10.1.15(VM-17:00000a0a010f) 10.10.1.16(VM-18:00000a0a0110) 10.10.1.17(VM-19:00000a0a0111) 10.10.1.18(VM-20:00000a0a0112)
686 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+15vm pc5:pcvm5-1(VM-1) pc5:pcvm5-6(VM-2) pc5:pcvm5-7(VM-3) pc4:pcvm4-3(VM-4) pc5:pcvm5-8(VM-5) pc4:pcvm4-4(VM-6) pc5:pcvm5-9(VM-7) pc4:pcvm4-5(VM-8) pc5:pcvm5-10(VM-9) pc5:pcvm5-2(VM-10) pc5:pcvm5-3(VM-11) pc5:pcvm5-4(VM-12) pc4:pcvm4-1(VM-13) pc4:pcvm4-2(VM-14) pc5:pcvm5-5(VM-15) 10.10.1.1(VM-1:00000a0a0101) 10.10.1.2(VM-2:00000a0a0102) 10.10.1.3(VM-3:00000a0a0103) 10.10.1.4(VM-4:UNKNOWN) 10.10.1.5(VM-5:00000a0a0105) 10.10.1.6(VM-6:UNKNOWN) 10.10.1.7(VM-7:00000a0a0107) 10.10.1.8(VM-8:UNKNOWN) 10.10.1.9(VM-9:00000a0a0109) 10.10.1.10(VM-10:00000a0a010a) 10.10.1.15(VM-11:00000a0a010f) 10.10.1.14(VM-12:00000a0a010e) 10.10.1.11(VM-13:UNKNOWN) 10.10.1.12(VM-14:UNKNOWN) 10.10.1.13(VM-15:00000a0a010d)
654 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+15vm pc2:pcvm2-1(VM-1) pc5:pcvm5-5(VM-2) pc5:pcvm5-6(VM-3) pc5:pcvm5-7(VM-4) pc5:pcvm5-8(VM-5) pc2:pcvm2-4(VM-6) pc2:pcvm2-5(VM-7) pc5:pcvm5-9(VM-8) pc5:pcvm5-10(VM-9) pc2:pcvm2-2(VM-10) pc5:pcvm5-1(VM-11) pc5:pcvm5-2(VM-12) pc5:pcvm5-3(VM-13) pc2:pcvm2-3(VM-14) pc5:pcvm5-4(VM-15) 10.10.1.2(VM-2:00000a0a0102) 10.10.1.3(VM-3:00000a0a0103) 10.10.1.4(VM-4:00000a0a0104) 10.10.1.5(VM-5:00000a0a0105) 10.10.1.8(VM-8:00000a0a0108) 10.10.1.9(VM-9:00000a0a0109) 10.10.1.15(VM-11:00000a0a010f) 10.10.1.14(VM-12:00000a0a010e) 10.10.1.11(VM-13:00000a0a010b) 10.10.1.13(VM-15:00000a0a010d) 10.10.1.1(VM-1:UNKNOWN) 10.10.1.6(VM-6:UNKNOWN) 10.10.1.7(VM-7:UNKNOWN) 10.10.1.10(VM-10:UNKNOWN) 10.10.1.12(VM-14:UNKNOWN)
622 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+2vmubuntu pc5:pcvm5-1(VM-1) pc1:pcvm1-3(VM-2) pc5:pcvm5-6(VM-3) pc1:pcvm1-4(VM-4) pc5:pcvm5-7(VM-5) pc1:pcvm1-5(VM-6) pc5:pcvm5-8(VM-7) pc5:pcvm5-9(VM-8) pc5:pcvm5-10(VM-9) pc1:pcvm1-1(VM-10) pc5:pcvm5-2(VM-11) pc5:pcvm5-3(VM-12) pc5:pcvm5-4(VM-13) pc5:pcvm5-5(VM-14) pc1:pcvm1-2(VM-15) 10.10.1.1(VM-1:00000a0a0101) 10.10.1.3(VM-3:00000a0a0103) 10.10.1.5(VM-5:00000a0a0105) 10.10.1.7(VM-7:00000a0a0107) 10.10.1.8(VM-8:00000a0a0108) 10.10.1.9(VM-9:00000a0a0109) 10.10.1.15(VM-11:00000a0a010f) 10.10.1.14(VM-12:00000a0a010e) 10.10.1.11(VM-13:00000a0a010b) 10.10.1.12(VM-14:00000a0a010c) 10.10.1.2(VM-2:UNKNOWN) 10.10.1.4(VM-4:UNKNOWN) 10.10.1.6(VM-6:UNKNOWN) 10.10.1.10(VM-10:UNKNOWN) 10.10.1.13(VM-15:UNKNOWN)
622 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+2vmubuntu pc5:pcvm5-1(VM-1) pc1:pcvm1-3(VM-2) pc5:pcvm5-6(VM-3) pc1:pcvm1-4(VM-4) pc5:pcvm5-7(VM-5) pc1:pcvm1-5(VM-6) pc5:pcvm5-8(VM-7) pc5:pcvm5-9(VM-8) pc5:pcvm5-10(VM-9) pc1:pcvm1-1(VM-10) pc5:pcvm5-2(VM-11) pc5:pcvm5-3(VM-12) pc5:pcvm5-4(VM-13) pc5:pcvm5-5(VM-14) pc1:pcvm1-2(VM-15) 10.10.1.1(VM-1:00000a0a0101) 10.10.1.3(VM-3:00000a0a0103) 10.10.1.5(VM-5:00000a0a0105) 10.10.1.7(VM-7:00000a0a0107) 10.10.1.8(VM-8:00000a0a0108) 10.10.1.9(VM-9:00000a0a0109) 10.10.1.15(VM-11:00000a0a010f) 10.10.1.14(VM-12:00000a0a010e) 10.10.1.11(VM-13:00000a0a010b) 10.10.1.12(VM-14:00000a0a010c) 10.10.1.2(VM-2:UNKNOWN) 10.10.1.4(VM-4:UNKNOWN) 10.10.1.6(VM-6:UNKNOWN) 10.10.1.10(VM-10:UNKNOWN) 10.10.1.13(VM-15:UNKNOWN)
  • Note, i semi-automated getting that information from the manifest using awk, as follows:
    • Download the XML data from the page (copy/paste)
    • Find every line that starts with <interface , and concatenate the next line, which contains </interface> to it
    • Find the node assignments:
      grep "<rs:vnode" tmpfile | awk '{print $6 " " $3 " " $4}' | awk -F= '{print $2 " " $3 " " $4}' | awk -F\" '{print $2 " " $4 " " $6}' | awk -F\. '{print $1 " " $4}' | awk '{print $1 ":" $3 "(" $4 ")"}'
      
    • Find the interface data for interfaces which have mac addresses defined:
      grep "<interface " tmpfile | grep mac_address | awk '{print $7 " " $2 " " $5}' | awk -F\" '{print $2 " " $4 " " $6}' | awk -F: '{print $1 " " $2}' | awk '{print $1 "(" $2 ":" $4 ")"}'
      
    • Find the interface data for interfaces which don't have mac addresses defined:
      grep "<interface " tmpfile | grep -v mac_address | awk '{print $6 " " $2}' | awk -F\" '{print $2 " " $4}' | awk -F: '{print $1}' | awk '{print $1 "(" $2 ":UNKNOWN)"}'
      
  • Incidentally, i came across this UI (https://boss.utah.geniracks.net/showpool.php), which shows the utilization of the nodes in the shared pool. That's not bad.
  • I poked around regarding how to do these:
    • Determine the control and dataplane MAC addresses assigned to each VM in each terminated experiment.
    • Determine any IP addresses assigned by InstaGENI to each VM in each terminated experiment.

Since i couldn't figure out anything really bulletproof, i created 31 to see whether Utah has a preferred solution to this. It's possible that some of this information can be obtained from the OpenVZ hosts. However, i can't get the information for e.g. Luisa's 20 VM experiment from earlier today, because the hosts have been swapped out and back in since then. This is an unusual situation in general, but not an unheard-of one. It would be better to cache information which might be forensically relevant on boss.

This test is also blocked by 26 from being fully completed, though i expect that the relevant parts of this will succeed too, and a cursory check should be sufficient.

Step 6: get OpenFlow state information

Using:

  • On the dataplane switch, get a list of controllers, and see if any additional controllers are serving experiments.
  • On the flowvisor VM, get a list of active FV slices from the FlowVisor
  • On the FOAM VM, get a list of active slivers from FOAM
  • Use FV, FOAM, or the switch to list the flowspace of a running OpenFlow experiment.

Verify:

  • A site administrator can get information about the OpenFlow resources used by running experiments.
  • When an OpenFlow experiment is started by InstaGENI, a new controller is added directly to the switch.
  • No new FlowVisor slices are added for new OpenFlow experiments started by InstaGENI.
  • No new FOAM slivers are added for new OpenFlow experiments started by InstaGENI.

Step 7: verify MAC addresses on the rack dataplane switch

Using:

  • Establish a privileged login to the dataplane switch
  • Obtain a list of the full MAC address table of the switch
  • On boss and the experimental hosts, use available data sources to determine which host or VM owns each MAC address.

Verify:

  • It is possible to identify and classify every MAC address visible on the switch

Step 8: verify active dataplane traffic

Using:

  • Establish a privileged login to the dataplane switch
  • Based on the information from Step 7, determine which interfaces are carrying traffic between the experimental VMs
  • Collect interface counters for those interfaces over a period of 10 minutes
  • Estimate the rate at which the experiment is sending traffic

Verify:

  • The switch reports interface counters, and an administrator can obtain plausible estimates of dataplane traffic quantities by looking at them.

Attachments (8)