[[PageOutline]] = IG-MON-3: GENI Active Experiment Inspection Test = This page captures status for the test case IG-MON-3, which verifies the ability to start, stop, and terminate experiments by experimenters and administrators. For overall status see the [wiki:GENIRacksHome/InstageniRacks/AcceptanceTestStatus InstaGENI Acceptance Test Status] page. ''Last update: 2013-03-06'' == Test Status == || '''Step''' || '''State''' || '''Tickets''' ||'''Notes''' || || 1 || [[Color(green,Pass)]] || || || || 2 || [[Color(green,Pass)]] || || || || 3 || [[Color(green,Pass)]] || || || || 4 || [[Color(green,Pass)]] || || || || 5 ||[[Color(lightgreen,Pass: most criteria)]]|| ||2013-02-28 ([instaticket:26], [instaticket:31]) terminated experiments display format is a little hard to use, but contains the requested information || || 6 || || || blocked on availability of OpenFlow functionality || || 7 || || || ready to test non-OpenFlow functionality || || 8 || || || ready to test non-OpenFlow functionality || [[BR]] || '''State Legend''' || '''Description''' || || [[Color(green,Pass)]] || Test completed and met all criteria || || [[Color(#98FB98,Pass: most criteria)]] || Test completed and met most criteria. Exceptions documented || || [[Color(red,Fail)]] || Test completed and failed to meet criteria. || || [[Color(yellow,Complete)]] || Test completed but will require re-execution due to expected changes || || [[Color(orange,Blocked)]] || Blocked by ticketed issue(s). || || [[Color(#63B8FF,In Progress)]] || Currently under test. || [[BR]] = Test Plan Steps = * An experimenter from the GPO starts up experiments to ensure there is data to look at: * An experimenter runs an experiment containing at least one rack OpenVZ VM, and terminates it. * An experimenter runs an experiment containing at least one rack OpenVZ VM, and leaves it running. * A site administrator uses available system and experiment data sources to determine current experimental state, including: * How many VMs are running and which experimenters own them * How many physical hosts are in use by experiments, and which experimenters own them * How many VMs were terminated within the past day, and which experimenters owned them * What !OpenFlow controllers the data plane switch, the rack !FlowVisor, and the rack FOAM are communicating with * A site administrator examines the switches and other rack data sources, and determines: * What MAC addresses are currently visible on the data plane switch and what experiments do they belong to? * For some experiment which was terminated within the past day, what data plane and control MAC and IP addresses did the experiment use? * For some experimental data path which is actively sending traffic on the data plane switch, do changes in interface counters show approximately the expected amount of traffic into and out of the switch? == Step 1. Start a VM experiment and leave it running == As experimenter create a 2 VMs experiment in the GPO InstaGENI rack. RSpec used: {{{ }}} Create a slice and sliver: {{{ $ omni.py createslice EG-MON-3 INFO:omni:Loading config file /home/lnevers/.gcf/omni_config INFO:omni:Using control framework pg INFO:omni:Created slice with Name EG-MON-3, URN urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+EG-MON-3, Expiration 2013-03-07 15:09:24+00:00 INFO:omni: ------------------------------------------------------------ INFO:omni: Completed createslice: Options as run: framework: pg Args: createslice EG-MON-3 Result Summary: Created slice with Name EG-MON-3, URN urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+EG-MON-3, Expiration 2013-03-07 15:09:24+00:00 INFO:omni: ============================================================ $ omni.py createsliver -a ig-gpo EG-MON-3 ./instageni-2vm-at-gpo.rspec INFO:omni:Loading config file /home/lnevers/.gcf/omni_config INFO:omni:Using control framework pg INFO:omni:Substituting AM nickname ig-gpo with URL https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+EG-MON-3 expires on 2013-03-07 15:09:24 UTC INFO:omni:Substituting AM nickname ig-gpo with URL https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN INFO:omni:Substituting AM nickname ig-gpo with URL https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN INFO:omni:Creating sliver(s) from rspec file ./instageni-2vm-at-gpo.rspec for slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+EG-MON-3 INFO:omni: (PG log url - look here for details on any failures: https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=388177e82b203e84aa3694dda7bf441d) INFO:omni:Got return from CreateSliver for slice EG-MON-3 at https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0: INFO:omni: INFO:omni: INFO:omni: ------------------------------------------------------------ INFO:omni: Completed createsliver: Options as run: aggregate: ['ig-gpo'] framework: pg Args: createsliver EG-MON-3 ./instageni-2vm-at-gpo.rspec Result Summary: Got Reserved resources RSpec from instageni-gpolab-bbn-com-protogeniv2 INFO:omni: ============================================================ }}} Determine login information for the VM resources assigned and that the VMs are "ready" for use: {{{ $ readyToLogin.py -a ig-gpo EG-MON-3 <...> VM-2's geni_status is: ready (am_status:ready) User lnevers logins to VM-2 using: xterm -e ssh -p 31291 -i /home/lnevers/.ssh/id_rsa lnevers@pc2.instageni.gpolab.bbn.com & VM-1's geni_status is: ready (am_status:ready) User lnevers logins to VM-1 using: xterm -e ssh -p 31290 -i /home/lnevers/.ssh/id_rsa lnevers@pc2.instageni.gpolab.bbn.com & }}} Login to VM-1, verify hostname, address assignment and initial ARP table entries: {{{ $ ssh -p 31290 -i /home/lnevers/.ssh/id_rsa lnevers@pc2.instageni.gpolab.bbn.com [lnevers@VM-1 ~]$ /sbin/ifconfig mv1.12 mv1.12 Link encap:Ethernet HWaddr 02:10:46:39:F3:47 inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::10:46ff:fe39:f347/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:6 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:6 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:552 (552.0 b) TX bytes:0 (0.0 b) [lnevers@VM-1 ~]$ /sbin/arp -a pc2.instageni.gpolab.bbn.com (192.1.242.141) at 10:60:4b:9b:82:14 [ether] on eth999 boss.instageni.gpolab.bbn.com (192.1.242.132) at 02:9e:26:f1:52:99 [ether] on eth999 }}} Login to VM-2, verify hostname, address assignment, MAC address for data plane interface, and initial ARP table entries: {{{ $ ssh -p 31291 -i /home/lnevers/.ssh/id_rsa lnevers@pc2.instageni.gpolab.bbn.com [lnevers@VM-2 ~]$ /sbin/ifconfig mv6.13 mv6.13 Link encap:Ethernet HWaddr 02:00:63:C7:43:61 inet addr:192.168.1.2 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::63ff:fec7:4361/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:6 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) [lnevers@VM-2 ~]$ /sbin/arp -a pc2.instageni.gpolab.bbn.com (192.1.242.141) at 10:60:4b:9b:82:14 [ether] on eth999 boss.instageni.gpolab.bbn.com (192.1.242.132) at 02:9e:26:f1:52:99 [ether] on eth999 }}} Install iperf on both VMs and exchange TCP traffic from VM-1 client to VM-2 server over the data plane interface and look at ARP table entries: {{{ [lnevers@VM-1 ~]$ /usr/bin/iperf -c 192.168.1.2 -t 60 ------------------------------------------------------------ Client connecting to 192.168.1.2, TCP port 5001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.1.1 port 55185 connected with 192.168.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-60.0 sec 687 MBytes 96.0 Mbits/sec [lnevers@VM-1 ~]$ /sbin/arp -a pc2.instageni.gpolab.bbn.com (192.1.242.141) at 10:60:4b:9b:82:14 [ether] on eth999 boss.instageni.gpolab.bbn.com (192.1.242.132) at 02:9e:26:f1:52:99 [ether] on eth999 VM-2-lan0 (192.168.1.2) at 02:00:63:c7:43:61 [ether] on mv1.12 }}} Show ARP entries on VM2 (iperf server): {{{ [lnevers@VM-2 ~]$ /sbin/arp -a pc2.instageni.gpolab.bbn.com (192.1.242.141) at 10:60:4b:9b:82:14 [ether] on eth999 boss.instageni.gpolab.bbn.com (192.1.242.132) at 02:9e:26:f1:52:99 [ether] on eth999 VM-1-lan0 (192.168.1.1) at 02:10:46:39:f3:47 [ether] on mv6.13 }}} Leave experiment running. == Step 2. Start a bare metal node experiment and leave it running == As experimenter create a 2 Raw PCs experiment in the GPO InstaGENI rack. RSpec used: {{{ }}} Create a slice and sliver: {{{ $ omni.py createslice IG-MON-3 INFO:omni:Loading config file /home/lnevers/.gcf/omni_config INFO:omni:Using control framework pg INFO:omni:Created slice with Name IG-MON-3, URN urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3, Expiration 2013-03-07 15:47:53+00:00 INFO:omni: ------------------------------------------------------------ INFO:omni: Completed createslice: Options as run: framework: pg Args: createslice IG-MON-3 Result Summary: Created slice with Name IG-MON-3, URN urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3, Expiration 2013-03-07 15:47:53+00:00 INFO:omni: ============================================================ $ omni.py createsliver -a ig-gpo IG-MON-3 ./instageni-2rawpc-at-gpo.rspec INFO:omni:Loading config file /home/lnevers/.gcf/omni_config INFO:omni:Using control framework pg INFO:omni:Substituting AM nickname ig-gpo with URL https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3 expires on 2013-03-07 15:47:53 UTC INFO:omni:Substituting AM nickname ig-gpo with URL https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN INFO:omni:Substituting AM nickname ig-gpo with URL https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN INFO:omni:Creating sliver(s) from rspec file ./instageni-2rawpc-at-gpo.rspec for slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3 INFO:omni: (PG log url - look here for details on any failures: https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=deb52735b826a50320d5274e8e4c6c8e) INFO:omni:Got return from CreateSliver for slice IG-MON-3 at https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0: INFO:omni: INFO:omni: INFO:omni: ------------------------------------------------------------ INFO:omni: Completed createsliver: Options as run: aggregate: ['ig-gpo'] framework: pg Args: createsliver IG-MON-3 ./instageni-2rawpc-at-gpo.rspec Result Summary: Got Reserved resources RSpec from instageni-gpolab-bbn-com-protogeniv2 INFO:omni: ============================================================ }}} Determine login information for the sliver resources: {{{ $ readyToLogin.py -a ig-gpo IG-MON-3 <...> PC-1's geni_status is: ready (am_status:ready) User lnevers logins to PC-1 using: xterm -e ssh -i /home/lnevers/.ssh/id_rsa lnevers@pc5.instageni.gpolab.bbn.com & PC-2's geni_status is: ready (am_status:ready) User lnevers logins to PC-2 using: xterm -e ssh -i /home/lnevers/.ssh/id_rsa lnevers@pc4.instageni.gpolab.bbn.com & }}} Login to PC-1, verify hostname, address assignment and initial ARP table entries: {{{ $ ssh -i /home/lnevers/.ssh/id_rsa lnevers@pc5.instageni.gpolab.bbn.com [lnevers@pc-1 ~]$ /sbin/ifconfig eth1 eth1 Link encap:Ethernet HWaddr 10:60:4B:9C:47:6A inet addr:192.168.1.1 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::1260:4bff:fe9c:476a/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:6 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:492 (492.0 b) Interrupt:37 Memory:f2000000-f2012800 [lnevers@pc-1 ~]$ /sbin/arp -a ops.instageni.gpolab.bbn.com (192.1.242.133) at 02:07:2d:51:f4:6a [ether] on eth0 boss.instageni.gpolab.bbn.com (192.1.242.132) at 02:9e:26:f1:52:99 [ether] on eth0 ? (192.1.242.129) at cc:ef:48:7a:7a:a9 [ether] on eth0 }}} Login to PC-2, verify hostname, address assignment and initial ARP table entries: {{{ $ ssh -i /home/lnevers/.ssh/id_rsa lnevers@pc4.instageni.gpolab.bbn.com [lnevers@pc-2 ~]$ /sbin/ifconfig eth1 eth1 Link encap:Ethernet HWaddr 10:60:4B:96:00:D6 inet addr:192.168.1.2 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::1260:4bff:fe96:d6/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:6 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 b) TX bytes:492 (492.0 b) Interrupt:37 Memory:f2000000-f2012800 [lnevers@pc-2 ~]$ /sbin/arp -a ops.instageni.gpolab.bbn.com (192.1.242.133) at 02:07:2d:51:f4:6a [ether] on eth0 boss.instageni.gpolab.bbn.com (192.1.242.132) at 02:9e:26:f1:52:99 [ether] on eth0 ? (192.1.242.129) at cc:ef:48:7a:7a:a9 [ether] on eth0 }}} Install iperf on both PCs and exchange TCP traffic from PC-1 client to PC-2 server over the data plane interface and look at ARP table entries: {{{ [lnevers@pc-1 ~]$ /usr/bin/iperf -c 192.168.1.2 -i 1 ------------------------------------------------------------ Client connecting to 192.168.1.2, TCP port 5001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.1.1 port 58436 connected with 192.168.1.2 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0- 1.0 sec 113 MBytes 947 Mbits/sec [ 3] 1.0- 2.0 sec 112 MBytes 940 Mbits/sec [ 3] 2.0- 3.0 sec 112 MBytes 943 Mbits/sec [ 3] 3.0- 4.0 sec 112 MBytes 942 Mbits/sec [ 3] 4.0- 5.0 sec 112 MBytes 940 Mbits/sec }}} Leave sliver running. == Step 3. Start an OpenFlow experiment and leave it running == As experimenter create a 2 VMs !OpenFlow experiment in the GPO InstaGENI rack. RSpec used: {{{ }}} Create a sliver and sliver for the 2 VMs on shared !OpenFlow VLAN 1750 in the GPO InstaGENI rack: {{{ $ omni.py createslice IG-MON-3-OF INFO:omni:Loading config file /home/lnevers/.gcf/omni_config INFO:omni:Using control framework pg INFO:omni:Created slice with Name IG-MON-3-OF, URN urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF, Expiration 2013-03-07 16:26:04+00:00 INFO:omni: ------------------------------------------------------------ INFO:omni: Completed createslice: Options as run: framework: pg Args: createslice IG-MON-3-OF Result Summary: Created slice with Name IG-MON-3-OF, URN urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF, Expiration 2013-03-07 16:26:04+00:00 INFO:omni: ============================================================ $ omni.py createsliver -a ig-gpo IG-MON-3-OF ./instageni-2vm-vlan1750-at-gpo.rspec INFO:omni:Loading config file /home/lnevers/.gcf/omni_config INFO:omni:Using control framework pg INFO:omni:Substituting AM nickname ig-gpo with URL https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF expires on 2013-03-07 16:26:04 UTC INFO:omni:Substituting AM nickname ig-gpo with URL https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN INFO:omni:Substituting AM nickname ig-gpo with URL https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0, URN unspecified_AM_URN INFO:omni:Creating sliver(s) from rspec file ./instageni-2vm-vlan1750-at-gpo.rspec for slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF INFO:omni: (PG log url - look here for details on any failures: https://boss.instageni.gpolab.bbn.com/spewlogfile.php3?logfile=8af1230117ca9b383b6510da21d7abc7) INFO:omni:Got return from CreateSliver for slice IG-MON-3-OF at https://instageni.gpolab.bbn.com:12369/protogeni/xmlrpc/am/2.0: INFO:omni: INFO:omni: INFO:omni: ------------------------------------------------------------ INFO:omni: Completed createsliver: Options as run: aggregate: ['ig-gpo'] framework: pg Args: createsliver IG-MON-3-OF ./instageni-2vm-vlan1750-at-gpo.rspec Result Summary: Got Reserved resources RSpec from instageni-gpolab-bbn-com-protogeniv2 INFO:omni: ============================================================ }}} A FOAM sliver is needed to allow the traffic exchange. Using this FOAM RSpec: {{{ }}} Create a sliver at the GPO InstaGENI rack FOAM to allow the traffic exchange: {{{ $ omni.py createsliver -a ig-of-gpo IG-MON-3-OF ./instageni-openflow-at-gpo.rspec INFO:omni:Loading config file /home/lnevers/.gcf/omni_config INFO:omni:Using control framework pg INFO:omni:Substituting AM nickname ig-of-gpo with URL https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1, URN unspecified_AM_URN WARNING:omni:You asked to use AM API 2, but the AM(s) you are contacting do not all speak that version. WARNING:omni:At the URLs you are contacting, all your AMs speak AM API v1. WARNING:omni:Switching to AM API v1. Next time call Omni with '-V1'. INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF expires on 2013-03-07 16:26:04 UTC INFO:omni:Substituting AM nickname ig-of-gpo with URL https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1, URN unspecified_AM_URN INFO:omni:Substituting AM nickname ig-of-gpo with URL https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1, URN unspecified_AM_URN INFO:omni:Creating sliver(s) from rspec file ./instageni-openflow-at-gpo.rspec for slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF INFO:omni:Got return from CreateSliver for slice IG-MON-3-OF at https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1: INFO:omni: INFO:omni: INFO:omni: INFO:omni: ------------------------------------------------------------ INFO:omni: Completed createsliver: Options as run: aggregate: ['ig-of-gpo'] api_version: 1 framework: pg Args: createsliver IG-MON-3-OF ./instageni-openflow-at-gpo.rspec Result Summary: Your AMs do not all speak requested API v2. At the URLs you are contacting, all your AMs speak AM API v1. Switching to AM API v1. Next time call Omni with '-V1'. Got Reserved resources RSpec from foam-instageni-gpolab-bbn-com INFO:omni: ============================================================ }}} This sliver is auto approved. State can be confirmed with omni command: {{{ $ omni.py sliverstatus -a ig-of-gpo IG-MON-3-OF -V1 INFO:omni:Loading config file /home/lnevers/.gcf/omni_config INFO:omni:Using control framework pg INFO:omni:Substituting AM nickname ig-of-gpo with URL https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1, URN unspecified_AM_URN INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF expires on 2013-03-07 16:26:04 UTC INFO:omni:Substituting AM nickname ig-of-gpo with URL https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1, URN unspecified_AM_URN INFO:omni:Status of Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF: INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF at AM https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 has overall SliverStatus: ready INFO:omni:Sliver status for Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF at AM URL https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 INFO:omni:{ "geni_status": "ready", "geni_urn": "urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF:8070b745-84c7-47eb-8e04-63a764997f3a", "foam_pend_reason": null, "foam_expires": "2013-03-07 16:26:04+00:00", "geni_resources": [ { "geni_urn": "urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF:8070b745-84c7-47eb-8e04-63a764997f3a", "geni_error": "", "geni_status": "ready" } ], "foam_status": "Approved" } INFO:omni: ------------------------------------------------------------ INFO:omni: Completed sliverstatus: Options as run: aggregate: ['ig-of-gpo'] api_version: 1 framework: pg Args: sliverstatus IG-MON-3-OF Result Summary: Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF expires on 2013-03-07 16:26:04 UTC Slice urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+IG-MON-3-OF at AM https://foam.instageni.gpolab.bbn.com:3626/foam/gapi/1 has overall SliverStatus: ready. Returned status of slivers on 1 of 1 possible aggregates. INFO:omni: ============================================================ }}} Determine login for assigned nodes: {{{ $ readyToLogin.py -a ig-gpo IG-MON-3-OF <...> gpo-ig's geni_status is: ready (am_status:ready) User lnevers logins to gpo-ig using: xterm -e ssh -p 32058 -i /home/lnevers/.ssh/id_rsa lnevers@pc2.instageni.gpolab.bbn.com & gpo-ig2's geni_status is: ready (am_status:ready) User lnevers logins to gpo-ig2 using: xterm -e ssh -p 32059 -i /home/lnevers/.ssh/id_rsa lnevers@pc2.instageni.gpolab.bbn.com & }}} Login to VM named 'gpo-ig' and send traffic to remote: {{{ $ ssh -p 32058 -i /home/lnevers/.ssh/id_rsa lnevers@pc2.instageni.gpolab.bbn.com [lnevers@ig-gpo ~]$ ping 10.42.18.42 -c 5 PING 10.42.18.42 (10.42.18.42) 56(84) bytes of data. 64 bytes from 10.42.18.42: icmp_req=1 ttl=64 time=6.73 ms 64 bytes from 10.42.18.42: icmp_req=2 ttl=64 time=0.077 ms 64 bytes from 10.42.18.42: icmp_req=3 ttl=64 time=0.075 ms 64 bytes from 10.42.18.42: icmp_req=4 ttl=64 time=0.075 ms 64 bytes from 10.42.18.42: icmp_req=5 ttl=64 time=0.075 ms --- 10.42.18.42 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4001ms rtt min/avg/max/mdev = 0.075/1.407/6.735/2.664 ms }}} Login to VM named 'gpo-ig2' and send traffic to remote: {{{ $ ssh -p 32059 -i /home/lnevers/.ssh/id_rsa lnevers@pc2.instageni.gpolab.bbn.com [lnevers@ig-gpo2 ~]$ ping 10.42.18.43 -c 5 PING 10.42.18.43 (10.42.18.43) 56(84) bytes of data. 64 bytes from 10.42.18.43: icmp_req=1 ttl=64 time=6.33 ms 64 bytes from 10.42.18.43: icmp_req=2 ttl=64 time=0.075 ms 64 bytes from 10.42.18.43: icmp_req=3 ttl=64 time=0.074 ms 64 bytes from 10.42.18.43: icmp_req=4 ttl=64 time=0.076 ms 64 bytes from 10.42.18.43: icmp_req=5 ttl=64 time=0.074 ms --- 10.42.18.43 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4000ms rtt min/avg/max/mdev = 0.074/1.327/6.337/2.505 ms [lnevers@ig-gpo2 ~]$ }}} Leave experiment running == Step 4. Administrator review of running experiments == As administrator determine experiments status, resource assignment, and configuration for the 3 experiments that are running from previous steps: - EG-MON-3 - a 2 VM experiment - IG-MON-3 - a 2 raw pc experiment - IG-MON-3-OF - a 2 VM !OpenFlow experiment Determine list of active experiments, by selecting "Experimentation" pulldown in "red dot" mode, and choosing "Experiment list", which maps to the URL https://boss.instageni.gpolab.bbn.com/showexp_list.php3. A capture of the active experiment page shows: [[Image(IG-MON-3-experiments.jpg)]] [[BR]] Each of the 3 experiment created for this test case (EG-MON-3, IG-MON-3, and IG-MON-3-OF) are present in the list. [[BR]] In the Active Experiment table, select the experiment EG-MON-3 (2 VM experiment) and this shows show all know details for the experiment: [[BR]] [[Image(IG-MON-3-2vm.jpg)]] [[BR]] In the Active Experiment table, select the experiment IG-MON-3 (2 raw pc experiment) and this shows show all know details for the experiment: [[BR]] [[Image(IG-MON-3-2pc.jpg)]] [[BR]] In the Active Experiment table, select the experiment IG-MON-3-OF (2 VM !OpenFlow experiment)and this shows show all know details for the experiment: [[BR]] [[Image(IG-MON-3-2vm-of.jpg)]] [[BR]] For each of the above experiment clicking on the topology diagram on the left hand side panel will result in a details page that includes specific configuration and allocation for the experiment. Clicking on the topology diagram for the experiment IG-MON-3 (2 raw pc experiment), this shows node mapping, IP Port allocation, and other useful information: [[Image(IG-MON-3-2pc-detail.jpg)]] For the experiment IG-MON-3-OF (2 VM !OpenFlow experiment), click the topology diagram and the experiment topology along with the experiment details for Virtual Nodes, Physical Nodes Mapping, Port Allocation, Virtual Links, Physical Links, Links Mappings, and VLAN settings are shown: [[Image(IG-MON-3-2vm-of-detail.jpg)]] [[BR]] Compared the MAC addresses show on this page with the ones collected in earlier steps from logging into the experiment nodes and found them to be the same. }}} As administrator, log into the OpenVZ host pc2.instageni.gpolab.bbn.com * Now, use the OpenVZ host itself to view activity: * As an admin, login to pc5.utah.geniracks.net * Poking around, i was led to a couple of prospective data sources: * Logs in `/var/emulab` * The `vzctl` RPM, containing a number of OpenVZ control commands * The latter seems to give a list of running VMs easily: {{{ vhost1,[/var/emulab],05:00(1)$ sudo vzlist -a CTID NPROC STATUS IP_ADDR HOSTNAME 1 15 running - virt1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net }}} * I also see a command to figure out which container is running a given PID. Suppose i run top and am concerned about an sshd process chewing up all system CPU: {{{ PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 51817 20001 20 0 116m 3780 872 R 94.4 0.0 0:05.74 sshd }}} * Since the user is numeric, i can assume this process is probably running in a container, so find out which one: {{{ vhost1,[/var/emulab],05:05(0)$ sudo vzpid 51766 Pid CTID Name 51766 1 sshd chaos 51804 51163 0 05:04 pts/0 00:00:00 grep --color=auto ssh }}} * and then look up the container info as above. * The files in `/var/emulab` give details about how each experiment was created. In particular: {{{ Information about experiment startup attributes: /var/emulab/boot/tmcc.pcvm5-1/ /var/emulab/boot/tmcc.pcvm5-2/ Logs of experiment progress: /var/emulab/logs/tbvnode-pcvm5-1.log /var/emulab/logs/tbvnode-pcvm5-2.log /var/emulab/logs/tmccproxy.pcvm5-1.log /var/emulab/logs/tmccproxy.pcvm5-2.log }}} * These may be useful for running and terminated experiments ''if'' the context IDs are unique. === Results of testing step 4: 2012-05-21 === * Per-host view of current state: * From [https://boss.utah.geniracks.net/nodecontrol_list.php3?showtype=dl360] in red dot mode, i can once again see that pc4 is allocated as phys1 to `pgeni-gpolab-bbn-com/ecgtest`. * I can see that pc1 and pc3 are configured as OpenVZ shared hosts, but i can't see what experiments they are running. * Per-experiment view of current state: * Browse to [https://boss.utah.geniracks.net/genislices.php] and find one slice running on the Component Manager: {{{ ID HRN Created Expires 535 bbn-pgeni.ecgtest (ecgtest) 2012-05-21 13:19:28 2012-05-22 10:02:36 }}} * Click `(ecgtest)` to view the details of that experiment at [https://boss.utah.geniracks.net/showexp.php3?experiment=536#details]. * This shows what nodes it's using, including that its VM has been put on pc3: {{{ Physical Node Mapping: ID Type OS Physical --------------- ------------ --------------- ------------ phys1 dl360 FEDORA15-STD pc4 virt1 pcvm OPENVZ-STD pcvm3-1 (pc3) }}} * Here are some other interesting things, all of which are similar to Friday's test: {{{ IP Port allocation: Low High --------------- ------------ 30000 30255 SSHD Port allocation ('ssh -p portnum'): ID Port SSH command --------------- ---------- ---------------------- Physical Lan/Link Mapping: ID Member IP MAC NodeID --------------- --------------- --------------- -------------------- --------- phys1-virt1-0 phys1:0 10.10.1.1 e8:39:35:b1:ec:9e pc4 1/1 <-> 1/37 procurve2 phys1-virt1-0 virt1:0 10.10.1.2 pcvm3-1 }}} * Now, use the OpenVZ host itself to view activity: * As an admin, login to pc3.utah.geniracks.net * Everything seems similar to when i looked Friday: {{{ vhost2,[~],13:57(0)$ sudo vzlist -a CTID NPROC STATUS IP_ADDR HOSTNAME 1 19 running - virt1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net }}} === Results of testing step 4: 2012-05-26 === * Per-host view of current state: * From [https://boss.utah.geniracks.net/nodecontrol_list.php3?showtype=dl360] in red dot mode, i can once again see that pc2 is allocated as phys1 to `pgeni-gpolab-bbn-com/ecgtest`. * I can see that pc3 and pc5 are configured as OpenVZ shared hosts, but i can't see what experiments they are running. * Using [https://boss.utah.geniracks.net/showpool.php], i can see that pc3 is running one VM and pc5 is running zero, but not what experiments each is running. I opened [instaticket:35] to ask whether a node-to-experiment mapping would be an easy modification to `showpool.php`. * Per-experiment view of current state: * Browse to [https://boss.utah.geniracks.net/genislices.php] and find two slices running on the Component Manager: {{{ ID HRN Created Expires 949 bbn-pgeni.lnubuntu12b (lnubuntu12b) 2012-05-25 09:54:08 2012-05-29 18:00:00 951 bbn-pgeni.ecgtest (ecgtest) 2012-05-26 05:30:20 2012-06-29 18:00:00 }}} * Click `(ecgtest)` to view the details of that experiment at [https://boss.utah.geniracks.net/showexp.php3?experiment=952#details]. * This shows what nodes it's using, including that its VM has been put on pc3: {{{ Physical Node Mapping: ID Type OS Physical --------------- ------------ --------------- ------------ phys1 dl360 FEDORA15-STD pc2 virt1 pcvm OPENVZ-STD pcvm3-1 (pc3) }}} * Here are some other interesting things: {{{ IP Port allocation: Low High --------------- ------------ 30000 30255 SSHD Port allocation ('ssh -p portnum'): ID Port SSH command --------------- ---------- ---------------------- Physical Lan/Link Mapping: ID Member IP MAC NodeID --------------- --------------- --------------- -------------------- --------- phys1-virt1-0 phys1:0 10.10.1.1 e8:39:35:b1:0c:7e pc2 1/1 <-> 1/28 procurve2 phys1-virt1-0 virt1:0 10.10.1.2 00:00:0a:0a:01:02 pcvm3-1 }}} * So, indeed, a MAC address is reported for the virtual node. However, virt1 itself still says: {{{ mv1.1 Link encap:Ethernet HWaddr 82:01:0A:0A:01:02 inet addr:10.10.1.2 Bcast:10.10.1.255 Mask:255.255.255.0 }}} * Now, use the OpenVZ host itself to view activity: * As an admin, login to pc3.utah.geniracks.net * Everything seems similar to when i looked Friday: {{{ vhost2,[~],06:00(0)$ sudo vzlist -a CTID NPROC STATUS IP_ADDR HOSTNAME 1 19 running - virt1.ecgtest.pgeni-gpolab-bbn-com.utah.geniracks.net }}} Earlier, i said: * Using [https://boss.utah.geniracks.net/showpool.php], i can see that pc3 is running one VM and pc5 is running zero, but not what experiments each is running. I opened [instaticket:35] to ask whether a node-to-experiment mapping would be an easy modification to `showpool.php`. * Leigh responded that this is already at the bottom of the node UI, see e.g. [https://boss.utah.geniracks.net/shownode.php3?node_id=pc3] (which you can get to by clicking through from showpool.php. So this is good for our purposes. == Step 5: get information about terminated experiments == '''Using:''' * On boss, use AM state, logs, or administrator interfaces to find evidence of the two terminated experiments. * Determine how many other experiments were run in the past day. * Determine which GENI user created each of the terminated experiments. * Determine the mapping of experiments to OpenVZ or exclusive hosts for each of the terminated experiments. * Determine the control and dataplane MAC addresses assigned to each VM in each terminated experiment. * ~~Determine any IP addresses assigned by InstaGENI to each VM in each terminated experiment.~~ * Given a control IP address which InstaGENI had assigned to a now-terminated VM, determine which experiment was given that control IP. * Given a data plane IP address which an experimenter had requested for a now-terminated VM, determine which experiment was given that IP. '''Verify:''' * A site administrator can get ownership and resource allocation information for recently-terminated experiments which used OpenVZ VMs. * A site administrator can get ownership and resource allocation information for recently-terminated experiments which used physical hosts. * A site administrator can get information about MAC addresses and IP addresses used by recently-terminated experiments. === Results of testing step 5: 2012-05-21 === * In red dot mode, [https://boss.utah.geniracks.net/genihistory.php], i can view lots of previous slivers, of which `ecgtest3` and `ecgtest2` are among the most recent * I can type: {{{ urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest2 }}} into the search box, and bring up all previous instances of slivers in that slice. * Note that this is an exact match, ''not'' a regexp: {{{ urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest }}} only pulls up `ecgtest` slivers, not `ecgtest2` or `ecgtest3`. And just searching for `ecgtest` reports nothing. * As promised by the default text in the search box, searching for: {{{ urn:publicid:IDN+pgeni.gpolab.bbn.com+user+chaos }}} does appear to get all of my slivers. * That UI shows that the following slivers were created in the past 24 hours: {{{ ID Slice HRN/URN Creator HRN/URN Created Destroyed Manifest 784 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest urn:publicid:IDN+pgeni.gpolab.bbn.com+user+chaos 2012-05-21 13:19:41 manifest 778 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest3 urn:publicid:IDN+pgeni.gpolab.bbn.com+user+chaos 2012-05-21 12:51:18 2012-05-21 12:56:36 manifest 772 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest2 urn:publicid:IDN+pgeni.gpolab.bbn.com+user+chaos 2012-05-21 12:17:44 2012-05-21 12:40:30 manifest 760 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest urn:publicid:IDN+pgeni.gpolab.bbn.com+user+chaos 2012-05-21 09:05:11 2012-05-21 09:27:04 manifest 718 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+20vm urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers 2012-05-21 08:03:37 2012-05-21 10:34:19 manifest 686 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+15vm urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers 2012-05-21 07:47:56 2012-05-21 10:52:30 manifest 654 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+15vm urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers 2012-05-21 07:32:17 2012-05-21 07:39:03 manifest 622 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+2vmubuntu urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers 2012-05-21 07:24:50 2012-05-21 07:29:53 manifest 616 urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+2vmubuntu urn:publicid:IDN+pgeni.gpolab.bbn.com+user+lnevers 2012-05-21 07:10:14 2012-05-21 07:23:27 manifest }}} * That display shows which GENI user created each experiment. * The clickable manifests can be used to get the sliver-to-resource mappings. Within each manifest, `` elements can be used to find the resources used by the experiment. These look like: {{{ }}} * In addition, the manifests contain ''dataplane'' IP addresses and MAC addresses for each experiment (though these are wrong or missing for VMs, per [instaticket:26]) * Here is all the information i can get this way: || '''Emulab ID''' || '''Sliver URN''' || '''Physical nodes''' || '''OpenVZ containers''' || '''Dataplane IPs and MACs''' || || 784 || urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest || pc4(phys1) || pc3:pcvm3-1(virt1) || 10.10.1.1(phys1:e83935b1ec9e) 10.10.1.2(virt1:00000a0a0102) || || 778 || urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest3 || pc5(phys1) pc4(phys2) || || 10.10.1.1(phys1:e4115bed1cb6) 10.10.1.2(phys2:e83935b1ec9e) || || 772 || urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest2 || || pc3:pcvm3-1(virt1) pc3:pcvm3-2(virt2) || 10.10.1.1(virt1:UNKNOWN) 10.10.1.2(virt2:UNKNOWN) || || 760 || urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+ecgtest || || pc5:pcvm5-21(virt01) pc5:pcvm5-22(virt02) pc5:pcvm5-23(virt03) pc5:pcvm5-24(virt04) pc5:pcvm5-25(virt05) pc5:pcvm5-26(virt06) pc5:pcvm5-27(virt07) pc5:pcvm5-28(virt08) pc5:pcvm5-29(virt09) pc5:pcvm5-30(virt10) pc1:pcvm1-1(virt11) || || || 718 || urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+20vm || || pc5:pcvm5-11(VM-1) pc2:pcvm2-8(VM-2) pc5:pcvm5-16(VM-3) pc5:pcvm5-17(VM-4) pc2:pcvm2-9(VM-5) pc2:pcvm2-10(VM-6) pc5:pcvm5-18(VM-7) pc5:pcvm5-19(VM-8) pc5:pcvm5-20(VM-9) pc5:pcvm5-12(VM-10) pc5:pcvm5-13(VM-11) pc2:pcvm2-1(VM-12) pc2:pcvm2-2(VM-13) pc2:pcvm2-3(VM-14) pc2:pcvm2-4(VM-15) pc5:pcvm5-14(VM-16) pc2:pcvm2-5(VM-17) pc2:pcvm2-6(VM-18) pc2:pcvm2-7(VM-19) pc5:pcvm5-15(VM-20) || 10.10.1.1(VM-1:00000a0a0101) 10.10.1.2(VM-2:00000a0a0102) 10.10.1.3(VM-3:00000a0a0103) 10.10.1.4(VM-4:00000a0a0104) 10.10.1.5(VM-5:00000a0a0105) 10.10.1.6(VM-6:00000a0a0106) 10.10.1.7(VM-7:00000a0a0107) 10.10.1.8(VM-8:00000a0a0108) 10.10.1.9(VM-9:00000a0a0109) 10.10.1.10(VM-10:00000a0a010a) 10.10.1.20(VM-11:00000a0a0114) 10.10.1.19(VM-12:00000a0a0113) 10.10.1.11(VM-13:00000a0a010b) 10.10.1.12(VM-14:00000a0a010c) 10.10.1.13(VM-15:00000a0a010d) 10.10.1.14(VM-16:00000a0a010e) 10.10.1.15(VM-17:00000a0a010f) 10.10.1.16(VM-18:00000a0a0110) 10.10.1.17(VM-19:00000a0a0111) 10.10.1.18(VM-20:00000a0a0112) || || 686 || urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+15vm || || pc5:pcvm5-1(VM-1) pc5:pcvm5-6(VM-2) pc5:pcvm5-7(VM-3) pc4:pcvm4-3(VM-4) pc5:pcvm5-8(VM-5) pc4:pcvm4-4(VM-6) pc5:pcvm5-9(VM-7) pc4:pcvm4-5(VM-8) pc5:pcvm5-10(VM-9) pc5:pcvm5-2(VM-10) pc5:pcvm5-3(VM-11) pc5:pcvm5-4(VM-12) pc4:pcvm4-1(VM-13) pc4:pcvm4-2(VM-14) pc5:pcvm5-5(VM-15) || 10.10.1.1(VM-1:00000a0a0101) 10.10.1.2(VM-2:00000a0a0102) 10.10.1.3(VM-3:00000a0a0103) 10.10.1.4(VM-4:UNKNOWN) 10.10.1.5(VM-5:00000a0a0105) 10.10.1.6(VM-6:UNKNOWN) 10.10.1.7(VM-7:00000a0a0107) 10.10.1.8(VM-8:UNKNOWN) 10.10.1.9(VM-9:00000a0a0109) 10.10.1.10(VM-10:00000a0a010a) 10.10.1.15(VM-11:00000a0a010f) 10.10.1.14(VM-12:00000a0a010e) 10.10.1.11(VM-13:UNKNOWN) 10.10.1.12(VM-14:UNKNOWN) 10.10.1.13(VM-15:00000a0a010d) || || 654 || urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+15vm || || pc2:pcvm2-1(VM-1) pc5:pcvm5-5(VM-2) pc5:pcvm5-6(VM-3) pc5:pcvm5-7(VM-4) pc5:pcvm5-8(VM-5) pc2:pcvm2-4(VM-6) pc2:pcvm2-5(VM-7) pc5:pcvm5-9(VM-8) pc5:pcvm5-10(VM-9) pc2:pcvm2-2(VM-10) pc5:pcvm5-1(VM-11) pc5:pcvm5-2(VM-12) pc5:pcvm5-3(VM-13) pc2:pcvm2-3(VM-14) pc5:pcvm5-4(VM-15) || 10.10.1.2(VM-2:00000a0a0102) 10.10.1.3(VM-3:00000a0a0103) 10.10.1.4(VM-4:00000a0a0104) 10.10.1.5(VM-5:00000a0a0105) 10.10.1.8(VM-8:00000a0a0108) 10.10.1.9(VM-9:00000a0a0109) 10.10.1.15(VM-11:00000a0a010f) 10.10.1.14(VM-12:00000a0a010e) 10.10.1.11(VM-13:00000a0a010b) 10.10.1.13(VM-15:00000a0a010d) 10.10.1.1(VM-1:UNKNOWN) 10.10.1.6(VM-6:UNKNOWN) 10.10.1.7(VM-7:UNKNOWN) 10.10.1.10(VM-10:UNKNOWN) 10.10.1.12(VM-14:UNKNOWN) || || 622 || urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+2vmubuntu || || pc5:pcvm5-1(VM-1) pc1:pcvm1-3(VM-2) pc5:pcvm5-6(VM-3) pc1:pcvm1-4(VM-4) pc5:pcvm5-7(VM-5) pc1:pcvm1-5(VM-6) pc5:pcvm5-8(VM-7) pc5:pcvm5-9(VM-8) pc5:pcvm5-10(VM-9) pc1:pcvm1-1(VM-10) pc5:pcvm5-2(VM-11) pc5:pcvm5-3(VM-12) pc5:pcvm5-4(VM-13) pc5:pcvm5-5(VM-14) pc1:pcvm1-2(VM-15) || 10.10.1.1(VM-1:00000a0a0101) 10.10.1.3(VM-3:00000a0a0103) 10.10.1.5(VM-5:00000a0a0105) 10.10.1.7(VM-7:00000a0a0107) 10.10.1.8(VM-8:00000a0a0108) 10.10.1.9(VM-9:00000a0a0109) 10.10.1.15(VM-11:00000a0a010f) 10.10.1.14(VM-12:00000a0a010e) 10.10.1.11(VM-13:00000a0a010b) 10.10.1.12(VM-14:00000a0a010c) 10.10.1.2(VM-2:UNKNOWN) 10.10.1.4(VM-4:UNKNOWN) 10.10.1.6(VM-6:UNKNOWN) 10.10.1.10(VM-10:UNKNOWN) 10.10.1.13(VM-15:UNKNOWN) || || 622 || urn:publicid:IDN+pgeni.gpolab.bbn.com+slice+2vmubuntu || || pc5:pcvm5-1(VM-1) pc1:pcvm1-3(VM-2) pc5:pcvm5-6(VM-3) pc1:pcvm1-4(VM-4) pc5:pcvm5-7(VM-5) pc1:pcvm1-5(VM-6) pc5:pcvm5-8(VM-7) pc5:pcvm5-9(VM-8) pc5:pcvm5-10(VM-9) pc1:pcvm1-1(VM-10) pc5:pcvm5-2(VM-11) pc5:pcvm5-3(VM-12) pc5:pcvm5-4(VM-13) pc5:pcvm5-5(VM-14) pc1:pcvm1-2(VM-15) || 10.10.1.1(VM-1:00000a0a0101) 10.10.1.3(VM-3:00000a0a0103) 10.10.1.5(VM-5:00000a0a0105) 10.10.1.7(VM-7:00000a0a0107) 10.10.1.8(VM-8:00000a0a0108) 10.10.1.9(VM-9:00000a0a0109) 10.10.1.15(VM-11:00000a0a010f) 10.10.1.14(VM-12:00000a0a010e) 10.10.1.11(VM-13:00000a0a010b) 10.10.1.12(VM-14:00000a0a010c) 10.10.1.2(VM-2:UNKNOWN) 10.10.1.4(VM-4:UNKNOWN) 10.10.1.6(VM-6:UNKNOWN) 10.10.1.10(VM-10:UNKNOWN) 10.10.1.13(VM-15:UNKNOWN) || * Note, i semi-automated getting that information from the manifest using awk, as follows: * Download the XML data from the page (copy/paste) * Find every line that starts with `` to it * Find the node assignments: {{{ grep "