= OG-MON-3: GENI Active Experiment Inspection Test = This page captures status for the test case OG-MON-3. For additional information see the [wiki:GENIRacksHome/OpenGENIRacks/AcceptanceTestStatusApr2014 Acceptance Test Status - April 2014] page overall status, or the [wiki:GENIRacksHome/AcceptanceTests/OpenGENIAcceptanceTestsPlan OpenGENI Acceptance Test Plan] for details about the planned evaluation. || '''Step''' || '''State'''||''' Notes ''' || '''Tickets''' || || Step 1 ||[[span(Pass, style=background-color: green )]] || || || || Step 2 ||[[span(Pass, style=background-color: green )]] || || || || Step 3 ||[[span(Pass, style=background-color: green )]] || || || [[BR]] || '''State Legend''' || '''Description''' || ||[[span(Pass, style=background-color: green )]] || Test completed and met all criteria || ||[[span(Pass: most criteria, style=background-color: #98FB98)]]|| Test completed and met most criteria. Exceptions documented || ||[[span(Fail, style=background-color: red)]] || Test completed and failed to meet criteria. || ||[[span(Complete, style=background-color: yellow)]] || Test completed but will require re-execution due to expected changes || ||[[span(Blocked, style=background-color: orange)]] || Blocked by ticketed issue(s). || ||[[span(In Progress, style=background-color: #63B8FF)]]|| Currently under test. || ||[[span(Not Planned)]] || This area is not part of initial evaluation || = Test Plan Steps = == Step 1. Set up experiments == An experimenter starts up experiments to ensure there is data to look at: * An experimenter runs an experiment containing at least one rack !OpenStack VM, and terminates it. * An experimenter runs an experiment containing at least one rack !OpenStack VM, and leaves it running. One experiment was set up named "stitchOF" which included 1VM at Clemson using a stitched path to GPO IG and running !OpenFlow. {{{ gram@clemson-clemson-control-1:~$ nova list --all-tenants +--------------------------------------+---------+--------+------------------------------------------------+ | ID | Name | Status | Networks | +--------------------------------------+---------+--------+------------------------------------------------+ | e4b5ce2c-ea78-4e39-acbe-1157af2e458d | gr-clem | ACTIVE | linkOF=192.168.1.2; GRAM-mgmt-net=192.168.10.3 | +--------------------------------------+---------+--------+------------------------------------------------+ gram@clemson-clemson-control-1:~$ keystone tenant-list +----------------------------------+------------------------------------+---------+ | id | name | enabled | +----------------------------------+------------------------------------+---------+ | 89a03e397f8b4743be8d6c0c470e9ede | admin | True | | 65c073cb3f83446eb400d76013f71bc0 | ch.geni.net:ln-test+slice+stitchOF | True | | 32b79311d95b4483a6d40a6db316fa3f | service | True | +----------------------------------+------------------------------------+---------+ }}} == Step 2. Administrator reviews active and historical experiment data == A site administrator uses available system and experiment data sources to determine current experimental state, including: * How many VMs are running and which experimenters own them * How many physical hosts are in use by experiments, and which experimenters own them * How many VMs were terminated within the past day, and which experimenters owned them * What !OpenFlow controllers the data plane switch and the rack VMOC are communicating with On the Control Node in the directory /etc/gram/snapshots/gram, review the snapshot for the time frame when the experiment was run and verify an entry for the sliver created at the beginning of this test. It is possible also possible to get information directly from the VMOC controller about the active experiments: {{{ gram@clemson-clemson-control-1:~$ echo "dump" |nc localhost 7001 VMOCSwitchControllerMap: Switches: Switches(unindexed): Switches (by Controller): Controllers (by VLAN) 3006 3007 VMOC Slice Registry: urn:publicid:IDN+ch.geni.net:ln-test+slice+stitchOF: {'vlan_configurations': [{'vlan': 3007, 'controller_url': u'tcp:192.1.249.185:33020'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-test+slice+stitchOF'} urn:publicid:IDN+ch.geni.net:ln-test+slice+gr-clem-eg-gpo: {'vlan_configurations': [], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-test+slice+gr-clem-eg-gpo'} tcp:192.1.249.185:33020 {'vlan_configurations': [{'vlan': 3007, 'controller_url': u'tcp:192.1.249.185:33020'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-test+slice+stitchOF'} tcp:192.1.249.185:33019 3007: {'vlan_configurations': [{'vlan': 3007, 'controller_url': u'tcp:192.1.249.185:33020'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-test+slice+stitchOF'} gram@clemson-clemson-control-1:~$ }}} == Step 3. Administrator reviews network data == A site administrator examines the switches and other rack data sources, and determines: * What MAC addresses are currently visible on the data plane switch and what experiments do they belong to? * For some experiment which was terminated within the past day, what data plane and control MAC and IP addresses did the experiment use? * For some experimental data path which is actively sending traffic on the data plane switch, do changes in interface counters show approximately the expected amount of traffic into and out of the switch? All required information is available to determine VMs used and in use in the snapshots. Also snapshot capture "mac_address", IP addresses and experimenter information for each VM allocated. Information found is sufficient. One example experiment OG-EXP-2-exp1 has two nodes running, login to the first node and get MAC: {{{ $ ssh -p 3000 -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@130.127.88.101 ... lnevers@vm-1:~$ /sbin/ifconfig ... eth1 Link encap:Ethernet HWaddr fa:16:3e:ce:9d:16 inet addr:192.168.1.1 Bcast:192.168.255.255 Mask:255.255.0.0 ... }}} Second node in experiment: {{{ $ ssh -p 3001 -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@130.127.88.101 ... lnevers@gr-clem:~$ /sbin/ifconfig ... eth1 Link encap:Ethernet HWaddr fa:16:3e:f0:67:4b inet addr:192.168.1.2 Bcast:192.168.255.255 Mask:255.255.0.0 ... }}} Switches information shows that counters and MAC addresses for those nodes in use. For the !PowerConnect 7048 switch: {{{ gram@clemson-clemson-control-1:~$ ssh lnevers@10.10.8.201 lnevers@10.10.8.201's password: PCT7048>ena Password:*************** PCT7048#show mac address-table Aging time is 300 Sec Vlan Mac Address Type Port -------- --------------------- ----------- --------------------- 1 D067.E5A9.B19F Management Vl1 200 0001.E8D7.589D Dynamic Gi1/0/41 200 90B1.1C0F.8A79 Dynamic Gi1/0/14 200 90B1.1C0F.D649 Dynamic Gi1/0/13 200 90B1.1C0F.F479 Dynamic Gi1/0/15 200 D067.E5A9.B19F Management Vl200 845 0007.7D26.29AA Dynamic Gi1/0/48 845 0017.DF2B.0800 Dynamic Gi1/0/48 845 001A.A02F.AD0B Dynamic Gi1/0/48 845 0026.B975.1D45 Dynamic Gi1/0/48 845 0026.B983.F9A4 Dynamic Gi1/0/48 845 0026.F13B.8A00 Dynamic Gi1/0/48 845 1803.73AD.5CA9 Dynamic Gi1/0/48 845 5254.0043.1FF4 Dynamic Gi1/0/48 845 782B.CB8F.740E Dynamic Gi1/0/48 845 782B.CB8F.E7C4 Dynamic Gi1/0/48 845 842B.2B19.E4D9 Dynamic Gi1/0/48 845 842B.2B19.E4DB Dynamic Gi1/0/48 845 D067.E5A9.B19F Management Vl845 845 D4BE.D98A.3A47 Dynamic Gi1/0/48 872 0007.7D26.29AA Dynamic Gi1/0/48 872 0023.5EA9.5CCA Dynamic Gi1/0/48 872 A036.9F11.0AB2 Dynamic Gi1/0/25 872 A036.9F11.0C4E Dynamic Gi1/0/27 872 A036.9F11.0D32 Dynamic Gi1/0/26 872 C81F.66B7.9C64 Dynamic Gi1/0/48 872 C81F.66B7.9C66 Dynamic Gi1/0/48 872 D067.E5A9.B19F Management Vl872 872 FA16.3EED.2D4C Dynamic Gi1/0/25 2500 FA16.3E1F.532C Dynamic Gi1/0/2 2500 FA16.3E5C.923D Dynamic Gi1/0/1 2500 FA16.3E8F.1AB7 Dynamic Gi1/0/2 2500 FA16.3E92.5863 Dynamic Gi1/0/1 Total MAC Addresses in use: 33 PCT7048#show arp Static ARP entries are only active when the IP address is reachable on a local subnet Age Time (seconds)............................. 1200 Response Time (seconds)........................ 1 Retries........................................ 4 Cache Size..................................... 6144 Dynamic Renew Mode ............................ Disable Total Entry Count Current / Peak .............. 5 / 5 Static Entry Count Configured / Active / Max .. 0 / 0 / 128 IP Address MAC Address Interface Type Age --------------- ----------------- -------------- -------- ----------- 10.10.8.100 90B1.1C0F.D649 Vl200 Dynamic 0h 6m 15s 10.10.8.201 D067.E5A9.B19F Vl200 Local n/a 130.127.39.129 0017.DF2B.0800 Vl845 Dynamic 0h 0m 55s 130.127.39.136 D067.E5A9.B19F Vl845 Local n/a 130.127.39.245 001A.A02F.AD0B Vl845 Dynamic 0h 0m 4s }}} For the Force10 S4810P !OpenFlow switch: -- Waiting on login setup -- {{{ gram@clemson-clemson-control-1:~$ ssh lnevers@10.10.8.200 lnevers@10.10.8.200's password: FTOS>ena Password: FTOS# FTOS#show mac-address-table Codes: *N - VLT Peer Synced MAC VlanId Mac Address Type Interface State 1 00:24:38:9c:13:00 Dynamic Te 0/3 Active 1 08:9e:01:18:01:47 Dynamic Te 0/3 Active FTOS# FTOS#show mac-address-table Codes: *N - VLT Peer Synced MAC VlanId Mac Address Type Interface State 1 00:24:38:9c:13:00 Dynamic Te 0/3 Active 1 08:9e:01:18:01:47 Dynamic Te 0/3 Active FTOS#show arp Protocol Address Age(min) Hardware Address Interface VLAN CPU ----------------------------------------------------------------------------------------- Internet 10.10.8.100 9 90:b1:1c:0f:d6:49 Ma 0/0 - CP FTOS# FTOS#show vlan id 1001 <<<=== experiment VLAN Codes: * - Default VLAN, G - GVRP VLANs, R - Remote Port Mirroring VLANs, P - Primary, C - Community, I - Isolated O - Openflow Q: U - Untagged, T - Tagged x - Dot1x untagged, X - Dot1x tagged o - OpenFlow untagged, O - OpenFlow tagged G - GVRP tagged, M - Vlan-stack, H - VSN tagged i - Internal untagged, I - Internal tagged, v - VLT untagged, V - VLT tagged NUM Status Description Q Ports O 1001 Active T Te 0/0-2 FTOS#show interfaces vlan 1001 description Interface OK Status Protocol Description Vlan 1001 YES up up FTOS#show interfaces vlan 1001 Vlan 1001 is up, line protocol is up Address is 00:01:e8:d7:58:9d, Current address is 00:01:e8:d7:58:9d Interface index is 1107526633 of-instance: 1 Internet address is not set MTU 1554 bytes, IP MTU 1500 bytes LineSpeed 10000 Mbit ARP type: ARPA, ARP Timeout 04:00:00 Last clearing of "show interface" counters 6w6d1h Queueing strategy: fifo Time since last interface status change: 6w6d1h FTOS# }}} The way to map an experiment to a slice is to look for the MAC address in the snapshot file. For this particular experiment the file /etc/gram/snapshots/gram/2014_04_23_13_18_13_0.json was reviewed and was able to map both MAC addresses back to the slice. There is a tool to dump the information in the snapshots to an easy to read version, but it is currently not working. If it was working, one would simply issue the following command on the control node: {{{ $python /etc/gram/dump_gram_snapshot.py --directory ./output/ --snapshot /etc/gram/snapshots/gram/2014_04_23_09_59_49_0.json }}}