= OG-MON-3: GENI Active Experiment Inspection Test = This page captures status for the test case OG-MON-3. For additional information see the [wiki:GENIRacksHome/OpenGENIRacks/AcceptanceTestStatusDec2013 Acceptance Test Status - December 2013] page overall status, or the [wiki:GENIRacksHome/AcceptanceTests/OpenGENIAcceptanceTestsPlan OpenGENI Acceptance Test Plan] for details about the planned evaluation. ''Last Update: 2013/12/13" || '''Step''' || '''State'''||''' Notes ''' || '''Tickets''' || || Step 1 ||[[span(Pass, style=background-color: green )]] || || || || Step 2 ||[[span(Pass, style=background-color: green )]] || || || || Step 3 || [[span(Pass, style=background-color: green )]] || || || [[BR]] || '''State Legend''' || '''Description''' || ||[[span(Pass, style=background-color: green )]] || Test completed and met all criteria || ||[[span(Pass: most criteria, style=background-color: #98FB98)]]|| Test completed and met most criteria. Exceptions documented || ||[[span(Fail, style=background-color: red)]] || Test completed and failed to meet criteria. || ||[[span(Complete, style=background-color: yellow)]] || Test completed but will require re-execution due to expected changes || ||[[span(Blocked, style=background-color: orange)]] || Blocked by ticketed issue(s). || ||[[span(In Progress, style=background-color: #63B8FF)]]|| Currently under test. || ||[[span(Not Planned)]] || This area is not part of initial evaluation || = Test Plan Steps = == Step 1. Set up experiments == An experimenter starts up experiments to ensure there is data to look at: * An experimenter runs an experiment containing at least one rack !OpenStack VM, and terminates it. * An experimenter runs an experiment containing at least one rack !OpenStack VM, and leaves it running. Several experiments were set up for OG-EXP-2 and OG-EXP-1. Some slivers were active and some deleted. == Step 2. Administrator reviews active and historical experiment data == A site administrator uses available system and experiment data sources to determine current experimental state, including: * How many VMs are running and which experimenters own them * How many physical hosts are in use by experiments, and which experimenters own them * How many VMs were terminated within the past day, and which experimenters owned them * What !OpenFlow controllers the data plane switch and the rack VMOC are communicating with On the Control Node in the directory /etc/gram/snapshots/gram, review the snapshot for the time frame when the experiments were run: {{{ lnevers@bbn-cam-ctrl-1:~$ python /etc/gram/dump_gram_snapshot.py --directory ./output/ --snapshot /etc/gram/snapshots/gram/2013_12_13_11_21_35_0.json ... Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-1 Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vm8d800a50-4bd1-46ac-ae99-01f3b1ea5b22 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vmec760da3-fe7e-4967-9ea0-2c6944acc32d User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interfacebe3c4381-8393-4065-9aec-2e3b86271b46 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+linkf6e67b6a-c5b8-40d8-93e3-36e5ca86efcf User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interfaceb356b302-13d7-47a6-8bf0-036d54d7b484 User: urn:publicid:IDN+ch.geni.net+user+lnevers .... lnevers@bbn-cam-ctrl-1:~$ python /etc/gram/dump_gram_snapshot.py --directory ./output/ --snapshot /etc/gram/snapshots/gram/2013_12_13_12_37_27_0.json Dumping snapshot /etc/gram/snapshots/gram/2013_12_13_12_37_27_0.json: Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-2-exp1 Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vmad31c104-ed08-4342-8541-c9ac035d2a0a User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vm12e67b21-ef66-4b8f-8e23-302dc3881570 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface6ec551f0-92a7-4aae-9676-e983336edfa1 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface41e93cab-503f-47c6-b50f-7ec572a0be64 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+link0d3f51ce-5aca-4366-bbee-c2e2865fb4e7 User: urn:publicid:IDN+ch.geni.net+user+lnevers .... Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-2-exp1 Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vmad31c104-ed08-4342-8541-c9ac035d2a0a User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vm12e67b21-ef66-4b8f-8e23-302dc3881570 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface6ec551f0-92a7-4aae-9676-e983336edfa1 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface41e93cab-503f-47c6-b50f-7ec572a0be64 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+link0d3f51ce-5aca-4366-bbee-c2e2865fb4e7 User: urn:publicid:IDN+ch.geni.net+user+lnevers .... lnevers@bbn-cam-ctrl-1:~$ python /etc/gram/dump_gram_snapshot.py --directory ./output/ --snapshot /etc/gram/snapshots/gram/2013_12_13_13_12_54_0.json Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-2-exp1 Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vmad31c104-ed08-4342-8541-c9ac035d2a0a User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vm12e67b21-ef66-4b8f-8e23-302dc3881570 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface6ec551f0-92a7-4aae-9676-e983336edfa1 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface41e93cab-503f-47c6-b50f-7ec572a0be64 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+link0d3f51ce-5aca-4366-bbee-c2e2865fb4e7 User: urn:publicid:IDN+ch.geni.net+user+lnevers Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-1 Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-1-exp2 Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interfacecf75916a-aad2-41a5-a278-c1d2e8dace9b User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+linkaf9f19cb-b0e3-40ed-97b4-29ba5c927763 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vmda02ca76-aeb0-4ca6-acfe-ced39859ac50 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface1229bf8c-b8d7-46ce-b90e-d64effc2d408 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vm2553f5e1-6d11-4f18-b000-caac56ce3d82 User: urn:publicid:IDN+ch.geni.net+user+lnevers }}} Note: Delete sliver show as as slice entry only, above show OG-EXP-1 sliver had been deleted. It is possible to determine VMs used and currently in use and "mac_address" The information is in the snapshots. Here is an example: {{{ lnevers@bbn-cam-ctrl-1:~$ python /etc/gram/dump_gram_snapshot.py --directory ./output/ --snapshot /etc/gram/snapshots/gram/2013_12_13_13_12_54_0.json .... "tenant_admin_name": "admin-ch.geni.net:ln-prj+slice+OG-EXP-2-exp1", "tenant_admin_pwd": "sliceMaster:-)", "tenant_admin_uuid": "7009bb8e095c4157af1183c018d395b8", "tenant_name": "ch.geni.net:ln-prj+slice+OG-EXP-2-exp1", "tenant_router_name": "R-ch.geni.net:ln-prj+slice+OG-EXP-2-exp1", "tenant_router_uuid": "959f2496-f86f-4f33-8345-097e55a662cc", "tenant_uuid": "9c99e3ebe4774d319b429ca7aea78d7d", "user_urn": null } manifest: ... "__type__": "NetworkInterface", "allocation_state": "geni_allocated", "creation": 1386974247.0, "device_number": null, "expiration": null, "ip_address": "172.16.1.1", "link": "urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+link0d3f51ce-5aca-4366-bbee-c2e2865fb4e7", "mac_address": "fa:16:3e:53:1f:31", "manifest_rspec": null, "name": "VM-1:if0", "operational_state": "geni_notready", "request_rspec": null, "slice": "9c99e3ebe4774d319b429ca7aea78d7d", "sliver_urn": "urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface6ec551f0-92a7-4aae-9676-e983336edfa1", "user_urn": "urn:publicid:IDN+ch.geni.net+user+lnevers", "uuid": "4e5abe07-b9fa-48ad-9198-ae068ed1e0d8", "virtual_machine": "urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vm12e67b21-ef66-4b8f-8e23-302dc3881570", "vlan_tag": null .... { }}} It is possible also possible to get information directly from the VMOC controller about the active experiments: {{{ lnevers@bbn-cam-ctrl-1:~$ echo "dump" |nc localhost 7001 VMOCSwitchControllerMap: Switches: [VMOCControllerConnection DPID 282037934860467836 URL https://localhost:9000 VLAN 1001] [VMOCControllerConnection DPID 282319409837178492 URL https://localhost:9000 VLAN 1002] Switches(unindexed): Switches (by Controller): [VMOCControllerConnection DPID 282319409837178492 URL https://localhost:9000 VLAN 1002] [VMOCControllerConnection DPID 282037934860467836 URL https://localhost:9000 VLAN 1001] Controllers (by VLAN) 1001 [VMOCControllerConnection DPID 282037934860467836 URL https://localhost:9000 VLAN 1001] 1002 [VMOCControllerConnection DPID 282319409837178492 URL https://localhost:9000 VLAN 1002] 1003 VMOC Slice Registry: urn:publicid:IDN+ch.geni.net:OpenGENI+slice+b0: {'vlan_configurations': [], 'slice_id': u'urn:publicid:IDN+ch.geni.net:OpenGENI+slice+b0'} urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln2routable: {'vlan_configurations': [{'vlan': 1001, 'controller_url': 'https://localhost:9000'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln2routable'} urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-2-exp1: {'vlan_configurations': [{'vlan': 1002, 'controller_url': 'https://localhost:9000'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-2-exp1'} urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln2server: {'vlan_configurations': [], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln2server'} https://localhost:9000 {'vlan_configurations': [{'vlan': 1001, 'controller_url': 'https://localhost:9000'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln2routable'} {'vlan_configurations': [{'vlan': 1002, 'controller_url': 'https://localhost:9000'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-2-exp1'} 1001: {'vlan_configurations': [{'vlan': 1001, 'controller_url': 'https://localhost:9000'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln2routable'} 1002: {'vlan_configurations': [{'vlan': 1002, 'controller_url': 'https://localhost:9000'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-2-exp1'} lnevers@bbn-cam-ctrl-1:~$ }}} == Step 3. Administrator reviews network data == A site administrator examines the switches and other rack data sources, and determines: * What MAC addresses are currently visible on the data plane switch and what experiments do they belong to? * For some experiment which was terminated within the past day, what data plane and control MAC and IP addresses did the experiment use? * For some experimental data path which is actively sending traffic on the data plane switch, do changes in interface counters show approximately the expected amount of traffic into and out of the switch? All required information is available to determine VMs used and in use in the snapshots. Also snapshot capture "mac_address", IP addresses and experimenter information for each VM allocated. Information found is sufficient. Switches show counters and mac_addresses in use. For the !PowerConnect 7048 switch: {{{ lnevers@bbn-cam-ctrl-1:~$ ssh admin@10.10.8.201 console>ena console#show interfaces status Port Name Duplex Speed Neg Link Flow Control State Status --------- ------------------------- ------ ------- ---- ------ ------------ Gi1/0/1 N/A Unknown Auto Down Inactive Gi1/0/2 N/A Unknown Auto Down Inactive Gi1/0/3 N/A Unknown Auto Down Inactive Gi1/0/4 N/A Unknown Auto Down Inactive Gi1/0/5 N/A Unknown Auto Down Inactive Gi1/0/6 N/A Unknown Auto Down Inactive Gi1/0/7 N/A Unknown Auto Down Inactive Gi1/0/8 N/A Unknown Auto Down Inactive Gi1/0/9 N/A Unknown Auto Down Inactive Gi1/0/10 N/A Unknown Auto Down Inactive Gi1/0/11 N/A Unknown Auto Down Inactive Gi1/0/12 N/A Unknown Auto Down Inactive Gi1/0/13 Full 1000 Auto Up Active Gi1/0/14 Full 1000 Auto Up Active Gi1/0/15 Full 1000 Auto Up Active Gi1/0/16 Full 1000 Auto Up Active Gi1/0/17 Full 1000 Auto Up Inactive Gi1/0/18 Full 1000 Auto Up Inactive Gi1/0/19 Full 1000 Auto Up Inactive --More-- or (q)uit console# ... many lines deleted ... console#show mac address-table Aging time is 300 Sec Vlan Mac Address Type Port -------- --------------------- ----------- --------------------- 1 D067.E5B8.FFFA Management Vl1 200 0001.E88A.FE7E Dynamic Gi1/0/16 200 90B1.1C0F.F2EB Dynamic Gi1/0/13 200 90B1.1C0F.FCAC Dynamic Gi1/0/14 200 90B1.1C48.41DA Dynamic Gi1/0/15 200 D067.E5B8.FFF9 Dynamic Gi1/0/17 300 0000.0C07.AC7E Dynamic Gi1/0/48 300 0000.858A.B566 Dynamic Gi1/0/48 300 0003.BA8C.A589 Dynamic Gi1/0/48 300 000E.0CB1.72DA Dynamic Gi1/0/48 300 0010.1833.EEED Dynamic Gi1/0/48 300 0011.1150.5146 Dynamic Gi1/0/48 300 0015.5D48.1405 Dynamic Gi1/0/48 300 0015.5D48.1407 Dynamic Gi1/0/48 300 0016.7645.2AA8 Dynamic Gi1/0/48 300 0019.B941.5B89 Dynamic Gi1/0/48 300 0019.B9D0.5405 Dynamic Gi1/0/48 ... many lines deleted ... }}} For the Force10 S4810P !OpenFlow switch: {{{ lnevers@bbn-cam-ctrl-1:~$ ssh admin@10.10.8.200 admin@10.10.8.200's password: FTOS>ena Password: FTOS#show interfaces TenGigabitEthernet 0/0 is up, line protocol is up Hardware is DellForce10Eth, address is 00:01:e8:8a:fe:7e Current address is 00:01:e8:8a:fe:7e Pluggable media present, SFP+ type is 10GBASE-CU2M Medium is MultiRate Interface index is 33620994 Internet address is not set MTU 1554 bytes, IP MTU 1500 bytes LineSpeed 10000 Mbit Flowcontrol rx on tx off ARP type: ARPA, ARP Timeout 04:00:00 Last clearing of "show interface" counters 1w3d19h Queueing strategy: fifo Input Statistics: 235820 packets, 51491703 bytes 118308 64-byte pkts, 504 over 64-byte pkts, 0 over 127-byte pkts 117008 over 255-byte pkts, 0 over 511-byte pkts, 0 over 1023-byte pkts 329 Multicasts, 149 Broadcasts 0 runts, 0 giants, 0 throttles 0 CRC, 0 overrun, 0 discarded Output Statistics: 267348 packets, 50755526 bytes, 0 underruns 147706 64-byte pkts, 2293 over 64-byte pkts, 35 over 127-byte pkts FTOS# ... many lines deleted ... FTOS#show mac-address-table Codes: *N - VLT Peer Synced MAC VlanId Mac Address Type Interface State 1 a0:36:9f:11:0d:60 Dynamic Te 0/1 Active FTOS# ... many lines deleted ... }}}