OG-MON-3: GENI Active Experiment Inspection Test
This page captures status for the test case OG-MON-3. For additional information see the Acceptance Test Status - December 2013 page overall status, or the OpenGENI Acceptance Test Plan for details about the planned evaluation.
Last Update: 2013/12/13"
| Step | State | Notes | Tickets |
| Step 1 | Pass | ||
| Step 2 | Pass | ||
| Step 3 | Pass |
| State Legend | Description |
| Pass | Test completed and met all criteria |
| Pass: most criteria | Test completed and met most criteria. Exceptions documented |
| Fail | Test completed and failed to meet criteria. |
| Complete | Test completed but will require re-execution due to expected changes |
| Blocked | Blocked by ticketed issue(s). |
| In Progress | Currently under test. |
| Not Planned | This area is not part of initial evaluation |
Test Plan Steps
Step 1. Set up experiments
An experimenter starts up experiments to ensure there is data to look at:
- An experimenter runs an experiment containing at least one rack OpenStack VM, and terminates it.
- An experimenter runs an experiment containing at least one rack OpenStack VM, and leaves it running.
Several experiments were set up for OG-EXP-2 and OG-EXP-1. Some slivers were active and some deleted.
Step 2. Administrator reviews active and historical experiment data
A site administrator uses available system and experiment data sources to determine current experimental state, including:
- How many VMs are running and which experimenters own them
- How many physical hosts are in use by experiments, and which experimenters own them
- How many VMs were terminated within the past day, and which experimenters owned them
- What OpenFlow controllers the data plane switch and the rack VMOC are communicating with
On the Control Node in the directory /etc/gram/snapshots/gram, review the snapshot for the time frame when the experiments were run:
lnevers@bbn-cam-ctrl-1:~$ python /etc/gram/dump_gram_snapshot.py --directory ./output/ --snapshot /etc/gram/snapshots/gram/2013_12_13_11_21_35_0.json ... Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-1 Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vm8d800a50-4bd1-46ac-ae99-01f3b1ea5b22 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vmec760da3-fe7e-4967-9ea0-2c6944acc32d User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interfacebe3c4381-8393-4065-9aec-2e3b86271b46 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+linkf6e67b6a-c5b8-40d8-93e3-36e5ca86efcf User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interfaceb356b302-13d7-47a6-8bf0-036d54d7b484 User: urn:publicid:IDN+ch.geni.net+user+lnevers .... lnevers@bbn-cam-ctrl-1:~$ python /etc/gram/dump_gram_snapshot.py --directory ./output/ --snapshot /etc/gram/snapshots/gram/2013_12_13_12_37_27_0.json Dumping snapshot /etc/gram/snapshots/gram/2013_12_13_12_37_27_0.json: Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-2-exp1 Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vmad31c104-ed08-4342-8541-c9ac035d2a0a User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vm12e67b21-ef66-4b8f-8e23-302dc3881570 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface6ec551f0-92a7-4aae-9676-e983336edfa1 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface41e93cab-503f-47c6-b50f-7ec572a0be64 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+link0d3f51ce-5aca-4366-bbee-c2e2865fb4e7 User: urn:publicid:IDN+ch.geni.net+user+lnevers .... Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-2-exp1 Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vmad31c104-ed08-4342-8541-c9ac035d2a0a User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vm12e67b21-ef66-4b8f-8e23-302dc3881570 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface6ec551f0-92a7-4aae-9676-e983336edfa1 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface41e93cab-503f-47c6-b50f-7ec572a0be64 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+link0d3f51ce-5aca-4366-bbee-c2e2865fb4e7 User: urn:publicid:IDN+ch.geni.net+user+lnevers .... lnevers@bbn-cam-ctrl-1:~$ python /etc/gram/dump_gram_snapshot.py --directory ./output/ --snapshot /etc/gram/snapshots/gram/2013_12_13_13_12_54_0.json Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-2-exp1 Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vmad31c104-ed08-4342-8541-c9ac035d2a0a User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vm12e67b21-ef66-4b8f-8e23-302dc3881570 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface6ec551f0-92a7-4aae-9676-e983336edfa1 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface41e93cab-503f-47c6-b50f-7ec572a0be64 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+link0d3f51ce-5aca-4366-bbee-c2e2865fb4e7 User: urn:publicid:IDN+ch.geni.net+user+lnevers Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-1 Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-1-exp2 Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interfacecf75916a-aad2-41a5-a278-c1d2e8dace9b User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+linkaf9f19cb-b0e3-40ed-97b4-29ba5c927763 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vmda02ca76-aeb0-4ca6-acfe-ced39859ac50 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface1229bf8c-b8d7-46ce-b90e-d64effc2d408 User: urn:publicid:IDN+ch.geni.net+user+lnevers Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vm2553f5e1-6d11-4f18-b000-caac56ce3d82 User: urn:publicid:IDN+ch.geni.net+user+lnevers
Note: Delete sliver show as as slice entry only, above show OG-EXP-1 sliver had been deleted.
It is possible to determine VMs used and currently in use and "mac_address" The information is in the snapshots. Here is an example:
lnevers@bbn-cam-ctrl-1:~$ python /etc/gram/dump_gram_snapshot.py --directory ./output/ --snapshot /etc/gram/snapshots/gram/2013_12_13_13_12_54_0.json
....
"tenant_admin_name": "admin-ch.geni.net:ln-prj+slice+OG-EXP-2-exp1",
"tenant_admin_pwd": "sliceMaster:-)",
"tenant_admin_uuid": "7009bb8e095c4157af1183c018d395b8",
"tenant_name": "ch.geni.net:ln-prj+slice+OG-EXP-2-exp1",
"tenant_router_name": "R-ch.geni.net:ln-prj+slice+OG-EXP-2-exp1",
"tenant_router_uuid": "959f2496-f86f-4f33-8345-097e55a662cc",
"tenant_uuid": "9c99e3ebe4774d319b429ca7aea78d7d",
"user_urn": null
}
manifest:
<?xml version="1.0" ?>
<rspec type="manifest" xmlns="http://www.geni.net/resources/rspec/3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.geni.net/resources/rspec/3
http://www.geni.net/resources/rspec/3/manifest.xsd http://www.protogeni.net/resources/rspec/ext/emulab/1 http://www.protogeni.net/resources/rspec/ext/emulab/1/vtop_ex
tension.xsd">
<node client_id="VM-1" component_manager_id="urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf:gram+authority+am" exclusive="false" sliver_id="urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+v
m12e67b21-ef66-4b8f-8e23-302dc3881570">
<interface client_id="VM-1:if0" sliver_id="urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface6ec551f0-92a7-4aae-9676-e983336edfa1">
<ip address="172.16.1.1" type="ip"/>
</interface>
<sliver_type name="m1.small">
<disk_image name="urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+imageubuntu-12.04" os="Linux" version="12"/>
</sliver_type>
<host name="VM-1"/>
</node>
<node client_id="VM-2" component_manager_id="urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf:gram+authority+am" exclusive="false" sliver_id="urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+v
mad31c104-ed08-4342-8541-c9ac035d2a0a">
<interface client_id="VM-2:if0" sliver_id="urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface41e93cab-503f-47c6-b50f-7ec572a0be64">
<ip address="172.16.1.2" type="ip"/>
</interface>
<sliver_type name="m1.small">
<disk_image name="urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+imageubuntu-12.04" os="Linux" version="12"/>
</sliver_type>
<host name="VM-2"/>
</node>
<link client_id="link" sliver_id="urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+link0d3f51ce-5aca-4366-bbee-c2e2865fb4e7" vlantag="1002">
<interface_ref client_id="VM-1:if0"/>
<interface_ref client_id="VM-2:if0"/>
<property dest_id="VM-2:if0" source_id="VM-1:if0"/>
<property dest_id="VM-1:if0" source_id="VM-2:if0"/>
</link>
</rspec>
...
"__type__": "NetworkInterface",
"allocation_state": "geni_allocated",
"creation": 1386974247.0,
"device_number": null,
"expiration": null,
"ip_address": "172.16.1.1",
"link": "urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+link0d3f51ce-5aca-4366-bbee-c2e2865fb4e7",
"mac_address": "fa:16:3e:53:1f:31",
"manifest_rspec": null,
"name": "VM-1:if0",
"operational_state": "geni_notready",
"request_rspec": null,
"slice": "9c99e3ebe4774d319b429ca7aea78d7d",
"sliver_urn": "urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface6ec551f0-92a7-4aae-9676-e983336edfa1",
"user_urn": "urn:publicid:IDN+ch.geni.net+user+lnevers",
"uuid": "4e5abe07-b9fa-48ad-9198-ae068ed1e0d8",
"virtual_machine": "urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vm12e67b21-ef66-4b8f-8e23-302dc3881570",
"vlan_tag": null
....
{
It is possible also possible to get information directly from the VMOC controller about the active experiments:
lnevers@bbn-cam-ctrl-1:~$ echo "dump" |nc localhost 7001
VMOCSwitchControllerMap:
Switches:
<vmoc.VMOCSwitchConnection.VMOCSwitchConnection object at 0x2360cd0>
[VMOCControllerConnection DPID 282037934860467836 URL https://localhost:9000 VLAN 1001]
[VMOCControllerConnection DPID 282319409837178492 URL https://localhost:9000 VLAN 1002]
Switches(unindexed):
<vmoc.VMOCSwitchConnection.VMOCSwitchConnection object at 0x2360cd0>
Switches (by Controller):
[VMOCControllerConnection DPID 282319409837178492 URL https://localhost:9000 VLAN 1002]
<vmoc.VMOCSwitchConnection.VMOCSwitchConnection object at 0x2360cd0>
[VMOCControllerConnection DPID 282037934860467836 URL https://localhost:9000 VLAN 1001]
<vmoc.VMOCSwitchConnection.VMOCSwitchConnection object at 0x2360cd0>
Controllers (by VLAN)
1001
[VMOCControllerConnection DPID 282037934860467836 URL https://localhost:9000 VLAN 1001]
1002
[VMOCControllerConnection DPID 282319409837178492 URL https://localhost:9000 VLAN 1002]
1003
VMOC Slice Registry:
urn:publicid:IDN+ch.geni.net:OpenGENI+slice+b0: {'vlan_configurations': [], 'slice_id': u'urn:publicid:IDN+ch.geni.net:OpenGENI+slice+b0'}
urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln2routable: {'vlan_configurations': [{'vlan': 1001, 'controller_url': 'https://localhost:9000'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln2routable'}
urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-2-exp1: {'vlan_configurations': [{'vlan': 1002, 'controller_url': 'https://localhost:9000'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-2-exp1'}
urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln2server: {'vlan_configurations': [], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln2server'}
https://localhost:9000
{'vlan_configurations': [{'vlan': 1001, 'controller_url': 'https://localhost:9000'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln2routable'}
{'vlan_configurations': [{'vlan': 1002, 'controller_url': 'https://localhost:9000'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-2-exp1'}
1001: {'vlan_configurations': [{'vlan': 1001, 'controller_url': 'https://localhost:9000'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln2routable'}
1002: {'vlan_configurations': [{'vlan': 1002, 'controller_url': 'https://localhost:9000'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-2-exp1'}
lnevers@bbn-cam-ctrl-1:~$
Step 3. Administrator reviews network data
A site administrator examines the switches and other rack data sources, and determines:
- What MAC addresses are currently visible on the data plane switch and what experiments do they belong to?
- For some experiment which was terminated within the past day, what data plane and control MAC and IP addresses did the experiment use?
- For some experimental data path which is actively sending traffic on the data plane switch, do changes in interface counters show approximately the expected amount of traffic into and out of the switch?
All required information is available to determine VMs used and in use in the snapshots. Also snapshot capture "mac_address", IP addresses and experimenter information for each VM allocated. Information found is sufficient. Switches show counters and mac_addresses in use.
For the PowerConnect 7048 switch:
lnevers@bbn-cam-ctrl-1:~$ ssh admin@10.10.8.201
console>ena
console#show interfaces status
Port Name Duplex Speed Neg Link Flow Control
State Status
--------- ------------------------- ------ ------- ---- ------ ------------
Gi1/0/1 N/A Unknown Auto Down Inactive
Gi1/0/2 N/A Unknown Auto Down Inactive
Gi1/0/3 N/A Unknown Auto Down Inactive
Gi1/0/4 N/A Unknown Auto Down Inactive
Gi1/0/5 N/A Unknown Auto Down Inactive
Gi1/0/6 N/A Unknown Auto Down Inactive
Gi1/0/7 N/A Unknown Auto Down Inactive
Gi1/0/8 N/A Unknown Auto Down Inactive
Gi1/0/9 N/A Unknown Auto Down Inactive
Gi1/0/10 N/A Unknown Auto Down Inactive
Gi1/0/11 N/A Unknown Auto Down Inactive
Gi1/0/12 N/A Unknown Auto Down Inactive
Gi1/0/13 Full 1000 Auto Up Active
Gi1/0/14 Full 1000 Auto Up Active
Gi1/0/15 Full 1000 Auto Up Active
Gi1/0/16 Full 1000 Auto Up Active
Gi1/0/17 Full 1000 Auto Up Inactive
Gi1/0/18 Full 1000 Auto Up Inactive
Gi1/0/19 Full 1000 Auto Up Inactive
--More-- or (q)uit
console#
... many lines deleted ...
console#show mac address-table
Aging time is 300 Sec
Vlan Mac Address Type Port
-------- --------------------- ----------- ---------------------
1 D067.E5B8.FFFA Management Vl1
200 0001.E88A.FE7E Dynamic Gi1/0/16
200 90B1.1C0F.F2EB Dynamic Gi1/0/13
200 90B1.1C0F.FCAC Dynamic Gi1/0/14
200 90B1.1C48.41DA Dynamic Gi1/0/15
200 D067.E5B8.FFF9 Dynamic Gi1/0/17
300 0000.0C07.AC7E Dynamic Gi1/0/48
300 0000.858A.B566 Dynamic Gi1/0/48
300 0003.BA8C.A589 Dynamic Gi1/0/48
300 000E.0CB1.72DA Dynamic Gi1/0/48
300 0010.1833.EEED Dynamic Gi1/0/48
300 0011.1150.5146 Dynamic Gi1/0/48
300 0015.5D48.1405 Dynamic Gi1/0/48
300 0015.5D48.1407 Dynamic Gi1/0/48
300 0016.7645.2AA8 Dynamic Gi1/0/48
300 0019.B941.5B89 Dynamic Gi1/0/48
300 0019.B9D0.5405 Dynamic Gi1/0/48
... many lines deleted ...
For the Force10 S4810P OpenFlow switch:
lnevers@bbn-cam-ctrl-1:~$ ssh admin@10.10.8.200
admin@10.10.8.200's password:
FTOS>ena
Password:
FTOS#show interfaces
TenGigabitEthernet 0/0 is up, line protocol is up
Hardware is DellForce10Eth, address is 00:01:e8:8a:fe:7e
Current address is 00:01:e8:8a:fe:7e
Pluggable media present, SFP+ type is 10GBASE-CU2M
Medium is MultiRate
Interface index is 33620994
Internet address is not set
MTU 1554 bytes, IP MTU 1500 bytes
LineSpeed 10000 Mbit
Flowcontrol rx on tx off
ARP type: ARPA, ARP Timeout 04:00:00
Last clearing of "show interface" counters 1w3d19h
Queueing strategy: fifo
Input Statistics:
235820 packets, 51491703 bytes
118308 64-byte pkts, 504 over 64-byte pkts, 0 over 127-byte pkts
117008 over 255-byte pkts, 0 over 511-byte pkts, 0 over 1023-byte pkts
329 Multicasts, 149 Broadcasts
0 runts, 0 giants, 0 throttles
0 CRC, 0 overrun, 0 discarded
Output Statistics:
267348 packets, 50755526 bytes, 0 underruns
147706 64-byte pkts, 2293 over 64-byte pkts, 35 over 127-byte pkts
FTOS#
... many lines deleted ...
FTOS#show mac-address-table
Codes: *N - VLT Peer Synced MAC
VlanId Mac Address Type Interface State
1 a0:36:9f:11:0d:60 Dynamic Te 0/1 Active
FTOS#
... many lines deleted ...
Last modified 10 years ago
Last modified on 05/23/14 13:31:51
