wiki:GENIRacksHome/OpenGENIRacks/AcceptanceTestStatusDec2013/OG-MON-3

Version 2 (modified by lnevers@bbn.com, 5 years ago) (diff)

--

OG-MON-3: GENI Active Experiment Inspection Test

This page captures status for the test case OG-MON-3. For additional information see the Acceptance Test Status - December 2013 page overall status, or the OpenGENI Acceptance Test Plan for details about the planned evaluation.

Last Update: 2013/12/13"

Step State Notes Tickets
Step 1 Pass
Step 2 Pass
Step 3 Pass


State Legend Description
Pass Test completed and met all criteria
Pass: most criteria Test completed and met most criteria. Exceptions documented
Fail Test completed and failed to meet criteria.
Complete Test completed but will require re-execution due to expected changes
Blocked Blocked by ticketed issue(s).
In Progress Currently under test.
Not Planned This area is not part of initial evaluation

Test Plan Steps

Step 1. Set up experiments

An experimenter starts up experiments to ensure there is data to look at:

  • An experimenter runs an experiment containing at least one rack OpenStack VM, and terminates it.
  • An experimenter runs an experiment containing at least one rack OpenStack VM, and leaves it running.

Several experiments were set up for OG-EXP-2, and OG-EXP-1. Some slivers were active and some deleted.

Step 2. Administrator reviews active and historical experiment data

A site administrator uses available system and experiment data sources to determine current experimental state, including:

  • How many VMs are running and which experimenters own them
  • How many physical hosts are in use by experiments, and which experimenters own them
  • How many VMs were terminated within the past day, and which experimenters owned them
  • What OpenFlow controllers the data plane switch and the rack VMOC are communicating with

On the Control Node in the directory /etc/gram/snapshots/gram, review the snapshot for the time frame when the experiments were run:

lnevers@bbn-cam-ctrl-1:~$ python /etc/gram/dump_gram_snapshot.py --directory ./output/ --snapshot /etc/gram/snapshots/gram/2013_12_13_11_21_35_0.json 
...
Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-1
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vm8d800a50-4bd1-46ac-ae99-01f3b1ea5b22 User: urn:publicid:IDN+ch.geni.net+user+lnevers
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vmec760da3-fe7e-4967-9ea0-2c6944acc32d User: urn:publicid:IDN+ch.geni.net+user+lnevers
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interfacebe3c4381-8393-4065-9aec-2e3b86271b46 User: urn:publicid:IDN+ch.geni.net+user+lnevers
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+linkf6e67b6a-c5b8-40d8-93e3-36e5ca86efcf User: urn:publicid:IDN+ch.geni.net+user+lnevers
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interfaceb356b302-13d7-47a6-8bf0-036d54d7b484 User: urn:publicid:IDN+ch.geni.net+user+lnevers
....
lnevers@bbn-cam-ctrl-1:~$ python /etc/gram/dump_gram_snapshot.py --directory ./output/ --snapshot /etc/gram/snapshots/gram/2013_12_13_12_37_27_0.json 
Dumping snapshot /etc/gram/snapshots/gram/2013_12_13_12_37_27_0.json:
Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-2-exp1
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vmad31c104-ed08-4342-8541-c9ac035d2a0a User: urn:publicid:IDN+ch.geni.net+user+lnevers
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vm12e67b21-ef66-4b8f-8e23-302dc3881570 User: urn:publicid:IDN+ch.geni.net+user+lnevers
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface6ec551f0-92a7-4aae-9676-e983336edfa1 User: urn:publicid:IDN+ch.geni.net+user+lnevers
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface41e93cab-503f-47c6-b50f-7ec572a0be64 User: urn:publicid:IDN+ch.geni.net+user+lnevers
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+link0d3f51ce-5aca-4366-bbee-c2e2865fb4e7 User: urn:publicid:IDN+ch.geni.net+user+lnevers
....
Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-2-exp1
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vmad31c104-ed08-4342-8541-c9ac035d2a0a User: urn:publicid:IDN+ch.geni.net+user+lnevers
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vm12e67b21-ef66-4b8f-8e23-302dc3881570 User: urn:publicid:IDN+ch.geni.net+user+lnevers
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface6ec551f0-92a7-4aae-9676-e983336edfa1 User: urn:publicid:IDN+ch.geni.net+user+lnevers
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface41e93cab-503f-47c6-b50f-7ec572a0be64 User: urn:publicid:IDN+ch.geni.net+user+lnevers
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+link0d3f51ce-5aca-4366-bbee-c2e2865fb4e7 User: urn:publicid:IDN+ch.geni.net+user+lnevers
....


lnevers@bbn-cam-ctrl-1:~$ python /etc/gram/dump_gram_snapshot.py --directory ./output/ --snapshot /etc/gram/snapshots/gram/2013_12_13_13_12_54_0.json 
Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-2-exp1
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vmad31c104-ed08-4342-8541-c9ac035d2a0a User: urn:publicid:IDN+ch.geni.net+user+lnevers
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vm12e67b21-ef66-4b8f-8e23-302dc3881570 User: urn:publicid:IDN+ch.geni.net+user+lnevers
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface6ec551f0-92a7-4aae-9676-e983336edfa1 User: urn:publicid:IDN+ch.geni.net+user+lnevers
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface41e93cab-503f-47c6-b50f-7ec572a0be64 User: urn:publicid:IDN+ch.geni.net+user+lnevers
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+link0d3f51ce-5aca-4366-bbee-c2e2865fb4e7 User: urn:publicid:IDN+ch.geni.net+user+lnevers
Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-1
Slice urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-1-exp2
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interfacecf75916a-aad2-41a5-a278-c1d2e8dace9b User: urn:publicid:IDN+ch.geni.net+user+lnevers
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+linkaf9f19cb-b0e3-40ed-97b4-29ba5c927763 User: urn:publicid:IDN+ch.geni.net+user+lnevers
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vmda02ca76-aeb0-4ca6-acfe-ced39859ac50 User: urn:publicid:IDN+ch.geni.net+user+lnevers
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface1229bf8c-b8d7-46ce-b90e-d64effc2d408 User: urn:publicid:IDN+ch.geni.net+user+lnevers
   Sliver urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vm2553f5e1-6d11-4f18-b000-caac56ce3d82 User: urn:publicid:IDN+ch.geni.net+user+lnevers

Note: Delete sliver show as as slice entry only, above show OG-EXP-1 sliver had been deleted.

It is possible to determine VMs used and currently in use and "mac_address" The information is in the snapshots. Here is an example:

lnevers@bbn-cam-ctrl-1:~$ python /etc/gram/dump_gram_snapshot.py --directory ./output/ --snapshot /etc/gram/snapshots/gram/2013_12_13_13_12_54_0.json  
....
    "tenant_admin_name": "admin-ch.geni.net:ln-prj+slice+OG-EXP-2-exp1", 
    "tenant_admin_pwd": "sliceMaster:-)", 
    "tenant_admin_uuid": "7009bb8e095c4157af1183c018d395b8", 
    "tenant_name": "ch.geni.net:ln-prj+slice+OG-EXP-2-exp1", 
    "tenant_router_name": "R-ch.geni.net:ln-prj+slice+OG-EXP-2-exp1", 
    "tenant_router_uuid": "959f2496-f86f-4f33-8345-097e55a662cc", 
    "tenant_uuid": "9c99e3ebe4774d319b429ca7aea78d7d", 
    "user_urn": null
}

manifest:
<?xml version="1.0" ?>
<rspec type="manifest" xmlns="http://www.geni.net/resources/rspec/3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.geni.net/resources/rspec/3      
    http://www.geni.net/resources/rspec/3/manifest.xsd         http://www.protogeni.net/resources/rspec/ext/emulab/1         http://www.protogeni.net/resources/rspec/ext/emulab/1/vtop_ex
tension.xsd">
    <node client_id="VM-1" component_manager_id="urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf:gram+authority+am" exclusive="false" sliver_id="urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+v
m12e67b21-ef66-4b8f-8e23-302dc3881570">
        <interface client_id="VM-1:if0" sliver_id="urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface6ec551f0-92a7-4aae-9676-e983336edfa1">
            <ip address="172.16.1.1" type="ip"/>
        </interface>
        <sliver_type name="m1.small">
            <disk_image name="urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+imageubuntu-12.04" os="Linux" version="12"/>
        </sliver_type>
        <host name="VM-1"/>
    </node>
    <node client_id="VM-2" component_manager_id="urn:publicid:IDN+geni:bbn-cam-ctrl-1:gcf:gram+authority+am" exclusive="false" sliver_id="urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+v
mad31c104-ed08-4342-8541-c9ac035d2a0a">
        <interface client_id="VM-2:if0" sliver_id="urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface41e93cab-503f-47c6-b50f-7ec572a0be64">
            <ip address="172.16.1.2" type="ip"/>
        </interface>
        <sliver_type name="m1.small">
            <disk_image name="urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+imageubuntu-12.04" os="Linux" version="12"/>
        </sliver_type>
        <host name="VM-2"/>
    </node>
    <link client_id="link" sliver_id="urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+link0d3f51ce-5aca-4366-bbee-c2e2865fb4e7" vlantag="1002">
        <interface_ref client_id="VM-1:if0"/>
        <interface_ref client_id="VM-2:if0"/>
        <property dest_id="VM-2:if0" source_id="VM-1:if0"/>
        <property dest_id="VM-1:if0" source_id="VM-2:if0"/>
    </link>
</rspec>
...
    "__type__": "NetworkInterface", 
    "allocation_state": "geni_allocated", 
    "creation": 1386974247.0, 
    "device_number": null, 
    "expiration": null, 
    "ip_address": "172.16.1.1", 
    "link": "urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+link0d3f51ce-5aca-4366-bbee-c2e2865fb4e7", 
    "mac_address": "fa:16:3e:53:1f:31", 
    "manifest_rspec": null, 
    "name": "VM-1:if0", 
    "operational_state": "geni_notready", 
    "request_rspec": null, 
    "slice": "9c99e3ebe4774d319b429ca7aea78d7d", 
    "sliver_urn": "urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+interface6ec551f0-92a7-4aae-9676-e983336edfa1", 
    "user_urn": "urn:publicid:IDN+ch.geni.net+user+lnevers", 
    "uuid": "4e5abe07-b9fa-48ad-9198-ae068ed1e0d8", 
    "virtual_machine": "urn:publicid:IDN+bbn-cam-ctrl-1.bbn.com+sliver+vm12e67b21-ef66-4b8f-8e23-302dc3881570", 
    "vlan_tag": null
....
{

It is possible also possible to get information directly from the VMOC controller about the active experiments:

lnevers@bbn-cam-ctrl-1:~$  echo "dump" |nc localhost 7001 
VMOCSwitchControllerMap:
   Switches:
      <vmoc.VMOCSwitchConnection.VMOCSwitchConnection object at 0x2360cd0>
         [VMOCControllerConnection DPID 282037934860467836 URL https://localhost:9000 VLAN 1001]
         [VMOCControllerConnection DPID 282319409837178492 URL https://localhost:9000 VLAN 1002]
   Switches(unindexed):
     <vmoc.VMOCSwitchConnection.VMOCSwitchConnection object at 0x2360cd0>
   Switches (by Controller):
     [VMOCControllerConnection DPID 282319409837178492 URL https://localhost:9000 VLAN 1002]
          <vmoc.VMOCSwitchConnection.VMOCSwitchConnection object at 0x2360cd0>
     [VMOCControllerConnection DPID 282037934860467836 URL https://localhost:9000 VLAN 1001]
          <vmoc.VMOCSwitchConnection.VMOCSwitchConnection object at 0x2360cd0>
   Controllers (by VLAN) 
     1001
          [VMOCControllerConnection DPID 282037934860467836 URL https://localhost:9000 VLAN 1001]
     1002
          [VMOCControllerConnection DPID 282319409837178492 URL https://localhost:9000 VLAN 1002]
     1003
 
VMOC Slice Registry: 
urn:publicid:IDN+ch.geni.net:OpenGENI+slice+b0: {'vlan_configurations': [], 'slice_id': u'urn:publicid:IDN+ch.geni.net:OpenGENI+slice+b0'}
urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln2routable: {'vlan_configurations': [{'vlan': 1001, 'controller_url': 'https://localhost:9000'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln2routable'}
urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-2-exp1: {'vlan_configurations': [{'vlan': 1002, 'controller_url': 'https://localhost:9000'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-2-exp1'}
urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln2server: {'vlan_configurations': [], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln2server'}
https://localhost:9000
    {'vlan_configurations': [{'vlan': 1001, 'controller_url': 'https://localhost:9000'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln2routable'}
   {'vlan_configurations': [{'vlan': 1002, 'controller_url': 'https://localhost:9000'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-2-exp1'}
1001: {'vlan_configurations': [{'vlan': 1001, 'controller_url': 'https://localhost:9000'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-prj+slice+ln2routable'}
1002: {'vlan_configurations': [{'vlan': 1002, 'controller_url': 'https://localhost:9000'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-prj+slice+OG-EXP-2-exp1'}
lnevers@bbn-cam-ctrl-1:~$ 

Step 3. Administrator reviews network data

A site administrator examines the switches and other rack data sources, and determines:

  • What MAC addresses are currently visible on the data plane switch and what experiments do they belong to?
  • For some experiment which was terminated within the past day, what data plane and control MAC and IP addresses did the experiment use?
  • For some experimental data path which is actively sending traffic on the data plane switch, do changes in interface counters show approximately the expected amount of traffic into and out of the switch?

All required information is available to determine VMs used and in use in the snapshots. Also snapshot capture "mac_address", IP addresses and experimenter information for each VM allocated. Information found is sufficient. Switches show counters and mac_addresses in use.

For the PowerConnect 7048 switch:

lnevers@bbn-cam-ctrl-1:~$ ssh admin@10.10.8.201
console>ena
console#show interfaces status 

Port       Name                       Duplex  Speed    Neg   Link   Flow Control
                                                             State  Status
---------  -------------------------  ------  -------  ----  ------ ------------
Gi1/0/1                               N/A     Unknown  Auto  Down   Inactive  
Gi1/0/2                               N/A     Unknown  Auto  Down   Inactive  
Gi1/0/3                               N/A     Unknown  Auto  Down   Inactive  
Gi1/0/4                               N/A     Unknown  Auto  Down   Inactive  
Gi1/0/5                               N/A     Unknown  Auto  Down   Inactive  
Gi1/0/6                               N/A     Unknown  Auto  Down   Inactive  
Gi1/0/7                               N/A     Unknown  Auto  Down   Inactive  
Gi1/0/8                               N/A     Unknown  Auto  Down   Inactive  
Gi1/0/9                               N/A     Unknown  Auto  Down   Inactive  
Gi1/0/10                              N/A     Unknown  Auto  Down   Inactive  
Gi1/0/11                              N/A     Unknown  Auto  Down   Inactive  
Gi1/0/12                              N/A     Unknown  Auto  Down   Inactive  
Gi1/0/13                              Full    1000     Auto  Up     Active    
Gi1/0/14                              Full    1000     Auto  Up     Active    
Gi1/0/15                              Full    1000     Auto  Up     Active    
Gi1/0/16                              Full    1000     Auto  Up     Active    
Gi1/0/17                              Full    1000     Auto  Up     Inactive  
Gi1/0/18                              Full    1000     Auto  Up     Inactive  
Gi1/0/19                              Full    1000     Auto  Up     Inactive  
--More-- or (q)uit
console#

... many lines deleted ...
 
console#show mac address-table 

Aging time is 300 Sec

Vlan     Mac Address           Type        Port                 
-------- --------------------- ----------- ---------------------
1        D067.E5B8.FFFA        Management  Vl1                  
200      0001.E88A.FE7E        Dynamic     Gi1/0/16             
200      90B1.1C0F.F2EB        Dynamic     Gi1/0/13             
200      90B1.1C0F.FCAC        Dynamic     Gi1/0/14             
200      90B1.1C48.41DA        Dynamic     Gi1/0/15             
200      D067.E5B8.FFF9        Dynamic     Gi1/0/17             
300      0000.0C07.AC7E        Dynamic     Gi1/0/48             
300      0000.858A.B566        Dynamic     Gi1/0/48             
300      0003.BA8C.A589        Dynamic     Gi1/0/48             
300      000E.0CB1.72DA        Dynamic     Gi1/0/48             
300      0010.1833.EEED        Dynamic     Gi1/0/48             
300      0011.1150.5146        Dynamic     Gi1/0/48             
300      0015.5D48.1405        Dynamic     Gi1/0/48             
300      0015.5D48.1407        Dynamic     Gi1/0/48             
300      0016.7645.2AA8        Dynamic     Gi1/0/48             
300      0019.B941.5B89        Dynamic     Gi1/0/48             
300      0019.B9D0.5405        Dynamic     Gi1/0/48             

... many lines deleted ...

For the Force10 S4810P OpenFlow switch:

lnevers@bbn-cam-ctrl-1:~$ ssh admin@10.10.8.200
admin@10.10.8.200's password: 
FTOS>ena
Password: 
FTOS#show interfaces 
TenGigabitEthernet 0/0 is up, line protocol is up
Hardware is DellForce10Eth, address is 00:01:e8:8a:fe:7e
    Current address is 00:01:e8:8a:fe:7e
Pluggable media present, SFP+ type is 10GBASE-CU2M
    Medium is MultiRate
Interface index is 33620994
Internet address is not set
MTU 1554 bytes, IP MTU 1500 bytes
LineSpeed 10000 Mbit
Flowcontrol rx on tx off
ARP type: ARPA, ARP Timeout 04:00:00
Last clearing of "show interface" counters 1w3d19h
Queueing strategy: fifo
Input Statistics:
     235820 packets, 51491703 bytes
     118308 64-byte pkts, 504 over 64-byte pkts, 0 over 127-byte pkts
     117008 over 255-byte pkts, 0 over 511-byte pkts, 0 over 1023-byte pkts
     329 Multicasts, 149 Broadcasts
     0 runts, 0 giants, 0 throttles
     0 CRC, 0 overrun, 0 discarded
Output Statistics:
     267348 packets, 50755526 bytes, 0 underruns
     147706 64-byte pkts, 2293 over 64-byte pkts, 35 over 127-byte pkts
FTOS#   

... many lines deleted ...

FTOS#show mac-address-table 

Codes: *N - VLT Peer Synced MAC
VlanId     Mac Address           Type          Interface        State
 1	a0:36:9f:11:0d:60	Dynamic    	Te 0/1    	Active
FTOS#

... many lines deleted ...