wiki:GENIRacksHome/OpenGENIRacks/AcceptanceTestStatusApr2014/OG-MON-3

Version 1 (modified by lnevers@bbn.com, 5 years ago) (diff)

--

GR-MON-3: GENI Active Experiment Inspection Test

This page captures status for the test case GR-MON-3. For additional information see the Acceptance Test Status - April 2014? page overall status, or the GRAM Acceptance Test Plan? for details about the planned evaluation.

Step State Notes Tickets
Step 1 Pass
Step 2 Pass
Step 3 Pass


State Legend Description
Pass Test completed and met all criteria
Pass: most criteria Test completed and met most criteria. Exceptions documented
Fail Test completed and failed to meet criteria.
Complete Test completed but will require re-execution due to expected changes
Blocked Blocked by ticketed issue(s).
In Progress Currently under test.
Not Planned This area is not part of initial evaluation

Test Plan Steps

Step 1. Set up experiments

An experimenter starts up experiments to ensure there is data to look at:

  • An experimenter runs an experiment containing at least one rack OpenStack VM, and terminates it.
  • An experimenter runs an experiment containing at least one rack OpenStack VM, and leaves it running.

One experiment was set up named "stitchOF" which included 1VM at Clemson using a stitched path to GPO IG and running OpenFlow.

gram@clemson-clemson-control-1:~$ nova list --all-tenants
+--------------------------------------+---------+--------+------------------------------------------------+
| ID                                   | Name    | Status | Networks                                       |
+--------------------------------------+---------+--------+------------------------------------------------+
| e4b5ce2c-ea78-4e39-acbe-1157af2e458d | gr-clem | ACTIVE | linkOF=192.168.1.2; GRAM-mgmt-net=192.168.10.3 |
+--------------------------------------+---------+--------+------------------------------------------------+
gram@clemson-clemson-control-1:~$ keystone tenant-list
+----------------------------------+------------------------------------+---------+
|                id                |                name                | enabled |
+----------------------------------+------------------------------------+---------+
| 89a03e397f8b4743be8d6c0c470e9ede |               admin                |   True  |
| 65c073cb3f83446eb400d76013f71bc0 | ch.geni.net:ln-test+slice+stitchOF |   True  |
| 32b79311d95b4483a6d40a6db316fa3f |              service               |   True  |
+----------------------------------+------------------------------------+---------+

Step 2. Administrator reviews active and historical experiment data

A site administrator uses available system and experiment data sources to determine current experimental state, including:

  • How many VMs are running and which experimenters own them
  • How many physical hosts are in use by experiments, and which experimenters own them
  • How many VMs were terminated within the past day, and which experimenters owned them
  • What OpenFlow controllers the data plane switch and the rack VMOC are communicating with

On the Control Node in the directory /etc/gram/snapshots/gram, review the snapshot for the time frame when the experiment was run and verify an entry for the sliver created at the beginning of this test.

It is possible also possible to get information directly from the VMOC controller about the active experiments:

gram@clemson-clemson-control-1:~$ echo "dump" |nc localhost 7001 
VMOCSwitchControllerMap:
   Switches:
      <vmoc.VMOCSwitchConnection.VMOCSwitchConnection object at 0x2e7bfd0>
   Switches(unindexed):
     <vmoc.VMOCSwitchConnection.VMOCSwitchConnection object at 0x2e7bfd0>
   Switches (by Controller):
   Controllers (by VLAN) 
     3006
      3007
 
VMOC Slice Registry: 
urn:publicid:IDN+ch.geni.net:ln-test+slice+stitchOF: {'vlan_configurations': [{'vlan': 3007, 'controller_url': u'tcp:192.1.249.185:33020'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-test+slice+stitchOF'}
urn:publicid:IDN+ch.geni.net:ln-test+slice+gr-clem-eg-gpo: {'vlan_configurations': [], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-test+slice+gr-clem-eg-gpo'}
tcp:192.1.249.185:33020
    {'vlan_configurations': [{'vlan': 3007, 'controller_url': u'tcp:192.1.249.185:33020'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-test+slice+stitchOF'}
tcp:192.1.249.185:33019
 3007: {'vlan_configurations': [{'vlan': 3007, 'controller_url': u'tcp:192.1.249.185:33020'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-test+slice+stitchOF'}
gram@clemson-clemson-control-1:~$ 

Step 3. Administrator reviews network data

A site administrator examines the switches and other rack data sources, and determines:

  • What MAC addresses are currently visible on the data plane switch and what experiments do they belong to?
  • For some experiment which was terminated within the past day, what data plane and control MAC and IP addresses did the experiment use?
  • For some experimental data path which is actively sending traffic on the data plane switch, do changes in interface counters show approximately the expected amount of traffic into and out of the switch?

All required information is available to determine VMs used and in use in the snapshots. Also snapshot capture "mac_address", IP addresses and experimenter information for each VM allocated. Information found is sufficient.

One example experiment GR-EXP-2-exp1 has two nodes running, login to the first node and get MAC:

$ ssh -p 3000  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@130.127.88.101
...
lnevers@vm-1:~$ /sbin/ifconfig
...
eth1      Link encap:Ethernet  HWaddr fa:16:3e:ce:9d:16  
          inet addr:192.168.1.1  Bcast:192.168.255.255  Mask:255.255.0.0 
...

Second node in experiment:

$ ssh -p 3001  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@130.127.88.101
...
lnevers@gr-clem:~$ /sbin/ifconfig 
...
eth1      Link encap:Ethernet  HWaddr fa:16:3e:f0:67:4b  
          inet addr:192.168.1.2  Bcast:192.168.255.255  Mask:255.255.0.0
...

Switches information shows that counters and MAC addresses for those nodes in use. For the PowerConnect 7048 switch:

gram@clemson-clemson-control-1:~$ ssh lnevers@10.10.8.201
lnevers@10.10.8.201's password: 

PCT7048>ena
Password:***************

PCT7048#show  mac address-table 

Aging time is 300 Sec

Vlan     Mac Address           Type        Port
-------- --------------------- ----------- ---------------------
1        D067.E5A9.B19F        Management  Vl1
200      0001.E8D7.589D        Dynamic     Gi1/0/41
200      90B1.1C0F.8A79        Dynamic     Gi1/0/14
200      90B1.1C0F.D649        Dynamic     Gi1/0/13
200      90B1.1C0F.F479        Dynamic     Gi1/0/15
200      D067.E5A9.B19F        Management  Vl200
845      0007.7D26.29AA        Dynamic     Gi1/0/48
845      0017.DF2B.0800        Dynamic     Gi1/0/48
845      001A.A02F.AD0B        Dynamic     Gi1/0/48
845      0026.B975.1D45        Dynamic     Gi1/0/48
845      0026.B983.F9A4        Dynamic     Gi1/0/48
845      0026.F13B.8A00        Dynamic     Gi1/0/48
845      1803.73AD.5CA9        Dynamic     Gi1/0/48
845      5254.0043.1FF4        Dynamic     Gi1/0/48
845      782B.CB8F.740E        Dynamic     Gi1/0/48
845      782B.CB8F.E7C4        Dynamic     Gi1/0/48
845      842B.2B19.E4D9        Dynamic     Gi1/0/48
845      842B.2B19.E4DB        Dynamic     Gi1/0/48
845      D067.E5A9.B19F        Management  Vl845
845      D4BE.D98A.3A47        Dynamic     Gi1/0/48
872      0007.7D26.29AA        Dynamic     Gi1/0/48
872      0023.5EA9.5CCA        Dynamic     Gi1/0/48
872      A036.9F11.0AB2        Dynamic     Gi1/0/25
872      A036.9F11.0C4E        Dynamic     Gi1/0/27
872      A036.9F11.0D32        Dynamic     Gi1/0/26
872      C81F.66B7.9C64        Dynamic     Gi1/0/48
872      C81F.66B7.9C66        Dynamic     Gi1/0/48
872      D067.E5A9.B19F        Management  Vl872
872      FA16.3EED.2D4C        Dynamic     Gi1/0/25
2500     FA16.3E1F.532C        Dynamic     Gi1/0/2
2500     FA16.3E5C.923D        Dynamic     Gi1/0/1
2500     FA16.3E8F.1AB7        Dynamic     Gi1/0/2
2500     FA16.3E92.5863        Dynamic     Gi1/0/1

Total MAC Addresses in use: 33
PCT7048#show arp         

Static ARP entries are only active
when the IP address is reachable on a local subnet


Age Time (seconds)............................. 1200
Response Time (seconds)........................ 1
Retries........................................ 4
Cache Size..................................... 6144
Dynamic Renew Mode ............................ Disable
Total Entry Count Current / Peak .............. 5 / 5
Static Entry Count Configured / Active / Max .. 0 / 0 / 128

IP Address       MAC Address        Interface       Type      Age
---------------  -----------------  --------------  --------  -----------
10.10.8.100      90B1.1C0F.D649     Vl200           Dynamic    0h  6m 15s
10.10.8.201      D067.E5A9.B19F     Vl200           Local         n/a
130.127.39.129   0017.DF2B.0800     Vl845           Dynamic    0h  0m 55s
130.127.39.136   D067.E5A9.B19F     Vl845           Local         n/a
130.127.39.245   001A.A02F.AD0B     Vl845           Dynamic    0h  0m  4s

For the Force10 S4810P OpenFlow switch: -- Waiting on login setup --

gram@clemson-clemson-control-1:~$ ssh lnevers@10.10.8.200
lnevers@10.10.8.200's password: 
FTOS>ena
Password: 
FTOS#
FTOS#show mac-address-table 

Codes: *N - VLT Peer Synced MAC
VlanId     Mac Address           Type          Interface        State
 1	00:24:38:9c:13:00	Dynamic    	Te 0/3    	Active
 1	08:9e:01:18:01:47	Dynamic    	Te 0/3    	Active
FTOS# 
FTOS#show mac-address-table 

Codes: *N - VLT Peer Synced MAC
VlanId     Mac Address           Type          Interface        State
 1	00:24:38:9c:13:00	Dynamic    	Te 0/3    	Active
 1	08:9e:01:18:01:47	Dynamic    	Te 0/3    	Active
FTOS#show arp

Protocol    Address         Age(min)  Hardware Address    Interface  VLAN             CPU
-----------------------------------------------------------------------------------------
Internet    10.10.8.100           9   90:b1:1c:0f:d6:49   Ma 0/0      -               CP
FTOS#
FTOS#show vlan id 1001  <<<=== experiment VLAN

Codes: * - Default VLAN, G - GVRP VLANs, R - Remote Port Mirroring VLANs, P - Primary, C - Community, I - Isolated
       O - Openflow
Q: U - Untagged, T - Tagged
   x - Dot1x untagged, X - Dot1x tagged
   o - OpenFlow untagged, O - OpenFlow tagged
   G - GVRP tagged, M - Vlan-stack, H - VSN tagged
   i - Internal untagged, I - Internal tagged, v - VLT untagged, V - VLT tagged

    NUM    Status    Description                     Q Ports
O   1001   Active                                    T Te 0/0-2



FTOS#show interfaces vlan 1001 description 
Interface                OK  Status     Protocol    Description
Vlan 1001                YES up         up          

FTOS#show interfaces vlan 1001    
Vlan 1001 is up, line protocol is up
Address is 00:01:e8:d7:58:9d, Current address is 00:01:e8:d7:58:9d
Interface index is 1107526633
of-instance: 1
Internet address is not set
MTU 1554 bytes, IP MTU 1500 bytes
LineSpeed 10000 Mbit
ARP type: ARPA, ARP Timeout 04:00:00
Last clearing of "show interface" counters 6w6d1h
Queueing strategy: fifo
Time since last interface status change: 6w6d1h
FTOS#

The way to map an experiment to a slice is to look for the MAC address in the snapshot file. For this particular experiment the file /etc/gram/snapshots/gram/2014_04_23_13_18_13_0.json was reviewed and was able to map both MAC addresses back to the slice. There is a tool to dump the information in the snapshots to an easy to read version, but it is currently not working. If it was working, one would simply issue the following command on the control node:

$python /etc/gram/dump_gram_snapshot.py --directory ./output/ --snapshot /etc/gram/snapshots/gram/2014_04_23_09_59_49_0.json