Changes between Initial Version and Version 1 of GENIRacksHome/OpenGENIRacks/AcceptanceTestStatusApr2014/OG-MON-3


Ignore:
Timestamp:
05/23/14 11:34:05 (10 years ago)
Author:
lnevers@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • GENIRacksHome/OpenGENIRacks/AcceptanceTestStatusApr2014/OG-MON-3

    v1 v1  
     1= GR-MON-3: GENI Active Experiment Inspection Test =
     2
     3This page captures status for the test case GR-MON-3. For additional information see the [wiki:GENIRacksHome/GRAMRacks/AcceptanceTestStatusApr2014 Acceptance Test Status - April 2014] page overall status, or the [wiki:GENIRacksHome/AcceptanceTests/GRAMAcceptanceTestsPlan GRAM Acceptance Test Plan] for details about the planned evaluation.
     4
     5
     6
     7
     8|| '''Step''' || '''State'''||''' Notes '''            || '''Tickets'''  ||
     9|| Step 1     ||[[span(Pass, style=background-color: green )]]  ||                         ||                ||
     10|| Step 2     ||[[span(Pass, style=background-color: green )]] ||                         ||                ||
     11|| Step 3     ||[[span(Pass, style=background-color: green )]] ||                         ||                ||
     12             
     13
     14[[BR]]
     15|| '''State Legend'''                                   || '''Description'''                                                    ||
     16||[[span(Pass, style=background-color: green )]]        || Test completed and met all criteria                                  ||
     17||[[span(Pass: most criteria, style=background-color: #98FB98)]]|| Test completed and met most criteria. Exceptions documented  ||
     18||[[span(Fail, style=background-color: red)]]           || Test completed and failed to meet criteria.                          ||
     19||[[span(Complete, style=background-color: yellow)]]    || Test completed but will require re-execution due to expected changes ||
     20||[[span(Blocked, style=background-color: orange)]]     || Blocked by ticketed issue(s).                                        ||
     21||[[span(In Progress, style=background-color: #63B8FF)]]|| Currently under test.                                                ||
     22||[[span(Not Planned)]]                                 || This area is not part of initial evaluation                          ||
     23
     24
     25= Test Plan Steps =
     26
     27== Step 1. Set up experiments ==
     28
     29
     30An experimenter starts up experiments to ensure there is data to look at:
     31   * An experimenter runs an experiment containing at least one rack !OpenStack VM, and terminates it.
     32   * An experimenter runs an experiment containing at least one rack !OpenStack VM, and leaves it running.
     33
     34One experiment was set up named "stitchOF" which included 1VM at Clemson using a stitched path to GPO IG and running !OpenFlow.
     35{{{
     36gram@clemson-clemson-control-1:~$ nova list --all-tenants
     37+--------------------------------------+---------+--------+------------------------------------------------+
     38| ID                                   | Name    | Status | Networks                                       |
     39+--------------------------------------+---------+--------+------------------------------------------------+
     40| e4b5ce2c-ea78-4e39-acbe-1157af2e458d | gr-clem | ACTIVE | linkOF=192.168.1.2; GRAM-mgmt-net=192.168.10.3 |
     41+--------------------------------------+---------+--------+------------------------------------------------+
     42gram@clemson-clemson-control-1:~$ keystone tenant-list
     43+----------------------------------+------------------------------------+---------+
     44|                id                |                name                | enabled |
     45+----------------------------------+------------------------------------+---------+
     46| 89a03e397f8b4743be8d6c0c470e9ede |               admin                |   True  |
     47| 65c073cb3f83446eb400d76013f71bc0 | ch.geni.net:ln-test+slice+stitchOF |   True  |
     48| 32b79311d95b4483a6d40a6db316fa3f |              service               |   True  |
     49+----------------------------------+------------------------------------+---------+
     50}}}
     51
     52== Step 2. Administrator reviews active and historical experiment data ==
     53
     54A site administrator uses available system and experiment data sources to determine current experimental state, including:
     55   * How many VMs are running and which experimenters own them
     56   * How many physical hosts are in use by experiments, and which experimenters own them
     57   * How many VMs were terminated within the past day, and which experimenters owned them
     58   * What !OpenFlow controllers the data plane switch and the rack VMOC are communicating with
     59
     60On the Control Node in the directory /etc/gram/snapshots/gram, review the snapshot for the time frame when the experiment was run and verify an entry for the sliver created at the beginning of this test.
     61
     62It is possible also possible to get information directly from the VMOC controller about the active experiments:
     63{{{
     64gram@clemson-clemson-control-1:~$ echo "dump" |nc localhost 7001
     65VMOCSwitchControllerMap:
     66   Switches:
     67      <vmoc.VMOCSwitchConnection.VMOCSwitchConnection object at 0x2e7bfd0>
     68   Switches(unindexed):
     69     <vmoc.VMOCSwitchConnection.VMOCSwitchConnection object at 0x2e7bfd0>
     70   Switches (by Controller):
     71   Controllers (by VLAN)
     72     3006
     73      3007
     74 
     75VMOC Slice Registry:
     76urn:publicid:IDN+ch.geni.net:ln-test+slice+stitchOF: {'vlan_configurations': [{'vlan': 3007, 'controller_url': u'tcp:192.1.249.185:33020'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-test+slice+stitchOF'}
     77urn:publicid:IDN+ch.geni.net:ln-test+slice+gr-clem-eg-gpo: {'vlan_configurations': [], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-test+slice+gr-clem-eg-gpo'}
     78tcp:192.1.249.185:33020
     79    {'vlan_configurations': [{'vlan': 3007, 'controller_url': u'tcp:192.1.249.185:33020'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-test+slice+stitchOF'}
     80tcp:192.1.249.185:33019
     81 3007: {'vlan_configurations': [{'vlan': 3007, 'controller_url': u'tcp:192.1.249.185:33020'}], 'slice_id': u'urn:publicid:IDN+ch.geni.net:ln-test+slice+stitchOF'}
     82gram@clemson-clemson-control-1:~$
     83}}}
     84
     85== Step 3. Administrator reviews network data ==
     86
     87A site administrator examines the switches and other rack data sources, and determines:
     88   * What MAC addresses are currently visible on the data plane switch and what experiments do they belong to?
     89   * For some experiment which was terminated within the past day, what data plane and control MAC and IP addresses did the experiment use?
     90   * For some experimental data path which is actively sending traffic on the data plane switch, do changes in interface counters show approximately the expected amount of traffic into and out of the switch?
     91
     92All required information is available to determine VMs used and in use in the snapshots.  Also snapshot capture "mac_address", IP addresses and experimenter information for each VM allocated. Information found is sufficient. 
     93
     94One example experiment GR-EXP-2-exp1 has two nodes running, login to the first node and get MAC:
     95{{{
     96$ ssh -p 3000  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@130.127.88.101
     97...
     98lnevers@vm-1:~$ /sbin/ifconfig
     99...
     100eth1      Link encap:Ethernet  HWaddr fa:16:3e:ce:9d:16 
     101          inet addr:192.168.1.1  Bcast:192.168.255.255  Mask:255.255.0.0
     102...
     103}}}
     104Second node in experiment:
     105{{{
     106$ ssh -p 3001  -i /home/lnevers/.ssh/geni_cert_portal_key lnevers@130.127.88.101
     107...
     108lnevers@gr-clem:~$ /sbin/ifconfig
     109...
     110eth1      Link encap:Ethernet  HWaddr fa:16:3e:f0:67:4b 
     111          inet addr:192.168.1.2  Bcast:192.168.255.255  Mask:255.255.0.0
     112...
     113}}}
     114
     115Switches information shows that counters and MAC addresses for those nodes in use.
     116For the !PowerConnect 7048 switch:
     117{{{
     118gram@clemson-clemson-control-1:~$ ssh lnevers@10.10.8.201
     119lnevers@10.10.8.201's password:
     120
     121PCT7048>ena
     122Password:***************
     123
     124PCT7048#show  mac address-table
     125
     126Aging time is 300 Sec
     127
     128Vlan     Mac Address           Type        Port
     129-------- --------------------- ----------- ---------------------
     1301        D067.E5A9.B19F        Management  Vl1
     131200      0001.E8D7.589D        Dynamic     Gi1/0/41
     132200      90B1.1C0F.8A79        Dynamic     Gi1/0/14
     133200      90B1.1C0F.D649        Dynamic     Gi1/0/13
     134200      90B1.1C0F.F479        Dynamic     Gi1/0/15
     135200      D067.E5A9.B19F        Management  Vl200
     136845      0007.7D26.29AA        Dynamic     Gi1/0/48
     137845      0017.DF2B.0800        Dynamic     Gi1/0/48
     138845      001A.A02F.AD0B        Dynamic     Gi1/0/48
     139845      0026.B975.1D45        Dynamic     Gi1/0/48
     140845      0026.B983.F9A4        Dynamic     Gi1/0/48
     141845      0026.F13B.8A00        Dynamic     Gi1/0/48
     142845      1803.73AD.5CA9        Dynamic     Gi1/0/48
     143845      5254.0043.1FF4        Dynamic     Gi1/0/48
     144845      782B.CB8F.740E        Dynamic     Gi1/0/48
     145845      782B.CB8F.E7C4        Dynamic     Gi1/0/48
     146845      842B.2B19.E4D9        Dynamic     Gi1/0/48
     147845      842B.2B19.E4DB        Dynamic     Gi1/0/48
     148845      D067.E5A9.B19F        Management  Vl845
     149845      D4BE.D98A.3A47        Dynamic     Gi1/0/48
     150872      0007.7D26.29AA        Dynamic     Gi1/0/48
     151872      0023.5EA9.5CCA        Dynamic     Gi1/0/48
     152872      A036.9F11.0AB2        Dynamic     Gi1/0/25
     153872      A036.9F11.0C4E        Dynamic     Gi1/0/27
     154872      A036.9F11.0D32        Dynamic     Gi1/0/26
     155872      C81F.66B7.9C64        Dynamic     Gi1/0/48
     156872      C81F.66B7.9C66        Dynamic     Gi1/0/48
     157872      D067.E5A9.B19F        Management  Vl872
     158872      FA16.3EED.2D4C        Dynamic     Gi1/0/25
     1592500     FA16.3E1F.532C        Dynamic     Gi1/0/2
     1602500     FA16.3E5C.923D        Dynamic     Gi1/0/1
     1612500     FA16.3E8F.1AB7        Dynamic     Gi1/0/2
     1622500     FA16.3E92.5863        Dynamic     Gi1/0/1
     163
     164Total MAC Addresses in use: 33
     165PCT7048#show arp         
     166
     167Static ARP entries are only active
     168when the IP address is reachable on a local subnet
     169
     170
     171Age Time (seconds)............................. 1200
     172Response Time (seconds)........................ 1
     173Retries........................................ 4
     174Cache Size..................................... 6144
     175Dynamic Renew Mode ............................ Disable
     176Total Entry Count Current / Peak .............. 5 / 5
     177Static Entry Count Configured / Active / Max .. 0 / 0 / 128
     178
     179IP Address       MAC Address        Interface       Type      Age
     180---------------  -----------------  --------------  --------  -----------
     18110.10.8.100      90B1.1C0F.D649     Vl200           Dynamic    0h  6m 15s
     18210.10.8.201      D067.E5A9.B19F     Vl200           Local         n/a
     183130.127.39.129   0017.DF2B.0800     Vl845           Dynamic    0h  0m 55s
     184130.127.39.136   D067.E5A9.B19F     Vl845           Local         n/a
     185130.127.39.245   001A.A02F.AD0B     Vl845           Dynamic    0h  0m  4s
     186}}}
     187
     188
     189For the Force10 S4810P !OpenFlow switch: -- Waiting on login setup --
     190{{{
     191gram@clemson-clemson-control-1:~$ ssh lnevers@10.10.8.200
     192lnevers@10.10.8.200's password:
     193FTOS>ena
     194Password:
     195FTOS#
     196FTOS#show mac-address-table
     197
     198Codes: *N - VLT Peer Synced MAC
     199VlanId     Mac Address           Type          Interface        State
     200 1      00:24:38:9c:13:00       Dynamic         Te 0/3          Active
     201 1      08:9e:01:18:01:47       Dynamic         Te 0/3          Active
     202FTOS#
     203FTOS#show mac-address-table
     204
     205Codes: *N - VLT Peer Synced MAC
     206VlanId     Mac Address           Type          Interface        State
     207 1      00:24:38:9c:13:00       Dynamic         Te 0/3          Active
     208 1      08:9e:01:18:01:47       Dynamic         Te 0/3          Active
     209FTOS#show arp
     210
     211Protocol    Address         Age(min)  Hardware Address    Interface  VLAN             CPU
     212-----------------------------------------------------------------------------------------
     213Internet    10.10.8.100           9   90:b1:1c:0f:d6:49   Ma 0/0      -               CP
     214FTOS#
     215FTOS#show vlan id 1001  <<<=== experiment VLAN
     216
     217Codes: * - Default VLAN, G - GVRP VLANs, R - Remote Port Mirroring VLANs, P - Primary, C - Community, I - Isolated
     218       O - Openflow
     219Q: U - Untagged, T - Tagged
     220   x - Dot1x untagged, X - Dot1x tagged
     221   o - OpenFlow untagged, O - OpenFlow tagged
     222   G - GVRP tagged, M - Vlan-stack, H - VSN tagged
     223   i - Internal untagged, I - Internal tagged, v - VLT untagged, V - VLT tagged
     224
     225    NUM    Status    Description                     Q Ports
     226O   1001   Active                                    T Te 0/0-2
     227
     228
     229
     230FTOS#show interfaces vlan 1001 description
     231Interface                OK  Status     Protocol    Description
     232Vlan 1001                YES up         up         
     233
     234FTOS#show interfaces vlan 1001   
     235Vlan 1001 is up, line protocol is up
     236Address is 00:01:e8:d7:58:9d, Current address is 00:01:e8:d7:58:9d
     237Interface index is 1107526633
     238of-instance: 1
     239Internet address is not set
     240MTU 1554 bytes, IP MTU 1500 bytes
     241LineSpeed 10000 Mbit
     242ARP type: ARPA, ARP Timeout 04:00:00
     243Last clearing of "show interface" counters 6w6d1h
     244Queueing strategy: fifo
     245Time since last interface status change: 6w6d1h
     246FTOS#
     247}}}
     248
     249The way to map an experiment to a slice is to look for the MAC address in the snapshot file.  For this particular experiment the file /etc/gram/snapshots/gram/2014_04_23_13_18_13_0.json  was reviewed and was able to map both MAC addresses back to the slice.  There is a tool to dump the information in the snapshots to an easy to read version, but it is currently not working.  If it was working, one would simply issue the following command on the control node:
     250{{{
     251$python /etc/gram/dump_gram_snapshot.py --directory ./output/ --snapshot /etc/gram/snapshots/gram/2014_04_23_09_59_49_0.json
     252}}}