Changes between Version 1 and Version 2 of GENIRacksHome/OpenGENIRacks/AcceptanceTestReportApr2014Final


Ignore:
Timestamp:
05/23/14 12:24:46 (10 years ago)
Author:
lnevers@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • GENIRacksHome/OpenGENIRacks/AcceptanceTestReportApr2014Final

    v1 v2  
    1 = GRAM Intermediate Acceptance Test Report - April 2014 =
     1= OpenGENI AM Intermediate Acceptance Test Report - April 2014 =
    22
    3 This pages captures the findings for each of the test cases in the [wiki:GENIRacksHome/AcceptanceTests/GRAMAcceptanceTestsPlan GRAM Acceptance Test Plan]. Status for the current evaluation can be found in the [wiki:GENIRacksHome/GRAMRacks/AcceptanceTestStatusApr2014 GRAM Acceptance Test Status - April 2014] page. As can be seen from the page there has been significant improvement since the last evaluation results captured in the [wiki:GENIRacksHome/GRAMRacks/AcceptanceTestReportDec2013 Acceptance Test Report - December 2013]. Also many tickets reported in earlier evaluation have been successfully addressed. Following is a list of the most critical issues as of 2014-05-19:
     3This pages captures the findings for each of the test cases in the [wiki:GENIRacksHome/AcceptanceTests/OpenGENIAcceptanceTestsPlan GRAM Acceptance Test Plan]. Status for the current evaluation can be found in the [wiki:GENIRacksHome/GRAMRacks/AcceptanceTestStatusApr2014 GRAM Acceptance Test Status - April 2014] page. As can be seen from the page there has been significant improvement since the last evaluation results captured in the [wiki:GENIRacksHome/GRAMRacks/AcceptanceTestReportDec2013 Acceptance Test Report - December 2013]. Also many tickets reported in earlier evaluation have been successfully addressed. Following is a list of the most critical issues as of 2014-05-19:
    44
    5  - ticket 123 - GRAM rack Dell Switches require that ETH_Typer is wild carded on flow mods
    6  - ticket 122 - Poor performance to remote IG with !OpenFlow over stitched connection  
    7  - ticket 121 - Need !OpenFlow Campus Resources connected to Clemson rack      
    8  - ticket 119 - Longer project and slice names causes GRAM internal error for !OpenStack
    9  - ticket 110 - No bandwidth shaping available on GRAM nodes   
    10  - ticket 107 - mismatch in VLAN allocation VS reported in sliver manifest     
    11  - ticket 103 - Tools listinst.sh and vmlist.sh return no information when there are active VMs         
    12  - ticket 94 - Deleting sliver that does not exist at the GRAM aggregate does not return error 
    13  - ticket 93 - Five VM sliver manifest has unexpected format and does not pass rspec lint.      
    14  - ticket 92 - Createsliver does not return after accepting requests, seems to return when request completed.  
     5 - ticket 123 - OpenGENI rack Dell Switches require that ETH_Typer is wild carded on flow mods
     6 - ticket 122 - Poor performance to remote IG with !OpenFlow over stitched connection  
     7 - ticket 121 - Need !OpenFlow Campus Resources connected to Clemson rack      
     8 - ticket 119 - Longer project and slice names causes OpenGENI internal error for !OpenStack
     9 - ticket 110 - No bandwidth shaping available on OpenGENI nodes       
     10 - ticket 107 - mismatch in VLAN allocation VS reported in sliver manifest     
     11 - ticket 103 - Tools listinst.sh and vmlist.sh return no information when there are active VMs         
     12 - ticket 94 - Deleting sliver that does not exist at the OpenGENI aggregate does not return error     
     13 - ticket 93 - Five VM sliver manifest has unexpected format and does not pass rspec lint.      
     14 - ticket 92 - Createsliver does not return after accepting requests, seems to return when request completed.  
    1515
    1616
    1717== Experimenter Acceptance Tests ==
    1818
    19 === GR-EXP-1: Bare Metal Support Acceptance Test ===
     19=== OG-EXP-1: Bare Metal Support Acceptance Test ===
    2020
    2121This test case was deemed "Fail" due to the fact that there is still no supported for bare metal nodes and there is no support for windows. Modified the test case to replace all dedicated nodes (aka bare metal) with VMs to get test coverage.Issues captured in tickets:
    2222
    23  - ticket 51 - Bare metal support feature is missing for GR-EXP-1
    24  - ticket 70 - MS Windows support requires handling of account setup in GRAM
     23 - ticket 51 - Bare metal support feature is missing for OG-EXP-1
     24 - ticket 70 - MS Windows support requires handling of account setup in OpenGENI
    2525
    26 === GR-EXP-2: GRAM Single Site Acceptance Test ===
     26=== OG-EXP-2: OpenGENI Single Site Acceptance Test ===
    2727
    2828This test was deemed "Pass". No issues were found.
    2929
    30 === GR-EXP-3: GRAM Single Site 100 VM Test ===
     30=== OG-EXP-3: OpenGENI Single Site 100 VM Test ===
    3131
    3232This test was deemed "Pass: most criteria".  No problems were found, but only two scenario were tested due to lack of time.
    3333 
    34 === GR-EXP-4: GRAM Multi-site Acceptance Test ===
     34=== OG-EXP-4: OpenGENI Multi-site Acceptance Test ===
    3535
    36  - ticket 110 - No bandwidth shaping available on GRAM nodes
     36 - ticket 110 - No bandwidth shaping available on OpenGENI nodes
    3737
    38 === GR-EXP-5: GRAM Network Resources Acceptance Test ===
     38=== OG-EXP-5: OpenGENI Network Resources Acceptance Test ===
    3939 - ticket 121 - Need !OpenFlow Campus Resources connected to Clemson rack
    4040
    41 === GR-EXP-6: GRAM and Meso-scale Multi-site !OpenFlow Acceptance Test ===
     41=== OG-EXP-6: OpenGENI and Meso-scale Multi-site !OpenFlow Acceptance Test ===
    4242
    4343
    44 === GR-EXP-7: Click Router Experiment Acceptance Test ===
     44=== OG-EXP-7: Click Router Experiment Acceptance Test ===
    4545
    4646This layer two test was not attempted during this test effort.
    … …  
    4848== Administration Acceptance Tests ==
    4949
    50 === GR-ADM-1: Rack Receipt and Inventory Test ===
     50=== OG-ADM-1: Rack Receipt and Inventory Test ===
    5151
    5252This test was deemed "Completed" and needing further testing because there was no rack delivery process and there is no GMOC monitoring:
    5353
    54  - ticket 49 - There is no GMOC data collection or GRAM monitoring data reporting
     54 - ticket 49 - There is no GMOC data collection or OpenGENI monitoring data reporting
    5555
    56 === GR-ADM-2: Rack Administrator Access Test ===
     56=== OG-ADM-2: Rack Administrator Access Test ===
    5757
    5858This test was deemed "Pass: most criteria" because the site Administrator account instructions are incomplete and because remote access not fully validated.
    … …  
    6161
    6262
    63 === GR-ADM-3: Full Rack Reboot Test ===
     63=== OG-ADM-3: Full Rack Reboot Test ===
    6464
    6565This test was run successfully and using existing documentation was able to complete a full reboot and have the aggregate running and available upon test completion.
    6666
    6767
    68 === GR-ADM-4: Emergency Stop Test ===
     68=== OG-ADM-4: Emergency Stop Test ===
    6969
    70 This test was not run because it requires GMOC interactions and monitoring, neither of which are available at this time for the GRAM Racks at Clemson.
     70This test was not run because it requires GMOC interactions and monitoring, neither of which are available at this time for the OpenGENI Racks at Clemson.
    7171 
    7272
    73 === GR-ADM-5: Software Update Test ===
     73=== OG-ADM-5: Software Update Test ===
    7474
    75 This test was run without any problem, GRAM software packages was installed as part of an upgrade.
     75This test was run without any problem, OpenGENI software packages was installed as part of an upgrade.
    7676
    77 === GR-ADM-6: Control Network Disconnection Test ===
     77=== OG-ADM-6: Control Network Disconnection Test ===
    7878
    79 This test was not run because it requires GMOC interactions and monitoring, neither of which are available at this time for the GRAM Racks at Clemson.
     79This test was not run because it requires GMOC interactions and monitoring, neither of which are available at this time for the OpenGENI Racks at Clemson.
    8080 
    81 === GR-ADM-7: Documentation Review Test ===
     81=== OG-ADM-7: Documentation Review Test ===
    8282
    8383Significant improvement in documentation.  This documentation test case checks for the existence of 14 separate documents. Each are listed below  with the overall status and associated ticket.
    8484
    85   1. GRAM Rack Installation Documentation  - Pass: most criteria - ticket 74       
    86   2. GRAM Rack Part List Documentation  - Pass: most criteria - ticket 75               
    87   3. GRAM Software Components Identification Documentation - Pass               
    88   4. GRAM and OS Upgrade Plans  - Pass             
    89   5. GRAM Software Components Functional Description and Access to Source code - Pass             
    90   6. GRAM Rack Resources Usage and Policy Documentation - In Progress - ticket 79             
    91   7. GRAM Rack Resources Administration - Pass: most criteria - ticket 80           
    92   8. GRAM Rack Administrative procedure for user accounts - Pass           
    93   9. GRAM Rack Administrative procedure for rack operations - Pass             
    94   10. GRAM Rack Administrative procedure for rack scheduled shutdown - Pass           
    95   11. GRAM Rack Administrative procedure for Emergency Stop - In Progress - ticket 84             
    96   12. GRAM Rack Administrative procedure for health status - Fail - ticket 85         
    97   13. GRAM Rack Site Operations documentation  - Fail - ticket 86     
    98   14. Site Specific GRAM Deployment Data  - Pass           
     85  1. OpenGENI Rack Installation Documentation  - Pass: most criteria - ticket 74       
     86  2. OpenGENI Rack Part List Documentation  - Pass: most criteria - ticket 75               
     87  3. OpenGENI Software Components Identification Documentation - Pass               
     88  4. OpenGENI and OS Upgrade Plans  - Pass             
     89  5. OpenGENI Software Components Functional Description and Access to Source code - Pass             
     90  6. OpenGENI Rack Resources Usage and Policy Documentation - In Progress - ticket 79             
     91  7. OpenGENI Rack Resources Administration - Pass: most criteria - ticket 80           
     92  8. OpenGENI Rack Administrative procedure for user accounts - Pass           
     93  9. OpenGENI Rack Administrative procedure for rack operations - Pass             
     94  10. OpenGENI Rack Administrative procedure for rack scheduled shutdown - Pass           
     95  11. OpenGENI Rack Administrative procedure for Emergency Stop - In Progress - ticket 84             
     96  12. OpenGENI Rack Administrative procedure for health status - Fail - ticket 85         
     97  13. OpenGENI Rack Site Operations documentation  - Fail - ticket 86     
     98  14. Site Specific OpenGENI Deployment Data  - Pass           
    9999
    100100== Monitoring Acceptance Tests ==
    101101
    102 === GR-MON-1: Control Network Software and VLAN Inspection Test ===
     102=== OG-MON-1: Control Network Software and VLAN Inspection Test ===
    103103
    104 This test was deemed "Pass: met most criteria". All steps were successful, except for GRAM git access is BBN internal only.
     104This test was deemed "Pass: met most criteria". All steps were successful, except for OpenGENI git access is BBN internal only.
    105105
    106  - ticket 66 - GRAM git access is BBN internal only, external access will be needed after initial evaluation
     106 - ticket 66 - OpenGENI git access is BBN internal only, external access will be needed after initial evaluation
    107107
    108 === GR-MON-2: GENI Software Configuration Inspection Test ===
     108=== OG-MON-2: GENI Software Configuration Inspection Test ===
    109109
    110110This test was deemed "Pass". No issue were found.
    111111
    112 === GR-MON-3: GENI Active Experiment Inspection Test ===
     112=== OG-MON-3: GENI Active Experiment Inspection Test ===
    113113
    114114This test was deemed "Pass". No issue were found.
    115115
    116 === GR-MON-4: Infrastructure Device Performance Test ===
     116=== OG-MON-4: Infrastructure Device Performance Test ===
    117117
    118118This test  was deemed "Blocked" because it cannot be executed without monitoring data collection at the GMOC. Ticket tracking missing GMOC support:
    119119
    120  - ticket 49 - There is no GMOC data collection or GRAM monitoring data reporting
     120 - ticket 49 - There is no GMOC data collection or OpenGENI monitoring data reporting
    121121
    122 === GR-MON-5: GMOC Data Collection Test ===
     122=== OG-MON-5: GMOC Data Collection Test ===
    123123
    124124This test  was deemed "Blocked" because it cannot be executed without monitoring data collection at the GMOC. Ticket tracking missing GMOC support:
    125125
    126  - ticket 49 - There is no GMOC data collection or GRAM monitoring data reporting
     126 - ticket 49 - There is no GMOC data collection or OpenGENI monitoring data reporting