Changes between Version 1 and Version 2 of GENIRacksHome/OpenGENIRacks/AcceptanceTestReportDec2013


Ignore:
Timestamp:
05/23/14 12:26:19 (5 years ago)
Author:
lnevers@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • GENIRacksHome/OpenGENIRacks/AcceptanceTestReportDec2013

    v1 v2  
    1 = GRAM Intermediate Acceptance Test Report - December 2013 =
     1= OpenGENI Intermediate Acceptance Test Report - December 2013 =
    22
    3 This pages captures the findings for each of the test cases in the [wiki:GENIRacksHome/AcceptanceTests/GRAMAcceptanceTestsPlan GRAM Acceptance Test Plan]. Status for the current evaluation can be found in the [wiki:GENIRacksHome/GRAMRacks/AcceptanceTestStatusDec2013 GRAM Acceptance Test Status - December 2013] page. As can be seen from the page there has been significant improvement since the last evaluation results captured in the [wiki:GENIRacksHome/GRAMRacks/AcceptanceTestReportMay2013 Acceptance Test Report - May 2013] (May 2013). Also many tickets reported in earlier evaluation have been successfully addressed. Following is a list of the most critical issues as of 2013-12-18:
     3This pages captures the findings for each of the test cases in the [wiki:GENIRacksHome/AcceptanceTests/OpenGENIAcceptanceTestsPlan GRAM Acceptance Test Plan]. Status for the current evaluation can be found in the [wiki:GENIRacksHome/GRAMRacks/AcceptanceTestStatusDec2013 GRAM Acceptance Test Status - December 2013] page. As can be seen from the page there has been significant improvement since the last evaluation results captured in the [wiki:GENIRacksHome/GRAMRacks/AcceptanceTestReportMay2013 Acceptance Test Report - May 2013] (May 2013). Also many tickets reported in earlier evaluation have been successfully addressed. Following is a list of the most critical issues as of 2013-12-18:
    44
    55 - ticket 100 There is no connectivity between racks
    6  - ticket 49 There is no GMOC data collection or GRAM monitoring data reporting for GR-MON-4/5 
     6 - ticket 49 There is no GMOC data collection or OpenGENI monitoring data reporting for OG-MON-4/5 
    77 - ticket 51 Bare metal support feature is missing
    8  - ticket 99    GRAM Software packaging
     8 - ticket 99    OpenGENI Software packaging
    99 - ticket 65 Remote console not available via web interface, some iDRAC access via SSH.
    10  - ticket 66 GRAM git access is BBN internal only, external access will be needed after initial eval
     10 - ticket 66 OpenGENI git access is BBN internal only, external access will be needed after initial eval
    1111
    1212
    1313== Experimenter Acceptance Tests ==
    1414
    15 === GR-EXP-1: Bare Metal Support Acceptance Test ===
     15=== OG-EXP-1: Bare Metal Support Acceptance Test ===
    1616
    1717This test case was deemed "Fail" due to the fact that there is still no supported for bare metal nodes and there is no support for windows. Modified the test case to replace all dedicated nodes (aka bare metal) with VMs to get test coverage.Issues captured in tickets:
    1818
    19  - ticket 51 Bare metal support feature is missing for GR-EXP-1
    20  - ticket 70 MS Windows support requires handling of account setup in GRAM
     19 - ticket 51 Bare metal support feature is missing for OG-EXP-1
     20 - ticket 70 MS Windows support requires handling of account setup in OpenGENI
    2121
    22 === GR-EXP-2: GRAM Single Site Acceptance Test ===
     22=== OG-EXP-2: OpenGENI Single Site Acceptance Test ===
    2323
    2424This test was deemed "Pass". No issues were found.
    2525
    26 === GR-EXP-3: GRAM Single Site 100 VM Test ===
     26=== OG-EXP-3: OpenGENI Single Site 100 VM Test ===
    2727
    2828This test was deemed "Pass: most criteria".  No problems were found, but only two scenario were tested due to lack of time.
    2929 
    30 === GR-EXP-4: GRAM Multi-site Acceptance Test ===
     30=== OG-EXP-4: OpenGENI Multi-site Acceptance Test ===
    3131
    3232This test was deemed "Blocked"  due to the lack of connectivity between racks.
     
    3434 - ticket 100 There is no connectivity between racks
    3535
    36 === GR-EXP-5: GRAM Network Resources Acceptance Test ===
     36=== OG-EXP-5: OpenGENI Network Resources Acceptance Test ===
    3737
    3838This test was deemed "Blocked"  due to the lack of connectivity between racks.
     
    4040 - ticket 100 There is no connectivity between racks
    4141
    42 === GR-EXP-6: GRAM and Meso-scale Multi-site !OpenFlow Acceptance Test ===
     42=== OG-EXP-6: OpenGENI and Meso-scale Multi-site !OpenFlow Acceptance Test ===
    4343
    4444This test was deemed "Blocked"  due to the lack of connectivity between racks.
     
    4646 - ticket 100 There is no connectivity between racks
    4747
    48 === GR-EXP-7: Click Router Experiment Acceptance Test ===
     48=== OG-EXP-7: Click Router Experiment Acceptance Test ===
    4949
    5050This test was deemed "Blocked"  due to the lack of connectivity between racks.
     
    5454== Administration Acceptance Tests ==
    5555
    56 === GR-ADM-1: Rack Receipt and Inventory Test ===
     56=== OG-ADM-1: Rack Receipt and Inventory Test ===
    5757
    5858This test was deemed "Completed" and needing further testing because there was no rack delivery process and there is no GMOC monitoring:
    5959
    60  - ticket 49 There is no GMOC data collection or GRAM monitoring data reporting
     60 - ticket 49 There is no GMOC data collection or OpenGENI monitoring data reporting
    6161
    62 === GR-ADM-2: Rack Administrator Access Test ===
     62=== OG-ADM-2: Rack Administrator Access Test ===
    6363
    6464This test was deemed "Pass: most criteria" because the site Administrator account instructions are incomplete and because remote access not fully validated.
     
    6868
    6969
    70 === GR-ADM-3: Full Rack Reboot Test ===
     70=== OG-ADM-3: Full Rack Reboot Test ===
    7171
    7272This test was deemed "In progress" because it was expected to have major impact on resource recovery.  Delayed until January 2014. Will verify ticket #83 when executed.
    7373
    7474
    75 === GR-ADM-4: Emergency Stop Test ===
     75=== OG-ADM-4: Emergency Stop Test ===
    7676
    7777This test was deemed "In Progress because it was expected to have major impact on resource recovery.  Delayed until January 2014. Will verify ticket #84 when executed.
    7878 
    7979
    80 === GR-ADM-5: Software Update Test ===
     80=== OG-ADM-5: Software Update Test ===
    8181
    82 This test was deemed "Complete" and needs to be re-executed due to the expected changes to the GRAM package which will introduce support for update operation, currently only debian package for initial install.
     82This test was deemed "Complete" and needs to be re-executed due to the expected changes to the OpenGENI package which will introduce support for update operation, currently only debian package for initial install.
    8383 
    84   - ticket 99  No GRAM package update; First install GRAM package not tested
     84  - ticket 99  No OpenGENI package update; First install GRAM package not tested
    8585
    86 === GR-ADM-6: Control Network Disconnection Test ===
     86=== OG-ADM-6: Control Network Disconnection Test ===
    8787
    8888This test was deemed "In Progress because it was expected to have major impact on resource recovery and because there is no GMOC monitoring:
    8989
    90  - ticket 49    There is no GMOC data collection or GRAM monitoring data reporting
     90 - ticket 49    There is no GMOC data collection or OpenGENI monitoring data reporting
    9191
    92 === GR-ADM-7: Documentation Review Test ===
     92=== OG-ADM-7: Documentation Review Test ===
    9393
    9494Significant improvement in documentation, but this test is deemed "In Progress". This documentation test case checks for the existence of 14 separate documents. Each are listed below  with the overall status and associated ticket.
    9595
    9696
    97   1. GRAM Rack Installation Documentation  - Pass: most criteria - ticket 74       
    98   2. GRAM Rack Part List Documentation  - Pass: most criteria - ticket 75               
    99   3. GRAM Software Components Identification Documentation - Pass               
    100   4. GRAM and OS Upgrade Plans  - In Progress - ticket  77               
    101   5. GRAM Software Components Functional Description and Access to Source code - Pass             
    102   6. GRAM Rack Resources Usage and Policy Documentation - Fail - ticket 79             
    103   7. GRAM Rack Resources Administration - Pass: most criteria - ticket 80           
    104   8. GRAM Rack Administrative procedure for user accounts - Pass: most criteria - ticket 81           
    105   9. GRAM Rack Administrative procedure for rack operations - Fail - ticket 82               
    106   10. GRAM Rack Administrative procedure for rack scheduled shutdown - In Progress - ticket 83           
    107   11. GRAM Rack Administrative procedure for Emergency Stop - In Progress - ticket 84             
    108   12. GRAM Rack Administrative procedure for health status - Fail - ticket 85         
    109   13. GRAM Rack Site Operations documentation  - Fail - ticket 86     
    110   14. Site Specific GRAM Deployment Data  - Fail - ticket 87           
     97  1. OpenGENI Rack Installation Documentation  - Pass: most criteria - ticket 74       
     98  2. OpenGENI Rack Part List Documentation  - Pass: most criteria - ticket 75               
     99  3. OpenGENI Software Components Identification Documentation - Pass               
     100  4. OpenGENI and OS Upgrade Plans  - In Progress - ticket  77               
     101  5. OpenGENI Software Components Functional Description and Access to Source code - Pass             
     102  6. OpenGENI Rack Resources Usage and Policy Documentation - Fail - ticket 79             
     103  7. OpenGENI Rack Resources Administration - Pass: most criteria - ticket 80           
     104  8. OpenGENI Rack Administrative procedure for user accounts - Pass: most criteria - ticket 81           
     105  9. OpenGENI Rack Administrative procedure for rack operations - Fail - ticket 82               
     106  10. OpenGENI Rack Administrative procedure for rack scheduled shutdown - In Progress - ticket 83           
     107  11. OpenGENI Rack Administrative procedure for Emergency Stop - In Progress - ticket 84             
     108  12. OpenGENI Rack Administrative procedure for health status - Fail - ticket 85         
     109  13. OpenGENI Rack Site Operations documentation  - Fail - ticket 86     
     110  14. Site Specific OpenGENI Deployment Data  - Fail - ticket 87           
    111111
    112112== Monitoring Acceptance Tests ==
    113113
    114 === GR-MON-1: Control Network Software and VLAN Inspection Test ===
     114=== OG-MON-1: Control Network Software and VLAN Inspection Test ===
    115115
    116 This test was deemed "Pass: met most criteria". All steps were successful, except for GRAM git access is BBN internal only.
     116This test was deemed "Pass: met most criteria". All steps were successful, except for OpenGENI git access is BBN internal only.
    117117
    118  - ticket 66 GRAM git access is BBN internal only, external access will be needed after initial evaluation
     118 - ticket 66 OpenGENI git access is BBN internal only, external access will be needed after initial evaluation
    119119
    120 === GR-MON-2: GENI Software Configuration Inspection Test ===
     120=== OG-MON-2: GENI Software Configuration Inspection Test ===
    121121
    122122This test was deemed "Pass". No issue were found.
    123123
    124 === GR-MON-3: GENI Active Experiment Inspection Test ===
     124=== OG-MON-3: GENI Active Experiment Inspection Test ===
    125125
    126126This test was deemed "Pass". No issue were found.
    127127
    128 === GR-MON-4: Infrastructure Device Performance Test ===
     128=== OG-MON-4: Infrastructure Device Performance Test ===
    129129
    130130This test  was deemed "Blocked" because it cannot be executed without monitoring data collection at the GMOC. Ticket tracking missing GMOC support:
    131131
    132  - ticket 49 There is no GMOC data collection or GRAM monitoring data reporting
     132 - ticket 49 There is no GMOC data collection or OpenGENI monitoring data reporting
    133133
    134 === GR-MON-5: GMOC Data Collection Test ===
     134=== OG-MON-5: GMOC Data Collection Test ===
    135135
    136136This test  was deemed "Blocked" because it cannot be executed without monitoring data collection at the GMOC. Ticket tracking missing GMOC support:
    137137
    138  - ticket 49 There is no GMOC data collection or GRAM monitoring data reporting
     138 - ticket 49 There is no GMOC data collection or OpenGENI monitoring data reporting