Changes between Initial Version and Version 1 of GENIRacksHome/OpenGENIRacks/AcceptanceTestReportApr2014


Ignore:
Timestamp:
05/23/14 11:33:08 (8 years ago)
Author:
lnevers@bbn.com
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • GENIRacksHome/OpenGENIRacks/AcceptanceTestReportApr2014

    v1 v1  
     1= GRAM Intermediate Acceptance Test Report - April 2014 =
     2
     3This pages captures the findings for each of the test cases in the [wiki:GENIRacksHome/AcceptanceTests/GRAMAcceptanceTestsPlan GRAM Acceptance Test Plan]. Results are grouped by major area including: Experimenter, Administrative and Monitoring Acceptance tests. Status for the current evaluation can be found in the [wiki:GENIRacksHome/GRAMRacks/AcceptanceTestStatusApr2014 GRAM Acceptance Test Status - April 2014] page. Once again there has been significant improvement since the last evaluation results captured in the [https://superior.bbn.com/trac/bbn-rack/wiki/AcceptanceTestReportDec2013 Acceptance Test Report - December 2013]. Most tickets reported in earlier evaluation have been successfully addressed.
     4
     5== Experimenter Acceptance Tests ==
     6
     7=== GR-EXP-1: Bare Metal Support Acceptance Test ===
     8
     9Test Case Status: "Fail"
     10
     11Test case fails due to lack of supported for bare metal nodes and for Windows OS. Test Case Tickets:
     12
     13 - 51 Bare metal support feature is missing for GR-EXP-1
     14 - 70 MS Windows support requires handling of account setup in GRAM
     15
     16=== GR-EXP-2: GRAM Single Site Acceptance Test ===
     17
     18Test Case Status: "Pass: most criteria"
     19
     20This test uses resources in one GRAM resources and it was successfully executed. Test Case Tickets:
     21
     22 - 104 Sliverstatus is inaccurate and nodes are shown as ready while they are not ready for use.
     23 - 106 There is an unexplained 6% packet loss with 1 pkt/sec traffic.
     24 - 110 There is no bandwidth shaping, experimenter specified capacity is ignored.
     25
     26=== GR-EXP-3: GRAM Single Site 100 VM Test ===
     27
     28Test Case Status: "Pass: most criteria"
     29
     30This limits test case verified the ability to request 100 VMs. No problems were found, but only two scenario have been tested:
     31 - Fifty slices with 2 VMs each
     32 - Twenty slices with 5 VMs each
     33
     34
     35=== GR-EXP-4: GRAM Multi-site Acceptance Test ===
     36
     37Test Case Status: "Pass". 
     38
     39GENI Network stitching was used to set up connection to InstaGENI and ExoGENI racks.  Various topologies were
     40tested, although could not try multi-point scenario becuase it is not yet supported by GENI Network Stitching.
     41Test Case Tickets:
     42
     43 - 106 There is an unexplained 6% packet loss with 1 pkt/sec traffic.
     44 - 110 There is no bandwidth shaping, experimenter specified capacity is ignored.
     45
     46=== GR-EXP-5: GRAM Network Resources Acceptance Test ===
     47
     48This test has not been executed yet, need site resources to be added to Clemson rack !OpenFlow switch.
     49
     50=== GR-EXP-6: GRAM and Meso-scale Multi-site !OpenFlow Acceptance Test ===
     51Test Case Status: "In Progress". 
     52
     53This test case status is being executed with local OpenFlow connections and over stitched OpenFlow connections to InstaGENI sites.  There is no testing with ExoGENI due to a bug in ExoGENI code that prevents user from specifying an !OpenFlow Controller. !OpenFlow controllers used successfully VMOC (patched POX) and !FloodLight. Test Case Tickets:
     54
     55 - 114 GRAM OpenFlow handshake failure with POX experimenter controllers - POX requires patching
     56 
     57=== GR-EXP-7: Click Router Experiment Acceptance Test ===
     58
     59This test is not planned at this time.
     60
     61== Administration Acceptance Tests ==
     62
     63=== GR-ADM-1: Rack Receipt and Inventory Test ===
     64
     65Test Case Status: "In Progress". 
     66
     67Most of this test cannot be completed at BBN, there was no rack delivery process and there is no GMOC monitoring:
     68
     69 - 49   There is no GMOC data collection or GRAM monitoring data reporting
     70
     71=== GR-ADM-2: Rack Administrator Access Test ===
     72
     73Test Case Status: "In Progress"
     74
     75Waiting on Admin account creation procedure execution in Clemson rack. Remote Console access is to be verified:
     76
     77 - 65  Remote console not available via web interface, some iDRAC access via SSH.
     78
     79=== GR-ADM-3: Full Rack Reboot Test ===
     80
     81Test Case Status: "Pass". 
     82
     83This test was executed and verified full rack power down and power up, along with process health at bootup.  No issues remain.
     84
     85=== GR-ADM-4: Emergency Stop Test ===
     86
     87This test is not planned at this time because it requires requires GMOC interaction and monitoring.  Without monitoring GMOC cannot participate in an emergency exercise:
     88
     89 - 49   There is no GMOC data collection or GRAM monitoring data reporting
     90 
     91=== GR-ADM-5: Software Update Test ===
     92
     93Test Case Status: "In Progress"
     94
     95GRAM Debian package creation for initial install and upgrades tested.
     96
     97
     98=== GR-ADM-6: Control Network Disconnection Test ===
     99
     100Test Case Status: "In Progress"
     101
     102This test requires monitoring which is not available:
     103
     104 - 49   There is no GMOC data collection or GRAM monitoring data reporting
     105
     106=== GR-ADM-7: Documentation Review Test ===
     107
     108Test Case Status: "In Progress"
     109
     110Significant improvement in documentation since the last evaluation. This documentation test case checks for the existence of 14 separate documents. Each are listed below  with the overall status and associated ticket.
     111
     112  1. GRAM Rack Installation Documentation  - Pass: most criteria - 74       
     113  2. GRAM Rack Part List Documentation  - Pass: most criteria - 75               
     114  3. GRAM Software Components Identification Documentation - Pass               
     115  4. GRAM and OS Upgrade Plans  - Pass - 77               
     116  5. GRAM Software Components Functional Description and Access to Source code - Pass             
     117  6. GRAM Rack Resources Usage and Policy Documentation - In Progress - 79             
     118  7. GRAM Rack Resources Administration - Pass: most criteria - 80           
     119  8. GRAM Rack Administrative procedure for user accounts - Pass: most criteria           
     120  9. GRAM Rack Administrative procedure for rack operations - Pass             
     121  10. GRAM Rack Administrative procedure for rack scheduled shutdown - Pass           
     122  11. GRAM Rack Administrative procedure for Emergency Stop - In Progress - 84             
     123  12. GRAM Rack Administrative procedure for health status - Fail         
     124  13. GRAM Rack Site Operations documentation  - Fail - 86     
     125  14. Site Specific GRAM Deployment Data  - Fail - 87           
     126
     127
     128 
     129== Monitoring Acceptance Tests ==
     130
     131=== GR-MON-1: Control Network Software and VLAN Inspection Test ===
     132
     133Test Case Status: "In Progress"
     134
     135Most checks were successful, except for GRAM git access is BBN internal only.
     136
     137 - 66 GRAM git access is BBN internal only, external access will be needed after initial evaluation
     138
     139=== GR-MON-2: GENI Software Configuration Inspection Test ===
     140
     141This test was deemed "Pass". No issue were found.
     142
     143=== GR-MON-3: GENI Active Experiment Inspection Test ===
     144
     145Test Case Status: "In Progress"
     146
     147Test is in progress, no issue were found so far.
     148
     149=== GR-MON-4: Infrastructure Device Performance Test ===
     150
     151Test Case Status: "Blocked"
     152
     153Cannot be executed without monitoring data collection:
     154
     155 - 49 There is no GMOC data collection or GRAM monitoring data reporting
     156
     157=== GR-MON-5: GMOC Data Collection Test ===
     158
     159Test Case Status: "Blocked"
     160
     161Cannot be executed without monitoring data collection:
     162
     163 - 49 There is no GMOC data collection or GRAM monitoring data reporting