wiki:GENIRacksHome/OpenGENIRacks/AcceptanceTestReportDec2013

Version 1 (modified by lnevers@bbn.com, 10 years ago) (diff)

--

GRAM Intermediate Acceptance Test Report - December 2013

This pages captures the findings for each of the test cases in the GRAM Acceptance Test Plan?. Status for the current evaluation can be found in the GRAM Acceptance Test Status - December 2013? page. As can be seen from the page there has been significant improvement since the last evaluation results captured in the Acceptance Test Report - May 2013? (May 2013). Also many tickets reported in earlier evaluation have been successfully addressed. Following is a list of the most critical issues as of 2013-12-18:

  • ticket 100 There is no connectivity between racks
  • ticket 49 There is no GMOC data collection or GRAM monitoring data reporting for GR-MON-4/5
  • ticket 51 Bare metal support feature is missing
  • ticket 99 GRAM Software packaging
  • ticket 65 Remote console not available via web interface, some iDRAC access via SSH.
  • ticket 66 GRAM git access is BBN internal only, external access will be needed after initial eval

Experimenter Acceptance Tests

GR-EXP-1: Bare Metal Support Acceptance Test

This test case was deemed "Fail" due to the fact that there is still no supported for bare metal nodes and there is no support for windows. Modified the test case to replace all dedicated nodes (aka bare metal) with VMs to get test coverage.Issues captured in tickets:

  • ticket 51 Bare metal support feature is missing for GR-EXP-1
  • ticket 70 MS Windows support requires handling of account setup in GRAM

GR-EXP-2: GRAM Single Site Acceptance Test

This test was deemed "Pass". No issues were found.

GR-EXP-3: GRAM Single Site 100 VM Test

This test was deemed "Pass: most criteria". No problems were found, but only two scenario were tested due to lack of time.

GR-EXP-4: GRAM Multi-site Acceptance Test

This test was deemed "Blocked" due to the lack of connectivity between racks.

  • ticket 100 There is no connectivity between racks

GR-EXP-5: GRAM Network Resources Acceptance Test

This test was deemed "Blocked" due to the lack of connectivity between racks.

  • ticket 100 There is no connectivity between racks

GR-EXP-6: GRAM and Meso-scale Multi-site OpenFlow Acceptance Test

This test was deemed "Blocked" due to the lack of connectivity between racks.

  • ticket 100 There is no connectivity between racks

GR-EXP-7: Click Router Experiment Acceptance Test

This test was deemed "Blocked" due to the lack of connectivity between racks.

  • ticket 100 There is no connectivity between racks

Administration Acceptance Tests

GR-ADM-1: Rack Receipt and Inventory Test

This test was deemed "Completed" and needing further testing because there was no rack delivery process and there is no GMOC monitoring:

  • ticket 49 There is no GMOC data collection or GRAM monitoring data reporting

GR-ADM-2: Rack Administrator Access Test

This test was deemed "Pass: most criteria" because the site Administrator account instructions are incomplete and because remote access not fully validated.

  • ticket 95 Administrative account instructions are not complete
  • ticket 65 Remote console not available via web interface, some iDRAC access via SSH.

GR-ADM-3: Full Rack Reboot Test

This test was deemed "In progress" because it was expected to have major impact on resource recovery. Delayed until January 2014. Will verify ticket #83 when executed.

GR-ADM-4: Emergency Stop Test

This test was deemed "In Progress because it was expected to have major impact on resource recovery. Delayed until January 2014. Will verify ticket #84 when executed.

GR-ADM-5: Software Update Test

This test was deemed "Complete" and needs to be re-executed due to the expected changes to the GRAM package which will introduce support for update operation, currently only debian package for initial install.

  • ticket 99 No GRAM package update; First install GRAM package not tested

GR-ADM-6: Control Network Disconnection Test

This test was deemed "In Progress because it was expected to have major impact on resource recovery and because there is no GMOC monitoring:

  • ticket 49 There is no GMOC data collection or GRAM monitoring data reporting

GR-ADM-7: Documentation Review Test

Significant improvement in documentation, but this test is deemed "In Progress". This documentation test case checks for the existence of 14 separate documents. Each are listed below with the overall status and associated ticket.

  1. GRAM Rack Installation Documentation - Pass: most criteria - ticket 74
  2. GRAM Rack Part List Documentation - Pass: most criteria - ticket 75
  3. GRAM Software Components Identification Documentation - Pass
  4. GRAM and OS Upgrade Plans - In Progress - ticket 77
  5. GRAM Software Components Functional Description and Access to Source code - Pass
  6. GRAM Rack Resources Usage and Policy Documentation - Fail - ticket 79
  7. GRAM Rack Resources Administration - Pass: most criteria - ticket 80
  8. GRAM Rack Administrative procedure for user accounts - Pass: most criteria - ticket 81
  9. GRAM Rack Administrative procedure for rack operations - Fail - ticket 82
  10. GRAM Rack Administrative procedure for rack scheduled shutdown - In Progress - ticket 83
  11. GRAM Rack Administrative procedure for Emergency Stop - In Progress - ticket 84
  12. GRAM Rack Administrative procedure for health status - Fail - ticket 85
  13. GRAM Rack Site Operations documentation - Fail - ticket 86
  14. Site Specific GRAM Deployment Data - Fail - ticket 87

Monitoring Acceptance Tests

GR-MON-1: Control Network Software and VLAN Inspection Test

This test was deemed "Pass: met most criteria". All steps were successful, except for GRAM git access is BBN internal only.

  • ticket 66 GRAM git access is BBN internal only, external access will be needed after initial evaluation

GR-MON-2: GENI Software Configuration Inspection Test

This test was deemed "Pass". No issue were found.

GR-MON-3: GENI Active Experiment Inspection Test

This test was deemed "Pass". No issue were found.

GR-MON-4: Infrastructure Device Performance Test

This test was deemed "Blocked" because it cannot be executed without monitoring data collection at the GMOC. Ticket tracking missing GMOC support:

  • ticket 49 There is no GMOC data collection or GRAM monitoring data reporting

GR-MON-5: GMOC Data Collection Test

This test was deemed "Blocked" because it cannot be executed without monitoring data collection at the GMOC. Ticket tracking missing GMOC support:

  • ticket 49 There is no GMOC data collection or GRAM monitoring data reporting