wiki:GENIRacksHome/OpenGENIRacks/AcceptanceTestReportApr2014

Version 5 (modified by lnevers@bbn.com, 5 years ago) (diff)

--

OpenGENI Intermediate Acceptance Test Report - April 2014

This pages captures the findings for each of the test cases in the OpenGENI Acceptance Test Plan. Results are grouped by major area including: Experimenter, Administrative and Monitoring Acceptance tests. Status for the current evaluation can be found in the OpenGENI Acceptance Test Status - April 2014 page. Once again there has been significant improvement since the last evaluation results captured in the Acceptance Test Report - December 2013. Most tickets reported in earlier evaluation have been successfully addressed.

Experimenter Acceptance Tests

OG-EXP-1: Bare Metal Support Acceptance Test

Test Case Status: "Fail"

Test case fails due to lack of supported for bare metal nodes and for Windows OS. Test Case Tickets:

  • 51 Bare metal support feature is missing for OG-EXP-1
  • 70 MS Windows support requires handling of account setup in OpenGENI

OG-EXP-2: OpenGENI Single Site Acceptance Test

Test Case Status: "Pass: most criteria"

This test uses resources in one OpenGENI resources and it was successfully executed. Test Case Tickets:

  • 104 Sliverstatus is inaccurate and nodes are shown as ready while they are not ready for use.
  • 106 There is an unexplained 6% packet loss with 1 pkt/sec traffic.
  • 110 There is no bandwidth shaping, experimenter specified capacity is ignored.

OG-EXP-3: OpenGENI Single Site 100 VM Test

Test Case Status: "Pass: most criteria"

This limits test case verified the ability to request 100 VMs. No problems were found, but only two scenario have been tested:

  • Fifty slices with 2 VMs each
  • Twenty slices with 5 VMs each

OG-EXP-4: OpenGENI Multi-site Acceptance Test

Test Case Status: "Pass".

GENI Network stitching was used to set up connection to InstaGENI and ExoGENI racks. Various topologies were tested, although could not try multi-point scenario becuase it is not yet supported by GENI Network Stitching. Test Case Tickets:

  • 106 There is an unexplained 6% packet loss with 1 pkt/sec traffic.
  • 110 There is no bandwidth shaping, experimenter specified capacity is ignored.

OG-EXP-5: OpenGENI Network Resources Acceptance Test

This test has not been executed yet, need site resources to be added to Clemson rack OpenFlow switch.

OG-EXP-6: OpenGENI and Meso-scale Multi-site OpenFlow Acceptance Test

Test Case Status: "In Progress".

This test case status is being executed with local OpenFlow connections and over stitched OpenFlow connections to InstaGENI sites. There is no testing with ExoGENI due to a bug in ExoGENI code that prevents user from specifying an OpenFlow Controller. OpenFlow controllers used successfully VMOC (patched POX) and FloodLight. Test Case Tickets:

  • 114 OpenGENI OpenFlow handshake failure with POX experimenter controllers - POX requires patching

OG-EXP-7: Click Router Experiment Acceptance Test

This test is not planned at this time.

Administration Acceptance Tests

OG-ADM-1: Rack Receipt and Inventory Test

Test Case Status: "In Progress".

Most of this test cannot be completed at BBN, there was no rack delivery process and there is no GMOC monitoring:

  • 49 There is no GMOC data collection or OpenGENI monitoring data reporting

OG-ADM-2: Rack Administrator Access Test

Test Case Status: "In Progress"

Waiting on Admin account creation procedure execution in Clemson rack. Remote Console access is to be verified:

  • 65 Remote console not available via web interface, some iDRAC access via SSH.

OG-ADM-3: Full Rack Reboot Test

Test Case Status: "Pass".

This test was executed and verified full rack power down and power up, along with process health at bootup. No issues remain.

OG-ADM-4: Emergency Stop Test

This test is not planned at this time because it requires requires GMOC interaction and monitoring. Without monitoring GMOC cannot participate in an emergency exercise:

  • 49 There is no GMOC data collection or OpenGENI monitoring data reporting

OG-ADM-5: Software Update Test

Test Case Status: "In Progress"

OpenGENI Debian package creation for initial install and upgrades tested.

OG-ADM-6: Control Network Disconnection Test

Test Case Status: "In Progress"

This test requires monitoring which is not available:

  • 49 There is no GMOC data collection or OpenGENI monitoring data reporting

OG-ADM-7: Documentation Review Test

Test Case Status: "In Progress"

Significant improvement in documentation since the last evaluation. This documentation test case checks for the existence of 14 separate documents. Each are listed below with the overall status and associated ticket.

  1. OpenGENI Rack Installation Documentation - Pass: most criteria - 74
  2. OpenGENI Rack Part List Documentation - Pass: most criteria - 75
  3. OpenGENI Software Components Identification Documentation - Pass
  4. OpenGENI and OS Upgrade Plans - Pass - 77
  5. OpenGENI Software Components Functional Description and Access to Source code - Pass
  6. OpenGENI Rack Resources Usage and Policy Documentation - In Progress - 79
  7. OpenGENI Rack Resources Administration - Pass: most criteria - 80
  8. OpenGENI Rack Administrative procedure for user accounts - Pass: most criteria
  9. OpenGENI Rack Administrative procedure for rack operations - Pass
  10. OpenGENI Rack Administrative procedure for rack scheduled shutdown - Pass
  11. OpenGENI Rack Administrative procedure for Emergency Stop - In Progress - 84
  12. OpenGENI Rack Administrative procedure for health status - Fail
  13. OpenGENI Rack Site Operations documentation - Fail - 86
  14. Site Specific OpenGENI Deployment Data - Fail - 87

Monitoring Acceptance Tests

OG-MON-1: Control Network Software and VLAN Inspection Test

Test Case Status: "In Progress"

Most checks were successful, except for OpenGENI git access is BBN internal only.

  • 66 OpenGENI git access is BBN internal only, external access will be needed after initial evaluation

OG-MON-2: GENI Software Configuration Inspection Test

This test was deemed "Pass". No issue were found.

OG-MON-3: GENI Active Experiment Inspection Test

Test Case Status: "In Progress"

Test is in progress, no issue were found so far.

OG-MON-4: Infrastructure Device Performance Test

Test Case Status: "Blocked"

Cannot be executed without monitoring data collection:

  • 49 There is no GMOC data collection or OpenGENI monitoring data reporting

OG-MON-5: GMOC Data Collection Test

Test Case Status: "Blocked"

Cannot be executed without monitoring data collection:

  • 49 There is no GMOC data collection or OpenGENI monitoring data reporting