wiki:GENIRacksHome/OpenGENIRacks/AcceptanceTestReportMay2013

Version 2 (modified by lnevers@bbn.com, 5 years ago) (diff)

--

OpenGENI Acceptance Test Report - May 2013

This pages captures the findings for each of the test cases in the OpenGENI Acceptance Test Plan.

Testing discovered some major issues which are critical in GENI. Each test case summarizes findings in this report and issues discovered are tracked in tickets. This is a summary of the most critical issues found and that are considered critical:

  • ticket 51 Bare metal support feature is missing
  • ticket 56 Allocated nodes do not use requested IP Address
  • ticket 57 Allocated nodes do not have sudo access setup for user
  • ticket 58 slivers with traffic are not deleted and remain available after sliver is deleted
  • ticket 63 Requesting 5 VM on 1 lan only results in ssh access via port 0
  • ticket 64 Creating 20 slices with 2 VMs on 1 vlan per sliver fails due to configuration limitations (10 VLAN available only)
  • ticket 68 Requesting host with two interfaces results in allocated host with one interface.
  • ticket 69 renewsliver fails with slice_object not defined error -- MAJOR ISSUE --
  • ticket 72 Unable to connect local campus resources nor remote meso-scale resources via OpenGENI OpenFlow
  • ticket 49 There is no GMOC data collection or OpenGENI monitoring data reporting for OG-MON-4/5

Experimenter Acceptance Tests

OG-EXP-1: Bare Metal Support Acceptance Test

This test case failed due to a several issues. There is no supported for bare metal nodes in OpenGENI. Modified the test case to replace all dedicated nodes (aka bare metal) with VMs to get test coverage. OpenStack does support MS Windows, but in order for OpenGENI to support this GENI required OS, there needs to be work done to add the set up of accounts and installation of SSH keys on the allocated VMs. Issues captured in tickets:

  • ticket 48 GPO PG account not recognized at by OpenGENI aggregate. (Used local Slice authority on controller node for testing).
  • ticket 51 Bare metal support feature is missing for OG-EXP-1
  • ticket 52 Advertisement RSpec fails rspeclint
  • ticket 54 performoperationalaction start fails with "code 13: Unsupported API call"
  • ticket 70 MS Windows support requires handling of account setup in OpenGENI

OG-EXP-2: OpenGENI Single Site Acceptance Test

This test was completed because most steps could be executed, but two major issue was discovered. There is no sudo privileges available for the experimenter which is required to perform various commonly executed operations (install package, configure interface, reboot node). Also user cannot specify IP Addresses to be used on nodes. Issues captured in tickets:

  • ticket 55 Requests for publically routable IP addresses is not supported
  • ticket 54 performoperationalaction start fails with "code 13: Unsupported API call"
  • ticket 56 Allocated nodes do not use requested IP Address
  • ticket 57 Allocated nodes do not have sudo access setup for user
  • ticket 58 slivers with traffic are not deleted and remain available after sliver is deleted

OG-EXP-3: OpenGENI Single Site 100 VM Test

This test case failed due to a several issues. Configuration limitations such as 10 VLAN restricted the number of experiments. Nodes with multiple interfaces do not always get all the requested interfaces. Same environment flakiness made compute node4 not available. Additional testing may have been run, but it was deemed not worth the time initial results.

  • ticket 62 compute node 4 is not available, even though it is configured as part of OpenGENI
  • ticket 63 Requesting 5 VM on 1 lan only results in ssh access via port 0
  • ticket 64 Creating 20 slices with 2 VMs on 1 vlan per sliver fails due to configuration limitations (10 VLAN available only)

OG-EXP-4: OpenGENI Multi-site Acceptance Test

This test case failed due to a several issues. Was unable to request a host with two interfaces, only one interface was allocated. Test requires running pingplus tool, which must be compiled on the nodes. There no sudo access and therefore cannot install "make" which is required to compile pingPlus. Also cannot install iperf for baseline measurements.

  • ticket 57 Allocated nodes do not have sudo access setup for user
  • ticket 67 readyToLogin.py fails for OpenGENI
  • ticket 68 Requesting host with two interfaces results in allocated host with one interface.

OG-EXP-5: OpenGENI Network Resources Acceptance Test

This test case failed because it could not be executed. OpenGENI does not support local campus non-rack resource and remote meso-scale site connections via rack OF switch. Ticket captures issue:

  • ticket 72 Unable to connect local campus resources nor remote meso-scale resources via OpenGENI OpenFlow

OG-EXP-6: OpenGENI and Meso-scale Multi-site OpenFlow Acceptance Test

This test case failed. Test case was modified to run in in one rack, also there are no remote Meso-scale OpenFlow aggregates. Failed sliver information is preserved in the aggregate and did not get cleaned up. Experimenter is not able to request IP addresses Ticket which is required for meso-scale connectivity. Could not install iperf for baseline measurements. Also some steps could not be completed because there are no tools available to install static flows into controller.

  • ticket 56 Allocated nodes do not use requested IP Address
  • ticket 57 Allocated nodes do not have sudo access setup for user
  • ticket 71 Failed create sliver attempts keep failed resource information in place

OG-EXP-7: Click Router Experiment Acceptance Test

This test case failed and could not be completed because it requires sudo privileges and multiple interface. Issues are captured by the following tickets:

  • ticket 57 Allocated nodes do not have sudo access setup for user
  • ticket 68 Requesting host with two interfaces results in allocated host with one interface.
  • ticket 69 renewsliver fails with slice_object not defined error

Administration Acceptance Tests

OG-ADM-1: Rack Receipt and Inventory Test

This test case walks through the delivery, initial connection and setup for a rack. Test case was not executed because no rack delivery, initial connection or set up occurred as part of the evaluation.

OG-ADM-2: Rack Administrator Access Test

This test case verified administrative access to all rack components. All components were accessible and console connections were available for all hosts and switches although no remote access was available for the consoles. An IPKVM was available in the rack, but was powered off and not configured for the current OpenGENI rack environment,this issue was captured in ticket:

  • ticket 65 Remote console not available, IPKVM powered off and disconnected.

OG-ADM-3: Full Rack Reboot Test

This test case verifies the existence of a procedure that a site administrator may need to perform for site maintenance. The procedure should define the shutdown and power sequence for all OpenGENI components. No such document was found, this issue was captured in ticket:

  • ticket 83 OpenGENI Rack Administrative procedure for rack scheduled shutdown - Documentation requirement

OG-ADM-4: Emergency Stop Test

This test case cannot be executed because an emergency stop is not supported for the initial evaluation. There is no GMOC involvement for the OpenGENI rack.

OG-ADM-5: Software Update Test

This test passed. A list of dependencies was determined for OpenGENI, but initial list of package candidate for updates were all up to date. Chose the glance package for updating. Glance is the OpenStack Image Registry and Delivery Service. Packages updated include Glance Daemons, API, Common, Registry, Python Library and Documentation. No issues were found.

OG-ADM-6: Control Network Disconnection Test

This test case cannot be executed because no GMOC monitoring is supported for the initial evaluation. There is no GMOC involvement for the OpenGENI rack.

OG-ADM-7: Documentation Review Test

This documentation test case checks for the existence of 14 separate documents. Each are listed below with the overall status and associated ticket.

  1. OpenGENI Rack Installation Documentation - Partially met - Ticket ticket 74
  2. OpenGENI Rack Part List Documentation - Fail - Ticket ticket 75
  3. OpenGENI Software Components Identification Documentation - Pass: most criteria - Ticket ticket 76
  4. OpenGENI and OS Upgrade Plans - Fail - Ticket ticket 77
  5. OpenGENI Software Components Functional Description and Access to Source code - Partially met - Ticket ticket 78
  6. OpenGENI Rack Resources Usage and Policy Documentation - Fail - Ticket ticket 79
  7. OpenGENI Rack Resources Administration - Fail - Ticket ticket 80
  8. OpenGENI Rack Administrative procedure for user accounts - Fail - Ticket ticket 81
  9. OpenGENI Rack Administrative procedure for rack operations - Fail - Ticket ticket 82
  10. OpenGENI Rack Administrative procedure for rack scheduled shutdown - Fail - Ticket ticket 83
  11. OpenGENI Rack Administrative procedure for Emergency Stop. - Fail - Ticket ticket 84
  12. OpenGENI Rack Administrative procedure for health status - Fail - Ticket ticket 85
  13. OpenGENI Rack Site Operations documentation - Fail - Ticket ticket 86
  14. Site Specific OpenGENI Deployment Data - Fail - Ticket ticket 87

Monitoring Acceptance Tests - May 2013

OG-MON-1: Control Network Software and VLAN Inspection Test

This test case was marked as "Pass: met most criteria". There is a requirement in GENI for the aggregate software to be available externally, which is not the case for OpenGENI at this time. This is reasonable for the stage of OpenGENI, so the test case passed because all function could be verified. These are the issues encountered by this test:

  • ticket 62 compute node 4 is not available, even though it is configured as part of OpenGENI
  • ticket 66 OpenGENI git access is BBN internal only, external access will be needed after initial evaluation

OG-MON-2: GENI Software Configuration Inspection Test

This test case passed. As an administrator, it is possible to get access to administrative interfaces to determine the configuration of OpenGENI resources as well as determining resource allocated to experiments. It is also possible to determine network resources configured and in use. No issue were reported for this test case.

OG-MON-3: GENI Active Experiment Inspection Test

This test case passed. As an administrator, it possible to determine all required information information to map compute resources to experiments and to experimenters, for historical and active experiments. It is possible to get information from the VMOC controller about the active experiments. It is also possible to get MAC address information as well as data path details for an experiment. No issue were reported for this test case.

OG-MON-4: Infrastructure Device Performance Test

This test could be executed because there is no GMOC monitoring available. Ticket tracking missing GMOC support:

  • ticket 49 There is no GMOC data collection or OpenGENI monitoring data reporting

OG-MON-5: GMOC Data Collection Test

This test could be executed because there is no monitoring data collection at the GMOC for the OpenGENI rack and the OpenGENI site is not reporting to the GMOC. Ticket tracking missing GMOC support:

  • ticket 49 There is no GMOC data collection or OpenGENI monitoring data reporting