| | 1 | = GRAM Acceptance Test Report - May 2013 = |
| | 2 | |
| | 3 | This pages captures the findings for each of the test cases in the [wiki:GENIRacksHome/AcceptanceTests/GRAMAcceptanceTestsPlan GRAM Acceptance Test Plan]. |
| | 4 | |
| | 5 | |
| | 6 | Testing discovered some major issues which are critical in GENI. Each test case summarizes findings in this report and issues |
| | 7 | discovered are tracked in tickets. This is a summary of the most critical issues found and that are considered critical: |
| | 8 | |
| | 9 | - ticket 51 Bare metal support feature is missing |
| | 10 | - ticket 56 Allocated nodes do not use requested IP Address |
| | 11 | - ticket 57 Allocated nodes do not have sudo access setup for user |
| | 12 | - ticket 58 slivers with traffic are not deleted and remain available after sliver is deleted |
| | 13 | - ticket 63 Requesting 5 VM on 1 lan only results in ssh access via port 0 |
| | 14 | - ticket 64 Creating 20 slices with 2 VMs on 1 vlan per sliver fails due to configuration limitations (10 VLAN available only) |
| | 15 | - ticket 68 Requesting host with two interfaces results in allocated host with one interface. |
| | 16 | - ticket 69 renewsliver fails with slice_object not defined error -- MAJOR ISSUE -- |
| | 17 | - ticket 72 Unable to connect local campus resources nor remote meso-scale resources via GRAM !OpenFlow |
| | 18 | - ticket 49 There is no GMOC data collection or GRAM monitoring data reporting for GR-MON-4/5 |
| | 19 | |
| | 20 | |
| | 21 | == Experimenter Acceptance Tests == |
| | 22 | |
| | 23 | === GR-EXP-1: Bare Metal Support Acceptance Test === |
| | 24 | |
| | 25 | This test case failed due to a several issues. There is no supported for bare metal nodes in GRAM. |
| | 26 | Modified the test case to replace all dedicated nodes (aka bare metal) with VMs to get test coverage. |
| | 27 | !OpenStack does support MS Windows, but in order for GRAM to support this GENI required OS, there needs |
| | 28 | to be work done to add the set up of accounts and installation of SSH keys on the allocated VMs. |
| | 29 | Issues captured in tickets: |
| | 30 | |
| | 31 | - ticket 48 GPO PG account not recognized at by GRAM aggregate. (Used local Slice authority on controller node for testing). |
| | 32 | - ticket 51 Bare metal support feature is missing for GR-EXP-1 |
| | 33 | - ticket 52 Advertisement RSpec fails rspeclint |
| | 34 | - ticket 54 performoperationalaction start fails with "code 13: Unsupported API call" |
| | 35 | - ticket 70 MS Windows support requires handling of account setup in GRAM |
| | 36 | |
| | 37 | === GR-EXP-2: GRAM Single Site Acceptance Test === |
| | 38 | |
| | 39 | This test was completed because most steps could be executed, but two major issue was discovered. |
| | 40 | There is no ''sudo'' privileges available for the experimenter which is required to perform various |
| | 41 | commonly executed operations (install package, configure interface, reboot node). |
| | 42 | Also user cannot specify IP Addresses to be used on nodes. Issues captured in tickets: |
| | 43 | |
| | 44 | - ticket 55 Requests for publically routable IP addresses is not supported |
| | 45 | - ticket 54 performoperationalaction start fails with "code 13: Unsupported API call" |
| | 46 | - ticket 56 Allocated nodes do not use requested IP Address |
| | 47 | - ticket 57 Allocated nodes do not have sudo access setup for user |
| | 48 | - ticket 58 slivers with traffic are not deleted and remain available after sliver is deleted |
| | 49 | |
| | 50 | === GR-EXP-3: GRAM Single Site 100 VM Test === |
| | 51 | |
| | 52 | This test case failed due to a several issues. Configuration limitations such as 10 VLAN |
| | 53 | restricted the number of experiments. Nodes with multiple interfaces do not always get all |
| | 54 | the requested interfaces. Same environment flakiness made compute node4 not available. |
| | 55 | Additional testing may have been run, but it was deemed not worth the time initial results. |
| | 56 | |
| | 57 | - ticket 62 compute node 4 is not available, even though it is configured as part of GRAM |
| | 58 | - ticket 63 Requesting 5 VM on 1 lan only results in ssh access via port 0 |
| | 59 | - ticket 64 Creating 20 slices with 2 VMs on 1 vlan per sliver fails due to configuration limitations (10 VLAN available only) |
| | 60 | |
| | 61 | === GR-EXP-4: GRAM Multi-site Acceptance Test === |
| | 62 | |
| | 63 | This test case failed due to a several issues. Was unable to request a host with two interfaces, |
| | 64 | only one interface was allocated. Test requires running pingplus tool, which must be compiled on |
| | 65 | the nodes. There no sudo access and therefore cannot install "make" which is required to compile |
| | 66 | pingPlus. Also cannot install iperf for baseline measurements. |
| | 67 | |
| | 68 | - ticket 57 Allocated nodes do not have sudo access setup for user |
| | 69 | - ticket 67 readyToLogin.py fails for GRAM |
| | 70 | - ticket 68 Requesting host with two interfaces results in allocated host with one interface. |
| | 71 | |
| | 72 | === GR-EXP-5: GRAM Network Resources Acceptance Test === |
| | 73 | |
| | 74 | This test case failed because it could not be executed. GRAM does not support local campus non-rack |
| | 75 | resource and remote meso-scale site connections via rack OF switch. Ticket captures issue: |
| | 76 | |
| | 77 | - ticket 72 Unable to connect local campus resources nor remote meso-scale resources via GRAM !OpenFlow |
| | 78 | |
| | 79 | === GR-EXP-6: GRAM and Meso-scale Multi-site !OpenFlow Acceptance Test === |
| | 80 | |
| | 81 | This test case failed. Test case was modified to run in in one rack, also there are no remote Meso-scale |
| | 82 | !OpenFlow aggregates. Failed sliver information is preserved in the aggregate and did not get cleaned up. |
| | 83 | Experimenter is not able to request IP addresses Ticket which is required for meso-scale connectivity. |
| | 84 | Could not install iperf for baseline measurements. Also some steps could not be completed because there |
| | 85 | are no tools available to install static flows into controller. |
| | 86 | |
| | 87 | - ticket 56 Allocated nodes do not use requested IP Address |
| | 88 | - ticket 57 Allocated nodes do not have sudo access setup for user |
| | 89 | - ticket 71 Failed create sliver attempts keep failed resource information in place |
| | 90 | |
| | 91 | === GR-EXP-7: Click Router Experiment Acceptance Test === |
| | 92 | This test case failed and could not be completed because it requires sudo privileges and multiple interface. |
| | 93 | Issues are captured by the following tickets: |
| | 94 | |
| | 95 | - ticket 57 Allocated nodes do not have sudo access setup for user |
| | 96 | - ticket 68 Requesting host with two interfaces results in allocated host with one interface. |
| | 97 | - ticket 69 renewsliver fails with slice_object not defined error |
| | 98 | |
| | 99 | |
| | 100 | == Administration Acceptance Tests == |
| | 101 | |
| | 102 | === GR-ADM-1: Rack Receipt and Inventory Test === |
| | 103 | |
| | 104 | This test case walks through the delivery, initial connection and setup for a rack. Test case was |
| | 105 | not executed because no rack delivery, initial connection or set up occurred as part of the evaluation. |
| | 106 | |
| | 107 | === GR-ADM-2: Rack Administrator Access Test === |
| | 108 | |
| | 109 | This test case verified administrative access to all rack components. All components were accessible |
| | 110 | and console connections were available for all hosts and switches although no remote access was |
| | 111 | available for the consoles. An IPKVM was available in the rack, but was powered off and not configured |
| | 112 | for the current GRAM rack environment,this issue was captured in ticket: |
| | 113 | |
| | 114 | - ticket 65 Remote console not available, IPKVM powered off and disconnected. |
| | 115 | |
| | 116 | === GR-ADM-3: Full Rack Reboot Test === |
| | 117 | |
| | 118 | This test case verifies the existence of a procedure that a site administrator may need to perform |
| | 119 | for site maintenance. The procedure should define the shutdown and power sequence for all GRAM |
| | 120 | components. No such document was found, this issue was captured in ticket: |
| | 121 | |
| | 122 | - ticket 83 GRAM Rack Administrative procedure for rack scheduled shutdown - Documentation requirement |
| | 123 | |
| | 124 | === GR-ADM-4: Emergency Stop Test === |
| | 125 | |
| | 126 | This test case cannot be executed because an emergency stop is not supported for the initial evaluation. |
| | 127 | There is no GMOC involvement for the GRAM rack. |
| | 128 | |
| | 129 | |
| | 130 | === GR-ADM-5: Software Update Test === |
| | 131 | |
| | 132 | This test passed. A list of dependencies was determined for GRAM, but initial list of package candidate |
| | 133 | for updates were all up to date. Chose the ''glance'' package for updating. Glance is the !OpenStack |
| | 134 | Image Registry and Delivery Service. Packages updated include Glance Daemons, API, Common, Registry, |
| | 135 | Python Library and Documentation. No issues were found. |
| | 136 | |
| | 137 | === GR-ADM-6: Control Network Disconnection Test === |
| | 138 | |
| | 139 | This test case cannot be executed because no GMOC monitoring is supported for the initial evaluation. |
| | 140 | There is no GMOC involvement for the GRAM rack. |
| | 141 | |
| | 142 | === GR-ADM-7: Documentation Review Test === |
| | 143 | |
| | 144 | This documentation test case checks for the existence of 14 separate documents. Each are listed below |
| | 145 | with the overall status and associated ticket. |
| | 146 | |
| | 147 | 1. GRAM Rack Installation Documentation - Partially met - Ticket ticket 74 |
| | 148 | 2. GRAM Rack Part List Documentation - Fail - Ticket ticket 75 |
| | 149 | 3. GRAM Software Components Identification Documentation - Pass: most criteria - Ticket ticket 76 |
| | 150 | 4. GRAM and OS Upgrade Plans - Fail - Ticket ticket 77 |
| | 151 | 5. GRAM Software Components Functional Description and Access to Source code - Partially met - Ticket ticket 78 |
| | 152 | 6. GRAM Rack Resources Usage and Policy Documentation - Fail - Ticket ticket 79 |
| | 153 | 7. GRAM Rack Resources Administration - Fail - Ticket ticket 80 |
| | 154 | 8. GRAM Rack Administrative procedure for user accounts - Fail - Ticket ticket 81 |
| | 155 | 9. GRAM Rack Administrative procedure for rack operations - Fail - Ticket ticket 82 |
| | 156 | 10. GRAM Rack Administrative procedure for rack scheduled shutdown - Fail - Ticket ticket 83 |
| | 157 | 11. GRAM Rack Administrative procedure for Emergency Stop. - Fail - Ticket ticket 84 |
| | 158 | 12. GRAM Rack Administrative procedure for health status - Fail - Ticket ticket 85 |
| | 159 | 13. GRAM Rack Site Operations documentation - Fail - Ticket ticket 86 |
| | 160 | 14. Site Specific GRAM Deployment Data - Fail - Ticket ticket 87 |
| | 161 | |
| | 162 | |
| | 163 | == Monitoring Acceptance Tests - May 2013 == |
| | 164 | |
| | 165 | === GR-MON-1: Control Network Software and VLAN Inspection Test === |
| | 166 | This test case was marked as "Pass: met most criteria". There is a requirement in GENI for the aggregate software |
| | 167 | to be available externally, which is not the case for GRAM at this time. This is reasonable for the stage of GRAM, |
| | 168 | so the test case passed because all function could be verified. These are the issues encountered by this test: |
| | 169 | |
| | 170 | - ticket 62 compute node 4 is not available, even though it is configured as part of GRAM |
| | 171 | - ticket 66 GRAM git access is BBN internal only, external access will be needed after initial evaluation |
| | 172 | |
| | 173 | |
| | 174 | === GR-MON-2: GENI Software Configuration Inspection Test === |
| | 175 | This test case passed. As an administrator, it is possible to get access to administrative interfaces to determine |
| | 176 | the configuration of GRAM resources as well as determining resource allocated to experiments. It is also possible |
| | 177 | to determine network resources configured and in use. No issue were reported for this test case. |
| | 178 | |
| | 179 | === GR-MON-3: GENI Active Experiment Inspection Test === |
| | 180 | |
| | 181 | This test case passed. As an administrator, it possible to determine all required information information to map |
| | 182 | compute resources to experiments and to experimenters, for historical and active experiments. It is possible to |
| | 183 | get information from the VMOC controller about the active experiments. It is also possible to get MAC address |
| | 184 | information as well as data path details for an experiment. No issue were reported for this test case. |
| | 185 | |
| | 186 | |
| | 187 | === GR-MON-4: Infrastructure Device Performance Test === |
| | 188 | |
| | 189 | This test could be executed because there is no GMOC monitoring available. Ticket tracking missing GMOC support: |
| | 190 | |
| | 191 | - ticket 49 There is no GMOC data collection or GRAM monitoring data reporting |
| | 192 | |
| | 193 | === GR-MON-5: GMOC Data Collection Test === |
| | 194 | |
| | 195 | This test could be executed because there is no monitoring data collection at the GMOC for the GRAM rack and the |
| | 196 | GRAM site is not reporting to the GMOC. Ticket tracking missing GMOC support: |
| | 197 | |
| | 198 | - ticket 49 There is no GMOC data collection or GRAM monitoring data reporting |
