1 | | [[PageOutline]] |
2 | | |
3 | | = GENI Desktop Project Status Report = |
4 | | |
5 | | Period: Solicitation 4 Year 2 |
6 | | |
7 | | == I. Major accomplishments == |
8 | | |
9 | | The following highlights our accomplishments during the last year. |
10 | | |
11 | | === A. Milestones achieved === |
12 | | |
13 | | * Modified the GENI Desktop to support "Speaks-for" authentication being developed/supported by the control frameworks and other GENI tools/services. |
14 | | |
15 | | * Incorporated user-driven feedback into the GENI Desktop to support new user-requested services and features. |
16 | | |
17 | | * Developed new training materials that incorporate the changes made to the GENI Desktop. |
18 | | |
19 | | * Integrated and leveraged existing tools and services (e.g., Jacks) into the GENI Desktop for managing topologies, experiments, and results. |
20 | | |
21 | | * Collected user feedback regarding usability of the GENI Desktop and made major changes to improve its ease-of-use and aesthetics. |
22 | | |
23 | | * Adapted the GENI Desktop archiving service for storing and retrieving experiment results and artifacts from iRoDs to support a new archival service with enhanced features. |
24 | | |
25 | | * Enhanced the set of scriptable resource management, instrumentation, and monitoring available to experimenters and other tools. |
26 | | |
27 | | * Enabled integration of the GENI Desktop with other experimenter tools. |
28 | | |
29 | | * Created documentation and tutorial materials to reflect this latest version of the GENI Desktop. |
30 | | |
31 | | === B. Deliverables made === |
32 | | |
33 | | * We enhanced the GENI Desktop to use "Speaks-for" credentials for accessing resources from other GENI components on behalf of users. |
34 | | |
35 | | * We implemented an initial version of the slice verification and configuration testing service. |
36 | | |
37 | | * We developed code for super slice support in the GENI Desktop. |
38 | | |
39 | | * We developed a completely new user interface for the GENI Desktop which we call "GENI Desktop Lite" that greatly improves the look-and-feel and ease-of-use of the GENI Desktop. |
40 | | |
41 | | * We integrated the Jacks tool into the GENI Desktop, enabling users to create topologies and instantiate experiments (i.e., slices) using the Jacks tool. We also integrated the Adopt-A-GENI (AAG) tool into the GENI desktop. |
42 | | |
43 | | * We developed and integrated a new archival service into the GENI Desktop that leverages VMs to hold, and later display, the archived experiment state in the same context as it was initially collected and viewed. |
44 | | |
45 | | * We designed and implemented a GENI Desktop Command Line Interface (gdcli) that enables users to write scripts that control, manage, and measure the performance of their slices through the GENI Desktop. |
46 | | |
47 | | * We demonstrated how the new gdcli can be used by other experimenter tools to integrate with the GENI Desktop. |
48 | | |
49 | | * We developed online documentation for the new gdcli interface and gave a tutorial entitled "Monitoring and Controlling Experiments with GENI Desktop Scripts and Modules" at the GEC 23 conference. |
50 | | |
51 | | == II. Description of work performed during the last year == |
52 | | |
53 | | The following provides a description of the progress made during the last year. |
54 | | |
55 | | === A. Activities and findings === |
56 | | |
57 | | Our activities this past year have resulted in enhanced functionality, |
58 | | improved ease-of-use, better authentication/security, a redesigned user |
59 | | interface, and the ability to control the GENI Desktop programmatically |
60 | | throught scripts. In particular, we: |
61 | | |
62 | | * Incorporated support for "Speaks-for" into the GENI Desktop |
63 | | |
64 | | * Designed and implemented a slice verification service |
65 | | |
66 | | * Designed and implemented a super slice abstraction |
67 | | |
68 | | * Developed a new archival services based on Xen VMs to restore entire contexts |
69 | | |
70 | | * Redesigned the look-and-feel of the GENI Desktop user interface to make it much easier to use |
71 | | |
72 | | * Integrated Jacks and Adopt-A-GENI (AAG) tools into the GENI Desktop |
73 | | |
74 | | * Designed and implemented a GENI Desktop Command Line Interface (gdcli) to enable script-based access to GENI Desktop functionality. |
75 | | |
76 | | The following briefly describes our activities this past year. We beginning with our efforts to |
77 | | improve the authentication/authorization used to access the GENI Desktop, and |
78 | | then move on to describe new functionality added to the GENI Desktop, |
79 | | followed by a description of our efforts to enhance the look-and-feel of the |
80 | | GENI Desktop, and |
81 | | lastly our efforts to allow scripting of the GENI Desktop. |
82 | | |
83 | | |
84 | | ==== Authorization: Supporting "Speaks-for" ==== |
85 | | |
86 | | The "Speaks-for" credential allows trusted tools to act for, instead of |
87 | | acting as, an experimenter to perform certain actions, such as requesting |
88 | | resources from aggregates, accessing allocated resources, and installing |
89 | | software on experimental nodes. We enhanced the GENI Desktop to use "Speaks-for" |
90 | | credentials for accessing resources on behalf of users. Users no longer |
91 | | need to provide the private key to the GENI Desktop. We implemented an interface |
92 | | for the user to authorize the GENI Desktop to speak for her/him. A GENI |
93 | | Desktop-specific certificate is signed using the private key of the user. |
94 | | Because the whole process happens within the browser on the client side, |
95 | | the private key never leaves the user's machine. The "Speaks-for" credential |
96 | | allows the GENI Desktop to talk to aggregates and perform all necessary |
97 | | actions on behalf of the user. |
98 | | |
99 | | ==== New Functionality (1): Jacks, Archival, Verification and Super Slice Services ==== |
100 | | |
101 | | Another goal this past year has been to begin integrating support for other tools |
102 | | into the GENI Desktop. To demonstrate the ability to support other tools, we |
103 | | began by integrating the Jacks tool into the GENI Desktop. Users can now add |
104 | | resources to their slices by selecting Jacks in the GENI Desktop which will |
105 | | direct them to a GENI Desktop page that embeds the Jacks tool and allows them |
106 | | to allocate the resources (i.e., which uses the OMNI tool). RSPECs |
107 | | created by Jacks can be saved by the GENI Desktop for future use. |
108 | | |
109 | | In addition, we integrated the Adopt-A-GENI (AAG) flow specification module into |
110 | | the GENI Desktop, allowing users to visually define OpenFlow paths across the |
111 | | topology that are then sent to the AAG module to be instantiated in the |
112 | | OpenFlow controller. Although the AAG functionality is logically a distinct |
113 | | service/tool, the messaging system between windows in the GENI Desktop made |
114 | | it possible to incorporate this new tool with relatively little effort. In |
115 | | addition, we were able to add a new AAG Controller node type to the Jacks |
116 | | wrapper, thereby integrating the AAG controller into the Jacks tool as well. |
117 | | |
118 | | The existing archival service in the GENI Desktop leveraged the iRoDs storage |
119 | | service to store and later retrieve measurement data collected by the GENI |
120 | | Desktop. A key limitation of this service was the inability to easily (and |
121 | | quickly) access, view, and make sense of archived measurement data. To |
122 | | address this need, we developed a new archival service that not only archives |
123 | | the measurement data, but also archives the software and context used to |
124 | | display the data. Because the data and the environment needed to view the |
125 | | data are archived, users can quickly access an archive and view the saved |
126 | | data using the same tools available at the time the data was collected. |
127 | | |
128 | | To support this new archival service, we implemented an archival server that |
129 | | not only captures the measurement data stored on the global node (where |
130 | | measurement data is collected), but it also captures the state of the drupal |
131 | | system used to display the data, including all web server (apache) and |
132 | | database (mysql) files. GENI Desktop users can request that an archive be |
133 | | made, which is then sent to the archive server. When a user visits the |
134 | | archive web page on the archive server, they can select from any of the |
135 | | archived snapshots. The archive server will dynamically launch a Xen VM, |
136 | | setup the apache, mysql, and Drupal state needed to view the measurement |
137 | | data, install the archived measurement data, create login credentials for the |
138 | | user, and share the credentials with the GD so the user is automatically logged |
139 | | into the archive VM. The result is that the user is presented with the same |
140 | | look-and-feel as if they had gone to the global node at the time the snapshot was taken. |
141 | | |
142 | | We implemented the slice verification and configuration testing service |
143 | | as a module in GENI Desktop by taking advantage of the module builder function |
144 | | of the GENI Desktop. Based on the manifest of an experiment, the verification |
145 | | service analyzes the topology and performs tests about the interfaces of all |
146 | | nodes in the experiment. The initial version we implemented checks whether |
147 | | each interface is up and whether it is reachable from a ping test. |
148 | | The results are presented in a table showing the status of all the interfaces of |
149 | | all the nodes in the experiment. |
150 | | Later versions of the verification service included |
151 | | additional checks (particularly automated bandwidth checks) and also made it |
152 | | possible for users to write their own verification scripts to test for things |
153 | | of importance to their experiment. |
154 | | |
155 | | Building a large experiment is a difficult task in GENI, partly because |
156 | | it is more likely to fail if we create an experiment with a lot of nodes. |
157 | | At the same time, we may have multiple related experiments and want to |
158 | | combine these relatively small experiments together to form a large experiment. |
159 | | We developed a new "super slice" service in the GENI Desktop to support this functionality. |
160 | | Users can use the GENI Desktop to create a super slice by combining multiple |
161 | | existing slices together. The GENI Desktop provides a GUI for users to display |
162 | | multiple slices at the same time and pick any pair of nodes from different |
163 | | slices to establish a link between them. The Super Slice service in the GENI |
164 | | Desktop currently can then automatically set up GRE tunnels between these selected pairs of nodes |
165 | | from different slices. |
166 | | |
167 | | ==== A New User Interface: GENI Desktop Lite ==== |
168 | | |
169 | | Over the years the number of features and capabilities offered by the GENI |
170 | | Desktop has continued to expand. Indeed, a key goal of the GENI Desktop was |
171 | | to provide users with a context for managing all aspects of their experiment |
172 | | from setup and deployment to monitoring and archiving of measurements and |
173 | | results. The downside to this expanded functionality is increased complexity |
174 | | using the tool. |
175 | | At the same time, the number of experimenters who are using the GENI Desktop to |
176 | | create, manage, monitor, and control their slices has been grown rapidly, |
177 | | due, in part, to users being exposed to the GENI Desktop as part |
178 | | of GENI tutorials, summer camps, demontrations and online documents and |
179 | | videos. Feedback from this user group indicated that the extensive |
180 | | functionality available in GENI Desktop made it difficult for new users to |
181 | | navigate and use. |
182 | | |
183 | | To address this need for a tool that could be easily learned and used by new |
184 | | users, we completely redesigned the look-and-feel of the GENI Desktop to |
185 | | reduce complexity and make it simple to create, run, and monitor |
186 | | experiments. Our new "GENI Desktop (GD) Lite" interface is now the default |
187 | | intereface that users see when they log into the GENI Desktop. |
188 | | Users can still access the (original) advanced user interface if needed, but in most |
189 | | cases find that the GD Lite interface is sufficient. |
190 | | The Lite intereface is designed to take users through the lifecycle of an |
191 | | experiment. The Lite intereface starts by helping users create a slice, |
192 | | assigning resources to the slice, and then giving them access to a simplified |
193 | | version of the GENI Desktop topology view where they can log in to nodes, run |
194 | | their experiment, monitor basic traffic types, and archive results. Initial |
195 | | feedback on the new intereface has been extremely positive. |
196 | | |
197 | | In addition to a major rewrite of the web code for the user interface, one of the key |
198 | | challenges that we had to address was automating the global node setup, initialization, |
199 | | and instrumentation. While these "backend" operations were clearly visible |
200 | | in the old user interface, they had to be hidden in the new interface. This |
201 | | meant that the GENI Desktop had to be able to add global nodes into the slice |
202 | | (one for each aggregate) on the user's behalf. This required working with |
203 | | the aggregates to support the GENI AM API calls needed to add resources to an |
204 | | existing slice. In addition, the GENI Desktop needed to be able to initialize and then |
205 | | instrumentize the slice in the background (i.e., while allowing the user to |
206 | | view and use the slice in the GENI Desktop). This required changes to the |
207 | | GENI Desktop to monitor the background initialization/instrumentation |
208 | | process and incrementally enable functionality as it became available. For |
209 | | example, while resources are being allocated the GENI Desktop can only display |
210 | | the topology and the status of the node initialization. As soon as the |
211 | | initilization completes, the file upload, ssh, and run command functionality become available in the user |
212 | | interface. Later when the instrumentation completes, functionality such as |
213 | | displaying basic traffic graphs or archiving measurement data become |
214 | | available in the user interface. In short, users are now taken directly to the |
215 | | GENI Desktop topology view, bypassing several setup steps required by the old |
216 | | user interface. Commonly needed functionality is then automatically added as |
217 | | it becomes available. As part of the new Lite interface, we also simplified |
218 | | the design of the web page(s) used to select a predefined RSPEC. |
219 | | |
220 | | ==== Programming the Desktop: The GENI Desktop CLI (gdcli) ==== |
221 | | |
222 | | The GENI Desktop greatly simplifies the task of instrumenting and monitoring |
223 | | a users' experiment (slice). However, users could only access the GENI |
224 | | Desktop via a web interface. In other words, there was not programmatic way |
225 | | for experimenters or tool developers to leverage the GENI Desktop |
226 | | functionality. |
227 | | |
228 | | To address this need we designed a new interface to the GENI Desktop that |
229 | | could be used to programatically upload files, run commands, download |
230 | | measurement graphs, etc --- functions previously only possible via the GENI |
231 | | Desktop web interface. In particular, we developed an application that runs |
232 | | on Linux (or other Unix-based systems), Mac, and Window called the gdcli |
233 | | program that can be used to interact with the GENI Desktop. The gdcli |
234 | | program can be used to: |
235 | | |
236 | | * Upload files to a select set of nodes |
237 | | * Run a command on a select set of nodes |
238 | | * Download a traffic measurement graph (as PNG or CSV) from a select set of nodes |
239 | | * Download a normal file from a select set of nodes |
240 | | * Get a list of slices |
241 | | * Check the status of a slice |
242 | | * Get the topology of a slice |
243 | | * Validate the setup of a slice |
244 | | * List the nodes in a slice |
245 | | * List the links in a slice |
246 | | |
247 | | The gdcli program can be called from any scripting language (e.g., |
248 | | python, perl, sh (bash), .BAT files, etc). As a result, users are able to |
249 | | write programs in their favorite scripting language that make calls to the |
250 | | GENI Desktop to upload/download files, download measurement graphs, run |
251 | | commands, etc. |
252 | | |
253 | | There were several challenges that we had to address |
254 | | while implementing the gdcli scripting interface. |
255 | | First, we needed a way to make calls to the GENI Desktop server (e.g., to |
256 | | download a traffic graph, or run a command). To solve this problem we |
257 | | enhanced the GENI Desktop server to support HTTP posts that included |
258 | | parameters to the request specifying, for example, the list of traffic graphs to be |
259 | | downloaded (i.e., the nodes/links names and the types of graphs desired). |
260 | | We implemented a python backend server specifically designed to process the |
261 | | request, perform the action, and return the results. |
262 | | The python backend shares access |
263 | | with the previous GENI Desktop PHP code to the databases and files used by the |
264 | | GENI Desktop, thereby ensuring that the results returned by the gdcli are the |
265 | | same information as would be seen in the GENI Desktop web interface. |
266 | | |
267 | | A second challenge was securing the access to, and communication with, the |
268 | | new python backend server. To ensure communication is secure, all communication |
269 | | occurs over a secure connection using https. The problem of authorization |
270 | | requires not only that the user authenticate themselves to the server, but |
271 | | that the server obtain a "speaks-for" certificate to act on behalf of the |
272 | | user. Because the existing speaks-for generation tools are designed for |
273 | | interactive web use, not scripting, we decided to require that users first |
274 | | authorize a speaks-for using the existing GENI Desktop web interface which |
275 | | can then be stored and used by the GENI Desktop (and our new backend server) |
276 | | until the speaks-for expires. However, this does not solve the authorization |
277 | | problem. To ensure the users has the right to issue commands to our python |
278 | | backend server, the web interface of the GENI Desktop also creates a secret |
279 | | key (say at the same time the user authorizes the speaks for) that the user |
280 | | must store on their local machine. The secret key is used when communicating |
281 | | with the python backend server to prove that the user has the right to invoke |
282 | | the requested operations on the GENI Desktop. In that sense, users can think |
283 | | of the gdcli secret key like an ssh key that must be present on their local |
284 | | machine in order to access the service. |
285 | | |
286 | | A third issue involved handling the results/output of a gdcli request. The |
287 | | gdcli tool provide two mechanisms for handling the output from a request. |
288 | | The first, and most simple mechanism, concatenates all the output files/graphs and prints |
289 | | them to standard output, allowing users to redirect output to other programs |
290 | | or tools. The second way gdcli handles output is to deposit each graph, |
291 | | downloaded file, or output from a run command into a different file on the |
292 | | local machine. Files are automatically assigned names that describe their |
293 | | content (based on the slice, the aggregate, the node or link, and the type |
294 | | of graph). Because the naming convention is known to experimenters, they can |
295 | | easily write scripts that know what filenames to look for, and then feed |
296 | | those files to the appropriate program for processing (e.g., copying traffic |
297 | | graphs into a web directory to create a user-defined traffic mashup view). |
298 | | |
299 | | === B. Project participants === |
300 | | |
301 | | The following individuals are involved with the project in one way or another: |
302 | | * Jim Griffioen - Project PI |
303 | | * Zongming Fei - Project Co-PI |
304 | | * Hussamuddin Nasir - Technician/Programmer |
305 | | * Charles Carpenter - Technician/Programmer |
306 | | * Xiongqi Wu - Ph.D. Student |
307 | | * Jeremy Reed - Ph.D. Student |
308 | | |
309 | | === C. Publications (individual and organizational) === |
310 | | |
311 | | === D. Outreach activities === |
312 | | |
313 | | * We gave a presentation about the GENI Desktop and its features during the Introduction to GENI Instrumentation & Measurement Tools portion of the Getting Started with GENI tutorial at GEC 22 and GEC 23. |
314 | | |
315 | | * We gave a demo of the latest GENI Desktop features during the demo session at GEC 21, GEC 22, and GEC 23. |
316 | | |
317 | | * We gave a tutorial entitled "Monitoring and Controlling Experiments with GENI Desktop Scripts and Modules" at the GEC 23 conference. |
318 | | |
319 | | * We developed and posted online-documentation and online-tutorials that describe the new features of the GENI Desktop for users. |
320 | | |
321 | | === E. Collaborations === |
322 | | |
323 | | * Most of our collaborations have been between the GPO Portal team and the aggregate teams at Utah and RENCI. |
324 | | |
325 | | === F. Other Contributions === |
326 | | |