[[PageOutline]] = GENI Desktop Project Final Report = == I. Major accomplishments == The following highlights our major accomplishments for the GENI Desktop project. === A. Milestones achieved === * Enhanced the functionality of the GENI Desktop and integrated it more tightly with the GENI portal, Flack and OMNI. * Modified the GENI Desktop to support "Speaks-for" authentication being developed/supported by the control frameworks and other GENI tools/services. * Incorporated user-driven feedback into the GENI Desktop to support new user-requested services and features. * Developed new training materials that incorporate the changes made to the GENI Desktop. * Integrated and leveraged existing tools and services (e.g., Jacks) into the GENI Desktop for managing topologies, experiments, and results. * Collected user feedback regarding usability of the GENI Desktop and made major changes to improve its ease-of-use and aesthetics. * Adapted the GENI Desktop archiving service for storing and retrieving experiment results and artifacts from iRoDs to support a new archival service with enhanced features. * Enhanced the set of scriptable resource management, instrumentation, and monitoring available to experimenters and other tools. * Enabled integration of the GENI Desktop with other experimenter tools. * Improved the usability of the GENI Desktop through several enhancements * Created documentation and tutorial materials to reflect this latest version of the GENI Desktop. === B. Deliverables made === * We developed several enhancements to the GENI Desktop -- many supporting what were GENI Portal functions. * We enhanced the GENI Desktop's interoperability with OMNI, ExoGENI, InstaGENI, and the iRods service. * We enhanced the GENI Desktop to use "Speaks-for" credentials for accessing resources from other GENI components on behalf of users. * We implemented the slice verification and configuration testing service. * We developed code for super slice support in the GENI Desktop. * We developed a completely new user interface for the GENI Desktop which we call "GENI Desktop Lite" that greatly improves the look-and-feel and ease-of-use of the GENI Desktop. * We integrated the Jacks tool into the GENI Desktop, enabling users to create topologies and instantiate experiments (i.e., slices) using the Jacks tool. We also integrated the Adopt-A-GENI (AAG) tool into the GENI desktop. * We developed and integrated a new archival service into the GENI Desktop that leverages VMs to hold, and later display, the archived experiment state in the same context as it was initially collected and viewed. * We designed and implemented a GENI Desktop Command Line Interface (gdcli) that enables users to write scripts that control, manage, and measure the performance of their slices through the GENI Desktop. * We demonstrated how the new gdcli can be used by other experimenter tools to integrate with the GENI Desktop. * We enhanced several GENI Desktop functions to improve the usability of the GENI Desktop. * We developed online documentation for the new gdcli interface and gave a tutorial entitled "Monitoring and Controlling Experiments with GENI Desktop Scripts and Modules" at the GEC 23 conference. == II. Description of work performed == The following provides a description of the major activities and findings for the project. === A. Activities and findings === Our activities over the period of this project have resulted in enhanced functionality, improved ease-of-use, better authentication/security, a redesigned user interface, and the ability to control the GENI Desktop programmatically throught scripts. In particular, we: * Enhanced the functionality of the GENI Desktop * Incorporated support for "Speaks-for" into the GENI Desktop * Designed and implemented a slice verification service * Designed and implemented a super slice abstraction * Developed a new archival services based on Xen VMs to restore entire contexts * Redesigned the look-and-feel of the GENI Desktop user interface to make it much easier to use * Integrated Jacks and Adopt-A-GENI (AAG) tools into the GENI Desktop * Designed and implemented a GENI Desktop Command Line Interface (gdcli) to enable script-based access to GENI Desktop functionality. The following describes our major activities. We begin with our efforts to enhance the slice management functionality and improve the authentication/authorization used to access the GENI Desktop, and then move on to describe new functionality added to the GENI Desktop, followed by a description of our efforts to enhance the look-and-feel of the GENI Desktop, our efforts to allow scripting of the GENI Desktop, and lastly several enhancements with regard to the usability. ==== Slice Management Functionality ==== We made significant changes to the look and feel of the slice management functionality in order to make it easier for users to interact with, and control, their slices. The first change was to allow the user to determine which slices appear on the page. Because users are often members of multiple projects, the system allows them to view only slices belonging to a particular project. A related change that we made was to limit the number of slices that are displayed within a project. Because some users were members of projects with many slices (sometimes numbering into the hundreds of slices), we now allow users to select which slices from the project will be shown or hidden. This allows users to easily focus on the slices that they are working with at the moment. A second important change was the addition of slice operations that allow users to perform slice operations that otherwise would have had to have been performed on the GENI portal or in Flack. In particular, users can now create a slice, assign an RSPEC to a slice (more on that later), and allocate the necessary resources all from within the GENI Desktop. When a user is done with a slice, the GENI Desktop can be used to delete the resources associated with a slice. These operations are performed through the newly designed slice description page. Each field on the page is "clickable" so that it can be viewed in more detail and/or edited. New fields include an image of the topology, an RSPEC field, an aggregate manager field, and an auto-renew field. The slice status field and the next action field have also been enhanced to better reflect the slice's current context/status. Users are now able to assign an RSPEC to a slice and have the GENI Desktop instantiate the resources (sliver) specified by the RSPEC. Users can contribute RSPECs to a public RSPEC list (available for anyone to use), or to their own private list. When creating an RSPEC to be (re)used in the future, the user can create a thumbnail image of the topology that will be displayed along with the slice or while selecting and RSPEC to assign to a slice. This gives users a visual representation of what the topology looks like so that the user does not have to mentally parse the textual RSPEC to discover the topology. ==== Authorization: Supporting "Speaks-for" ==== The "Speaks-for" credential allows trusted tools to act for, instead of acting as, an experimenter to perform certain actions, such as requesting resources from aggregates, accessing allocated resources, and installing software on experimental nodes. We enhanced the GENI Desktop to use "Speaks-for" credentials for accessing resources on behalf of users. Users no longer need to provide the private key to the GENI Desktop. We implemented an interface for the user to authorize the GENI Desktop to speak for her/him. A GENI Desktop-specific certificate is signed using the private key of the user. Because the whole process happens within the browser on the client side, the private key never leaves the user's machine. The "Speaks-for" credential allows the GENI Desktop to talk to aggregates and perform all necessary actions on behalf of the user. ==== New Functionality: Jacks, Archival, Verification and Super Slice Services ==== We integrated the Jacks tool into the GENI Desktop. Users can now add resources to their slices by selecting Jacks in the GENI Desktop which will direct them to a GENI Desktop page that embeds the Jacks tool and allows them to allocate the resources (i.e., which uses the OMNI tool). RSPECs created by Jacks can be saved by the GENI Desktop for future use. In addition, we integrated the Adopt-A-GENI (AAG) flow specification module into the GENI Desktop, allowing users to visually define OpenFlow paths across the topology that are then sent to the AAG module to be instantiated in the OpenFlow controller. Although the AAG functionality is logically a distinct service/tool, the messaging system between windows in the GENI Desktop made it possible to incorporate this new tool with relatively little effort. In addition, we were able to add a new AAG Controller node type to the Jacks wrapper, thereby integrating the AAG controller into the Jacks tool as well. The existing archival service in the GENI Desktop leveraged the iRoDs storage service to store and later retrieve measurement data collected by the GENI Desktop. A key limitation of this service was the inability to easily (and quickly) access, view, and make sense of archived measurement data. To address this need, we developed a new archival service that not only archives the measurement data, but also archives the software and context used to display the data. Because the data and the environment needed to view the data are archived, users can quickly access an archive and view the saved data using the same tools available at the time the data was collected. To support this new archival service, we implemented an archival server that not only captures the measurement data stored on the global node (where measurement data is collected), but it also captures the state of the drupal system used to display the data, including all web server (apache) and database (mysql) files. GENI Desktop users can request that an archive be made, which is then sent to the archive server. When a user visits the archive web page on the archive server, they can select from any of the archived snapshots. The archive server will dynamically launch a Xen VM, setup the apache, mysql, and Drupal state needed to view the measurement data, install the archived measurement data, create login credentials for the user, and share the credentials with the GD so the user is automatically logged into the archive VM. The result is that the user is presented with the same look-and-feel as if they had gone to the global node at the time the snapshot was taken. We implemented the slice verification and configuration testing service as a module in GENI Desktop by taking advantage of the module builder function of the GENI Desktop. Based on the manifest of an experiment, the verification service analyzes the topology and performs tests about the interfaces of all nodes in the experiment. The initial version we implemented checks whether each interface is up and whether it is reachable from a ping test. The results are presented in a table showing the status of all the interfaces of all the nodes in the experiment. Later versions of the verification service included additional checks (particularly automated bandwidth checks) and also made it possible for users to write their own verification scripts to test for things of importance to their experiment. Building a large experiment is a difficult task in GENI, partly because it is more likely to fail if we create an experiment with a lot of nodes. At the same time, we may have multiple related experiments and want to combine these relatively small experiments together to form a large experiment. We developed a new "super slice" service in the GENI Desktop to support this functionality. Users can use the GENI Desktop to create a super slice by combining multiple existing slices together. The GENI Desktop provides a GUI for users to display multiple slices at the same time and pick any pair of nodes from different slices to establish a link between them. The Super Slice service in the GENI Desktop currently can then automatically set up GRE tunnels between these selected pairs of nodes from different slices. ==== A New User Interface: GENI Desktop Lite ==== Over the years the number of features and capabilities offered by the GENI Desktop has continued to expand. Indeed, a key goal of the GENI Desktop was to provide users with a context for managing all aspects of their experiment from setup and deployment to monitoring and archiving of measurements and results. The downside to this expanded functionality is increased complexity using the tool. At the same time, the number of experimenters who are using the GENI Desktop to create, manage, monitor, and control their slices has been grown rapidly, due, in part, to users being exposed to the GENI Desktop as part of GENI tutorials, summer camps, demonstrations and online documents and videos. Feedback from this user group indicated that the extensive functionality available in GENI Desktop made it difficult for new users to navigate and use. To address this need for a tool that could be easily learned and used by new users, we completely redesigned the look-and-feel of the GENI Desktop to reduce complexity and make it simple to create, run, and monitor experiments. Our new "GENI Desktop (GD) Lite" interface is now the default interface that users see when they log into the GENI Desktop. Users can still access the (original) advanced user interface if needed, but in most cases find that the GD Lite interface is sufficient. The Lite interface is designed to take users through the lifecycle of an experiment. The Lite interface starts by helping users create a slice, assigning resources to the slice, and then giving them access to a simplified version of the GENI Desktop topology view where they can log in to nodes, run their experiment, monitor basic traffic types, and archive results. Initial feedback on the new interface has been extremely positive. In addition to a major rewrite of the web code for the user interface, one of the key challenges that we had to address was automating the global node setup, initialization, and instrumentation. While these "backend" operations were clearly visible in the old user interface, they had to be hidden in the new interface. This meant that the GENI Desktop had to be able to add global nodes into the slice (one for each aggregate) on the user's behalf. This required working with the aggregates to support the GENI AM API calls needed to add resources to an existing slice. In addition, the GENI Desktop needed to be able to initialize and then instrumentize the slice in the background (i.e., while allowing the user to view and use the slice in the GENI Desktop). This required changes to the GENI Desktop to monitor the background initialization/instrumentation process and incrementally enable functionality as it became available. For example, while resources are being allocated the GENI Desktop can only display the topology and the status of the node initialization. As soon as the initilization completes, the file upload, ssh, and run command functionality become available in the user interface. Later when the instrumentation completes, functionality such as displaying basic traffic graphs or archiving measurement data become available in the user interface. In short, users are now taken directly to the GENI Desktop topology view, bypassing several setup steps required by the old user interface. Commonly needed functionality is then automatically added as it becomes available. As part of the new Lite interface, we also simplified the design of the web page(s) used to select a predefined RSPEC. ==== Programming the Desktop: The GENI Desktop CLI (gdcli) ==== The GENI Desktop greatly simplifies the task of instrumenting and monitoring a users' experiment (slice). However, users could only access the GENI Desktop via a web interface. In other words, there was not programmatic way for experimenters or tool developers to leverage the GENI Desktop functionality. To address this need we designed a new interface to the GENI Desktop that could be used to programatically upload files, run commands, download measurement graphs, etc --- functions previously only possible via the GENI Desktop web interface. In particular, we developed an application that runs on Linux (or other Unix-based systems), Mac, and Window called the gdcli program that can be used to interact with the GENI Desktop. The gdcli program can be used to: * Upload files to a select set of nodes * Run a command on a select set of nodes * Download a traffic measurement graph (as PNG or CSV) from a select set of nodes * Download a normal file from a select set of nodes * Get a list of slices * Check the status of a slice * Get the topology of a slice * Validate the setup of a slice * List the nodes in a slice * List the links in a slice The gdcli program can be called from any scripting language (e.g., python, perl, sh (bash), .BAT files, etc). As a result, users are able to write programs in their favorite scripting language that make calls to the GENI Desktop to upload/download files, download measurement graphs, run commands, etc. There were several challenges that we had to address while implementing the gdcli scripting interface. First, we needed a way to make calls to the GENI Desktop server (e.g., to download a traffic graph, or run a command). To solve this problem we enhanced the GENI Desktop server to support HTTP posts that included parameters to the request specifying, for example, the list of traffic graphs to be downloaded (i.e., the nodes/links names and the types of graphs desired). We implemented a python backend server specifically designed to process the request, perform the action, and return the results. The python backend shares access with the previous GENI Desktop PHP code to the databases and files used by the GENI Desktop, thereby ensuring that the results returned by the gdcli are the same information as would be seen in the GENI Desktop web interface. A second challenge was securing the access to, and communication with, the new python backend server. To ensure communication is secure, all communication occurs over a secure connection using https. The problem of authorization requires not only that the user authenticate themselves to the server, but that the server obtain a "speaks-for" certificate to act on behalf of the user. Because the existing speaks-for generation tools are designed for interactive web use, not scripting, we decided to require that users first authorize a speaks-for using the existing GENI Desktop web interface which can then be stored and used by the GENI Desktop (and our new backend server) until the speaks-for expires. However, this does not solve the authorization problem. To ensure the users has the right to issue commands to our python backend server, the web interface of the GENI Desktop also creates a secret key (say at the same time the user authorizes the speaks for) that the user must store on their local machine. The secret key is used when communicating with the python backend server to prove that the user has the right to invoke the requested operations on the GENI Desktop. In that sense, users can think of the gdcli secret key like an ssh key that must be present on their local machine in order to access the service. A third issue involved handling the results/output of a gdcli request. The gdcli tool provide two mechanisms for handling the output from a request. The first, and most simple mechanism, concatenates all the output files/graphs and prints them to standard output, allowing users to redirect output to other programs or tools. The second way gdcli handles output is to deposit each graph, downloaded file, or output from a run command into a different file on the local machine. Files are automatically assigned names that describe their content (based on the slice, the aggregate, the node or link, and the type of graph). Because the naming convention is known to experimenters, they can easily write scripts that know what filenames to look for, and then feed those files to the appropriate program for processing (e.g., copying traffic graphs into a web directory to create a user-defined traffic mashup view). ==== Improving the Usability: Continued Improvement ==== We made continued improvement on the usability of the GENI Desktop. 1) We refactored the GENI Desktop code to make it more robust and extensible. To make the GENI Desktop more robust and extensible (i.e., easier for users or third parties to add new functionality), we redesigned, streamlined, simplified, and consolidated large portions of both the graphical user interface and the backend services/database that support the GENI Desktop. In particular, we overhauled the GENI Desktop interface based on HTML5, removing dependencies on Flash code that were present in the original implementation. We also streamlined and simplified much of the backend processing, making it faster and more secure/robust to failures. 2) We implemented a new way to view the topology based on JACKS To be consistent with the JACKs interface used to create GENI topologies, we added JACKs to the GENI Desktop as a way to view the logical topology. To allow users to interact with the topology, we enhanced JACKs to send messages to other GENI Desktop components in response to user interactions with the topology (e.g., selecting a node). This involved working with the Utah team to make the GENI Desktop messaging system accept messages from JACKS. 3) We developed new support for GENI Desktop account management functions New account management functions allow users to set the default topology view, the default modules and other behavior for the account. This simplified users interaction with their slices. 4) We implemented enhanced session management functions for the GENI Desktop. To make the GENI Desktop more secure, we implemented new session management functionality that improves security by ensuring the user credential is stored and is valid only for the current GENI Desktop session. Each new session requires that new credential information be established with the GENI Desktop server, but the (re) establishment of this information is largely hidden from the user. 5) We improved the GENI Desktop archive service. The GENI Desktop allows users to archive measurement data to the GENI iRods server for future use. To recreate the experience of viewing traffic data in the GENI Desktop, data and traffic graphs can be archived upon request from the user. Later, a user can retrieve the archived data using a Vagrant-based VM provided by the GENI Desktop which downloads the archive and displays data and graphs in an environment that has the same look and feel as the GENI Desktop environment that originally displayed the data. === B. Project participants === The following individuals are involved with the project in one way or another: * Jim Griffioen - Project PI * Zongming Fei - Project Co-PI * Hussamuddin Nasir - Technician/Programmer * Charles Carpenter - Technician/Programmer * Xiongqi Wu - Ph.D. Student * Jeremy Reed - Ph.D. Student === C. Publications (individual and organizational) === === D. Outreach activities === * We gave a presentation about the GENI Desktop and its features during the Introduction to GENI Instrumentation & Measurement Tools portion of the Getting Started with GENI tutorial at GEC19, GEC 22 and GEC 23. * We gave a demo of the latest GENI Desktop features during the demo session at GEC 19, GEC 20, GEC 21, GEC 22, and GEC 23. * We gave a tutorial entitled "Monitoring and Controlling Experiments with GENI Desktop Scripts and Modules" at the GEC 23 conference. * We developed and posted online-documentation and online-tutorials that describe the new features of the GENI Desktop for users. === E. Collaborations === * Most of our collaborations have been between the GPO Portal team and the aggregate teams at Utah and RENCI. === F. Other Contributions ===