Changes between Initial Version and Version 1 of GeniDesktop/GeniDesktop_final_report

Timestamp:

05/24/17 21:32:47 (7 years ago)

Author:

fei@netlab.uky.edu

Comment:

--

GeniDesktop/GeniDesktop_final_report

@@ +1 @@
+[[PageOutline]]
+
+= GENI Desktop Project Final Report =
+
+== I. Major accomplishments ==
+
+The following highlights our major accomplishments for the GENI Desktop project. 
+
+=== A. Milestones achieved ===
+
+ * Enhanced the functionality of the GENI Desktop and integrated it more tightly with the GENI portal, Flack and OMNI.
+
+ * Modified the GENI Desktop to support "Speaks-for" authentication being developed/supported by the control frameworks and other GENI tools/services.  
+
+ * Incorporated user-driven feedback into the GENI Desktop to support new user-requested services and features.  
+
+ * Developed new training materials that incorporate the changes made to the GENI Desktop. 
+
+ * Integrated and leveraged existing tools and services (e.g., Jacks) into the GENI Desktop for managing topologies, experiments, and results.
+ 
+ * Collected user feedback regarding usability of the GENI Desktop and made major changes to improve its ease-of-use and aesthetics.
+
+ * Adapted the GENI Desktop archiving service for storing and retrieving experiment results and artifacts from iRoDs to support a new archival service with enhanced features.
+
+ * Enhanced the set of scriptable resource management, instrumentation, and monitoring available to experimenters and other tools.
+ 
+ * Enabled integration of the GENI Desktop with other experimenter tools.
+
+ * Improved the usability of the GENI Desktop through several enhancements 
+
+ * Created documentation and tutorial materials to reflect this latest version of the GENI Desktop.
+
+=== B. Deliverables made ===
+
+ * We developed several enhancements to the GENI Desktop -- many supporting what were GENI Portal functions.
+
+ * We enhanced the GENI Desktop's interoperability with OMNI, ExoGENI, InstaGENI, and the iRods service.
+
+ * We enhanced the GENI Desktop to use "Speaks-for" credentials for accessing resources from other GENI components on behalf of users.
+
+ * We implemented the slice verification and configuration testing service.
+
+ * We developed code for super slice support in the GENI Desktop. 
+
+ * We developed a completely new user interface for the GENI Desktop which we call "GENI Desktop Lite" that greatly improves the look-and-feel and ease-of-use of the GENI Desktop. 
+
+ * We integrated the Jacks tool into the GENI Desktop, enabling users to create topologies and instantiate experiments (i.e., slices) using the Jacks tool.  We also integrated the Adopt-A-GENI (AAG) tool into the GENI desktop.
+
+ * We developed and integrated a new archival service into the GENI Desktop that leverages VMs to hold, and later display, the archived experiment state in the same context as it was initially collected and viewed.
+
+ * We designed and implemented a GENI Desktop Command Line Interface (gdcli) that enables users to write scripts that control, manage, and measure the performance of their slices through the GENI Desktop.
+
+ * We demonstrated how the new gdcli can be used by other experimenter tools to integrate with the GENI Desktop.
+
+ * We enhanced several GENI Desktop functions to improve the usability of the GENI Desktop.
+
+ * We developed online documentation for the new gdcli interface and gave a tutorial entitled "Monitoring and Controlling Experiments with GENI Desktop Scripts and Modules" at the GEC 23 conference.
+
+== II. Description of work performed ==
+
+The following provides a description of the major activities and findings for the project. 
+
+=== A. Activities and findings ===
+
+Our activities over the period of this project have resulted in enhanced functionality,
+improved ease-of-use, better authentication/security, a redesigned user
+interface, and the ability to control the GENI Desktop programmatically
+throught scripts.  In particular, we:
+ 
+ * Enhanced the functionality of the GENI Desktop
+
+ * Incorporated support for "Speaks-for" into the GENI Desktop
+
+ * Designed and implemented a slice verification service
+
+ * Designed and implemented a super slice abstraction
+
+ * Developed a new archival services based on Xen VMs to restore entire contexts
+
+ * Redesigned the look-and-feel of the GENI Desktop user interface to make it much easier to use
+
+ * Integrated Jacks and Adopt-A-GENI (AAG) tools into the GENI Desktop
+
+ * Designed and implemented a GENI Desktop Command Line Interface (gdcli) to enable script-based access to GENI Desktop functionality.
+
+The following describes our major activities.  We begin with our efforts to
+enhance the slice management functionality  
+and improve the authentication/authorization used to access the GENI Desktop, and
+then move on to describe new functionality added to the GENI Desktop,
+followed by a description of our efforts to enhance the look-and-feel of the
+GENI Desktop, 
+our efforts to allow scripting of the GENI Desktop, 
+and lastly several enhancements with regard to the usability. 
+
+==== Slice Management Functionality ====
+
+We made significant changes to the look and feel of the slice management functionality
+in order to make it easier for users to interact with, and control, their
+slices.
+
+The first change was to allow the user to determine which slices appear on
+the page.  Because users are often members of multiple projects, the system
+allows them to view only slices belonging to a particular project.  A
+related change that we made was to limit the number of slices
+that are displayed within a project.  Because some users were members of
+projects with many slices (sometimes numbering into the hundreds of slices),
+we now allow users to select which slices from the project will be shown or
+hidden.  This allows users to easily focus on the slices that they are
+working with at the moment.
+
+A second important change was the addition of slice operations that allow
+users to perform slice operations that otherwise would have had to have been
+performed on the GENI portal or in Flack.  In particular, users can now create a
+slice, assign an RSPEC to a slice (more on that later), and allocate the necessary
+resources all from within the GENI Desktop.  When a user is done with a
+slice, the GENI Desktop can be used to delete the resources associated with a
+slice.  These operations are performed through the newly designed slice
+description page.  Each field on the page is "clickable" so that it can be
+viewed in more detail and/or edited.  New fields include an image
+of the topology, an RSPEC field, an aggregate manager field, and an auto-renew
+field.  The slice status field and the next action field have also been
+enhanced to better reflect the slice's current context/status.
+
+Users are now able to assign an RSPEC to a slice and have the GENI Desktop
+instantiate the resources (sliver) specified by the RSPEC.  Users can
+contribute RSPECs to a public RSPEC list (available for anyone to use), or to
+their own private list.  When creating an RSPEC to be (re)used in the future,
+the user can create a thumbnail image of the topology that will be displayed
+along with the slice or while selecting and RSPEC to assign to a slice.  This
+gives users a visual representation of what the topology looks like so that
+the user does not have to mentally parse the textual RSPEC to discover the
+topology.  
+
+
+==== Authorization: Supporting "Speaks-for" ====
+
+The "Speaks-for" credential allows trusted tools to act for, instead of 
+acting as, an experimenter to perform certain actions, such as requesting 
+resources from aggregates, accessing allocated resources, and installing 
+software on experimental nodes. We enhanced the GENI Desktop to use "Speaks-for" 
+credentials for accessing resources on behalf of users. Users no longer 
+need to provide the private key to the GENI Desktop. We implemented an interface
+for the user to authorize the GENI Desktop to speak for her/him.  A GENI
+Desktop-specific certificate is signed using the private key of the user. 
+Because the whole process happens within the browser on the client side,
+the private key never leaves the user's machine. The "Speaks-for" credential 
+allows the GENI Desktop to talk to aggregates and perform all necessary 
+actions on behalf of the user. 
+
+==== New Functionality: Jacks, Archival, Verification and Super Slice Services ====
+
+We integrated the Jacks tool into the GENI Desktop.  Users can now add
+resources to their slices by selecting Jacks in the GENI Desktop which will
+direct them to a GENI Desktop page that embeds the Jacks tool and allows them
+to allocate the resources (i.e., which uses the OMNI tool).  RSPECs
+created by Jacks can be saved by the GENI Desktop for future use.
+
+In addition, we integrated the Adopt-A-GENI (AAG) flow specification module into
+the GENI Desktop, allowing users to visually define OpenFlow paths across the
+topology that are then sent to the AAG module to be instantiated in the
+OpenFlow controller.  Although the AAG functionality is logically a distinct
+service/tool, the messaging system between windows in the GENI Desktop made
+it possible to incorporate this new tool with relatively little effort.  In
+addition, we were able to add a new AAG Controller node type to the Jacks
+wrapper, thereby integrating the AAG controller into the Jacks tool as well.
+
+The existing archival service in the GENI Desktop leveraged the iRoDs storage
+service to store and later retrieve measurement data collected by the GENI
+Desktop.  A key limitation of this service was the inability to easily (and
+quickly) access, view, and make sense of archived measurement data.  To
+address this need, we developed a new archival service that not only archives
+the measurement data, but also archives the software and context used to
+display the data.  Because the data and the environment needed to view the
+data are archived, users can quickly access an archive and view the saved
+data using the same tools available at the time the data was collected.
+
+To support this new archival service, we implemented an archival server that
+not only captures the measurement data stored on the global node (where
+measurement data is collected), but it also captures the state of the drupal
+system used to display the data, including all web server (apache) and
+database (mysql) files.  GENI Desktop users can request that an archive be
+made, which is then sent to the archive server.  When a user visits the
+archive web page on the archive server, they can select from any of the
+archived snapshots.  The archive server will dynamically launch a Xen VM,
+setup the apache, mysql, and Drupal state needed to view the measurement
+data, install the archived measurement data, create login credentials for the
+user, and share the credentials with the GD so the user is automatically logged
+into the archive VM.  The result is that the user is presented with the same
+look-and-feel as if they had gone to the global node at the time the snapshot was taken.
+
+We implemented the slice verification and configuration testing service
+as a module in GENI Desktop by taking advantage of the module builder function 
+of the GENI Desktop. Based on the manifest of an experiment, the verification
+service analyzes the topology and performs tests about the interfaces of all 
+nodes in the experiment. The initial version we implemented checks whether 
+each interface is up and whether it is reachable from a ping test. 
+The results are presented in a table showing the status of all the interfaces of 
+all the nodes in the experiment.
+Later versions of the verification service included
+additional checks (particularly automated bandwidth checks) and also made it
+possible for users to write their own verification scripts to test for things
+of importance to their experiment.  
+
+Building a large experiment is a difficult task in GENI, partly because 
+it is more likely to fail if we create an experiment with a lot of nodes.
+At the same time, we may have multiple related experiments and want to 
+combine these relatively small experiments together to form a large experiment.
+We developed a new "super slice" service in the GENI Desktop to support this functionality.
+Users can use the GENI Desktop to create a super slice by combining multiple 
+existing slices together. The GENI Desktop provides a GUI for users to display
+multiple slices at the same time and pick any pair of nodes from different 
+slices to establish a link between them. The Super Slice service in the GENI
+Desktop currently can then automatically set up GRE tunnels between these selected pairs of nodes 
+from different slices. 
+
+==== A New User Interface: GENI Desktop Lite ====
+
+Over the years the number of features and capabilities offered by the GENI
+Desktop has continued to expand.  Indeed, a key goal of the GENI Desktop was
+to provide users with a context for managing all aspects of their experiment
+from setup and deployment to monitoring and archiving of measurements and
+results.  The downside to this expanded functionality is increased complexity
+using the tool.
+At the same time, the number of experimenters who are using the GENI Desktop to
+create, manage, monitor, and control their slices has been grown rapidly,
+due, in part, to users being exposed to the GENI Desktop as part 
+of GENI tutorials, summer camps, demonstrations and online documents and
+videos.  Feedback from this user group indicated that the extensive
+functionality available in GENI Desktop made it difficult for new users to
+navigate and use.
+
+To address this need for a tool that could be easily learned and used by new
+users, we completely redesigned the look-and-feel of the GENI Desktop to
+reduce complexity and make it simple to create, run, and monitor
+experiments.  Our new "GENI Desktop (GD) Lite" interface is now the default
+interface that users see when they log into the GENI Desktop.
+Users can still access the (original) advanced user interface if needed, but in most
+cases find that the GD Lite interface is sufficient.
+The Lite interface is designed to take users through the lifecycle of an
+experiment.  The Lite interface starts by helping users create a slice,
+assigning resources to the slice, and then giving them access to a simplified
+version of the GENI Desktop topology view where they can log in to nodes, run
+their experiment, monitor basic traffic types, and archive results.  Initial
+feedback on the new interface has been extremely positive.
+
+In addition to a major rewrite of the web code for the user interface, one of the key
+challenges that we had to address was automating the global node setup, initialization,
+and instrumentation.  While these "backend" operations were clearly visible
+in the old user interface, they had to be hidden in the new interface.  This
+meant that the GENI Desktop had to be able to add global nodes into the slice
+(one for each aggregate) on the user's behalf.  This required working with
+the aggregates to support the GENI AM API calls needed to add resources to an
+existing slice.  In addition, the GENI Desktop needed to be able to initialize and then
+instrumentize the slice in the background (i.e., while allowing the user to
+view and use the slice in the GENI Desktop).  This required changes to the
+GENI Desktop to monitor the background initialization/instrumentation
+process and incrementally enable functionality as it became available.  For
+example, while resources are being allocated the GENI Desktop can only display
+the topology and the status of the node initialization.  As soon as the
+initilization completes, the file upload, ssh, and run command functionality become available in the user
+interface.  Later when the instrumentation completes, functionality such as
+displaying basic traffic graphs or archiving measurement data become
+available in the user interface.  In short, users are now taken directly to the
+GENI Desktop topology view, bypassing several setup steps required by the old
+user interface.  Commonly needed functionality is then automatically added as
+it becomes available.   As part of the new Lite interface, we also simplified
+the design of the web page(s) used to select a predefined RSPEC.
+
+==== Programming the Desktop:  The GENI Desktop CLI (gdcli) ====
+
+The GENI Desktop greatly simplifies the task of instrumenting and monitoring
+a users' experiment (slice).   However, users could only access the GENI
+Desktop via a web interface.   In other words, there was not programmatic way
+for experimenters or tool developers to leverage the GENI Desktop
+functionality.
+
+To address this need we designed a new interface to the GENI Desktop that
+could be used to programatically upload files, run commands, download
+measurement graphs, etc --- functions previously only possible via the GENI
+Desktop web interface.  In particular, we developed an application that runs
+on Linux (or other Unix-based systems), Mac, and Window called the gdcli
+program that can be used to interact with the GENI Desktop.  The gdcli
+program can be used to:
+
+ * Upload files to a select set of nodes
+ * Run a command on a select set of nodes
+ * Download a traffic measurement graph (as PNG or CSV) from a select set of nodes
+ * Download a normal file from a select set of nodes
+ * Get a list of slices
+ * Check the status of a slice
+ * Get the topology of a slice
+ * Validate the setup of a slice
+ * List the nodes in a slice
+ * List the links in a slice
+
+The gdcli program can be called from any scripting language (e.g., 
+python, perl, sh (bash), .BAT files, etc).  As a result, users are able to
+write programs in their favorite scripting language that make calls to the
+GENI Desktop to upload/download files, download measurement graphs, run
+commands, etc.
+
+There were several challenges that we had to address
+while implementing the gdcli scripting interface.
+First, we needed a way to make calls to the GENI Desktop server (e.g., to
+download a traffic graph, or run a command).  To solve this problem we
+enhanced the GENI Desktop server to support HTTP posts that included
+parameters to the request specifying, for example, the list of traffic graphs to be
+downloaded (i.e., the nodes/links names and the types of graphs desired).
+We implemented a python backend server specifically designed to process the
+request, perform the action, and return the results.
+The python backend shares access
+with the previous GENI Desktop PHP code to the databases and files used by the
+GENI Desktop, thereby ensuring that the results returned by the gdcli are the
+same information as would be seen in the GENI Desktop web interface.
+
+A second challenge was securing the access to, and communication with, the
+new python backend server.  To ensure communication is secure, all communication
+occurs over a secure connection using https.  The problem of authorization
+requires not only that the user authenticate themselves to the server, but
+that the server obtain a "speaks-for" certificate to act on behalf of the
+user.   Because the existing speaks-for generation tools are designed for
+interactive web use, not scripting, we decided to require that users first
+authorize a speaks-for using the existing GENI Desktop web interface which
+can then be stored and used by the GENI Desktop (and our new backend server)
+until the speaks-for expires.  However, this does not solve the authorization
+problem. To ensure the users has the right to issue commands to our python
+backend server, the web interface of the GENI Desktop also creates a secret
+key (say at the same time the user authorizes the speaks for) that the user
+must store on their local machine.  The secret key is used when communicating
+with the python backend server to prove that the user has the right to invoke
+the requested operations on the GENI Desktop.  In that sense, users can think
+of the gdcli secret key like an ssh key that must be present on their local
+machine in order to access the service.
+
+A third issue involved handling the results/output of a gdcli request.  The
+gdcli tool provide two mechanisms for handling the output from a request.
+The first, and most simple mechanism, concatenates all the output files/graphs and prints
+them to standard output, allowing users to redirect output to other programs
+or tools. The second way gdcli handles output is to deposit each graph,
+downloaded file, or output from a run command into a different file on the
+local machine.  Files are automatically assigned names that describe their
+content (based on the slice, the aggregate, the node or link, and the type
+of graph).  Because the naming convention is known to experimenters, they can
+easily write scripts that know what filenames to look for, and then feed
+those files to the appropriate program for processing (e.g., copying traffic
+graphs into a web directory to create a user-defined traffic mashup view).
+
+==== Improving the Usability: Continued Improvement ====
+
+We made continued improvement on the usability of the GENI Desktop. 
+
+1) We refactored the GENI Desktop code to make it more robust and extensible.
+To make the GENI Desktop more robust and extensible (i.e., easier for users
+or third parties to add new functionality), we redesigned, streamlined,
+simplified, and consolidated large portions of both the graphical user
+interface and the backend services/database that support the GENI Desktop.
+In particular, we overhauled the GENI Desktop interface based on HTML5,
+removing dependencies on Flash code that were present in the original
+implementation.  We also streamlined and simplified much of the backend
+processing, making it faster and more secure/robust to failures.
+
+2) We implemented a new way to view the topology based on JACKS
+To be consistent with the JACKs interface used to create GENI topologies, we
+added JACKs to the GENI Desktop as a way to view the logical topology.  To
+allow users to interact with the topology, we enhanced JACKs to send messages
+to other GENI Desktop components in response to user interactions with the
+topology (e.g., selecting a node).  This involved working with the Utah team
+to make the GENI Desktop messaging system accept messages from JACKS.
+
+3) We developed new support for GENI Desktop account management functions
+New account management functions allow users to set the default topology
+view, the default modules and other behavior for the account. This simplified
+users interaction with their slices.
+
+4) We implemented enhanced session management functions for the GENI Desktop.
+To make the GENI Desktop more secure, we implemented new session management
+functionality that improves security by ensuring the user credential is
+stored and is valid only for the current GENI Desktop session.  Each new session
+requires that new credential information be established with the GENI Desktop
+server, but the (re) establishment of this information is largely hidden from
+the user.
+
+5) We improved the GENI Desktop archive service.
+The GENI Desktop allows users to archive measurement data to the GENI
+iRods server for future use. To recreate the experience of viewing traffic
+data in the GENI Desktop, data and traffic graphs can be archived upon
+request from the user.  Later, a user can retrieve the archived data using
+a Vagrant-based VM provided by the GENI Desktop which downloads the archive and
+displays data and graphs in an environment that has the same look and feel as
+the GENI Desktop environment that originally displayed the data.
+
+
+=== B. Project participants ===
+
+The following individuals are involved with the project in one way or another:
+ * Jim Griffioen - Project PI 
+ * Zongming Fei - Project Co-PI 
+ * Hussamuddin Nasir - Technician/Programmer
+ * Charles Carpenter - Technician/Programmer
+ * Xiongqi Wu - Ph.D. Student
+ * Jeremy Reed - Ph.D. Student
+
+=== C. Publications (individual and organizational) ===
+
+=== D. Outreach activities ===
+
+ * We gave a presentation about the GENI Desktop and its features during the Introduction to GENI Instrumentation & Measurement Tools portion of the Getting Started with GENI tutorial at GEC19, GEC 22 and GEC 23.
+ 
+ * We gave a demo of the latest GENI Desktop features during the demo session at GEC 19, GEC 20, GEC 21, GEC 22, and GEC 23.
+ 
+ * We gave a tutorial entitled "Monitoring and Controlling Experiments with GENI Desktop Scripts and Modules" at the GEC 23 conference.
+ 
+ * We developed and posted online-documentation and online-tutorials that describe the new features of the GENI Desktop for users.
+ 
+=== E. Collaborations ===
+
+ * Most of our collaborations have been between the GPO Portal team and the aggregate teams at Utah and RENCI.
+
+=== F. Other Contributions ===