wiki:InstrumentationTools-Nov2011-status

Version 1 (modified by Vic Thomas, 13 years ago) (diff)

--

INSTOOLS Project Status Report

Period: 6/30/10 - 11/15/10

I. Major accomplishments

The following highlights our accomplishments during the last reporting period.

A. Milestones achieved

We continued to make progress toward our milestones. Some of our accomplishments include:

  • continuing to support operation of the Kentucky ProtoGENI aggregate, enabling basic experimentation through the ProtoGENI clearinghouse.

  • updating the instrumentation software to support virtual nodes and provide a VNC interface to experiment nodes.

B. Deliverables made

  • We contributed the latest version of our code to the ProtoGENI GIT repository.

II. Description of work performed during last quarter

The following provides a description of our activities and results during the last reporting period.

A. Activities and findings

During this period of time we made significant progress on several fronts. We enhanced our software to support instrumentation of virtual nodes and improved the web interface to provide X window access to experimental nodes through a VNC-based interface.

As more experiments start to use the ProtoGENI facilities, making efficient use of the resources has become one of our priorities. To that end, we worked with the Utah group to add support for virtual nodes based on OpenVZ so that each physical PC can be used as multiple experimental nodes. Adding support for virtual nodes turned out to be fraught with problems, and so we had to go through several rounds of bug fixes and upgrades to get the latest development version of the code working. In addition we had to manually fix the OPENVZ image for the shared nodes because the change had not been pushed out to the image running at Utah.

Unlike BSD jails, we found that the OpenVZ image supports the abilty to run independent SNMP daemons on each OpenVZ node (i.e., one SNMP daemon per VM). This feature allows us to perform per-sliver monitoring without the need to write new monitoring software to understand the OpenVZ virtual interfaces.

To conserve public IP address space, the ProtoGENI implementation of virtual nodes does not give public IP addresses to virtual nodes. While the IP addresses assigned to a virtual node can be used within the experimental network created by the user (i.e., the slice), accessing the node from the normal Internet is a problem. To address this problem, the ProtoGENI team added the ability to reach an ssh server via port forwarding to the OpenVZ VM nodes. This meant we had to modify our code to extract the services information from the manifest and then use the login information to access the virtual nodes. In particular, our INSTOOLS software queries the manifest for the sshd port and hostname, and uses that information to login into the VMs. We have to modify all ssh calls being made in both the client side and server side code to work using port forwarding.

We also found that the Measurement Controller (MC) could be virtualized. We applied the same technique that we used on the experimental nodes in order to gain ssh access to the MC. However, we encountered the same problem of accessing the web server on the MC, because port forwarding is only setup for ssh access, not for web access. To address this we currently require that the user set up their own ssh tunneling, but we are talking with Utah about ways to automate this in the future. As a result of this limitation, we currently prefer non-shared nodes for the MC because users can easily access the web services there without the need to tunnel.

We gave a hands-on tutorial as part of the ProtoGENI tutorial at GEC9. The hands-on tutorial allowed experimenters to create ProtoGENI slices and then instrumentize their slice using the INSTOOLS software. To ensure we had enough resources available for the slices of all the attendees, we utilized the virtual node support, allocating virtual nodes for the users' topologies. The experience pointed out a few areas that we could make better, but overall most attendees were able to instrumentize their slice and monitor its behavior. Incidently, after the GEC9 conference we have noticed that some of the attendees are starting to use our system for their experiments.

Another major addition to our software this period was the ability to access X window software running on the experimental nodes (slivers) via the MC web interace. Our goal was to leverage existing network monitoring tools, such as Wireshark and EtherApe, in order to observe the behavior of experimental nodes. These tools are helpful to collect node statistics and visualize the link traffic. However, they need X window support. To support such access, we added the ability to dynamically load X-window software onto the experimental nodes and then provide indirect access through the MC Web browser and the VNC protocol. It has been added to the the MC's Drupal content management system as one of the menu options. The only way for a user to access the programs running on the experimental nodes is to first login to the Drupal CMS running on the MC. This way, we can assure the slice owner that they are the only ones allowed to access/run these programs. Our Drupal interface currently has two VNC templates that are preconfigured to run xterm and wireshark respectively on the nodes in the slice via VNC. The MC runs a JAVA-based VNC client (in the CMS) that mirrors the VNC connections from the nodes. The VNC communication between the nodes and the MC are protected by a system generated random password that is unique for every slice and invisible to the user.

We gave an overview of the INSTOOLS software at the GEC9 Instrumentation and Measurement Workshop in Washington D.C. We demonstrated the major features of the INSTOOLS software, including the ability to access X-based software like wireshark doing packet-level capture on experimental nodes.

B. Project participants

The following individuals have helped with the project in one way or another during the last quarter:

  • Jim Griffioen - Project PI
  • Zongming Fei - Project Co-PI
  • Hussamuddin Nasir - Technician/programmer
  • Xiongqi Wu - Research Assistant
  • Jeremy Reed - Research Assistant (half time)
  • Lowell Pike - Network administrator
  • Woody Marvel - Assists in Emulab administration

C. Publications (individual and organizational)

D. Outreach activities

  • Gave an overview of our INSTOOLS software at the GEC9 Instrumentation and Measurement Workshop.
  • Gave a hands-on tutorial as part of the ProtoGENI tutorial at GEC9.

E. Collaborations

Most of our collaborations continue to be with the Utah ProtoGENI team, and we continue to be actively involved in the bi-weekly meetings of the ProtoGENI cluster.

F. Other Contributions