Version 1 (modified by, 13 years ago) (diff)


Internet Scale Overlay Hosting Progress Report 4/1/2009-6/30/2009)

1. Project Activities this Quarter

Public Demonstration at GEC 4

We carried out our first public demonstration of the SPPs at GEC 4. This demonstration involved two fully configured SPP nodes and demonstrated all the control mechanisms needed to run an application, including (1) downloading slice definitions from PLC and instantiating slices on SPP nodes, (2) user login to a slice's vServer and establishment of connections from the vServer to a remote machine to download files, (3) configuration of external ports that remote machines can use to connect to programs running within the slice's vServer, (4) configuration of a fastpath on the NPE, including configuration of logical interfaces, routes, filters, queues, buffers and network bandwidth, (5) running of application traffic from remote machines through an overlay network consisting of three logical nodes on two different SPPs and (6) configuration and operation of performance monitoring software, demonstrating real-time remote display.

The demo went smoothly and received a steady stream of visitors throughout the demo period. The cramped space and high noise level in the meeting room made for less than ideal conditions, but those who were able to observe the demo had a good opportunity to see the systems in action, and get at least a basic understanding of how they could be used.

Completion of Initial Version of Reservation Software

We have completed a first version of the reservation system that will allow users to reserve SPP resources in advance. We have designed and implemented the complete API for the reservation system. This includes new methods implemented by the Resource Manager Proxy (RMP) to accept reservation requests from users, and corresponding methods on the System Resource Manager (SRM), which accepts requests from all GPEs and makes system-wide resource allocation decisions. This API has been fully impelmented and tested, and the reservations are managed by the SRM and periodically stored to disk, to ensure that reservations are not lost as a result of a system crash. When a slice makes an actual resource allocation request, the SRM checks for a valid reservation before accepting the request. A command line utility has also been provided to read a reservation request from a file and invoke the API to reserve the requested resources.

There are two elements of the reservation system that have yet to be completed. First, we have not yet implemented the logic to withdraw resources from a slice when its reservation comes to an end. Second, we do not yet support the advance reservation of external port numbers. We expect both of these elements to be completed after the systems are deployed, but before they are made available to a larger user population.

Preparation for Deployment

Our largest single activity this quarter has been preparing for deployment of the SPP nodes. While we are a little behind schedule, we are making good progress and expect to be able to ship systems to Indiana University by 7/15. That should allow them to complete the installation of at least the first system by mid August.

The preparation has involved two major elements. First, is just the physical preparation of all the system components. While we have had the systems up and running for some time in a development setting, that setting is a relatively forgiving one, allowing us to reconfigure elements on the fly as needed. Once the systems are deployed, this will no longer be the case, so it's important for us to make sure that everything is setup to support remote configuration of anything that may need attention. Part of this process has been dealing with various system initialization issues arising from the ATCA equipment we are using. These are detailed further under bug fixes. We are in the final stages of preparing detailed installation instructions, including photographs and video clips of key steps. We have coordinated with staff at Indiana University so that we can understand their needs and to communicate our requirements for the installation. One issue that will require close attention is the specification and configuration of MAC addresses at both ends of the VLAN links used to connect the SPP nodes to the I2 routers. Because the SPP does not support ARP on these interfaces, the MAC addresses will need to be manually configured. This is straightforward to do, but because it is unusual to configure MACs manually, it is important that all involved pay close attention.

In addition to the preparation of the physical equipment, we have devoted a lot of attention to fully automating the system startup process, to minimize the need for manual intervention. The SPP nodes contain a number of independent physical and software subsystems that must be initialized when the system starts up or reboots. Some of these initialization steps must be carefully sequenced to ensure that the entire system comes up cleanly and reliably. During our testing activities, we could afford to be a little more relaxed about such issues, but since we will have no physical access to the components, once they are deployed, we want to minimize the opportunities for things to go wrong during startup. We expect the steps we have taken will go a long way towards ensuring consistent and reliable operation of the SPP nodes.

System Architecture Document

One of our near-term deliverables is a document describing the system architecture of the SPP nodes in detail. Work on this document is underway, and we expect it will be completed by the deadline of 7/15.

User Documentation

We have begun work on a wiki that will ultimately provide complete user documentation on how to use SPP nodes to setup and operate experimental networks. As a first step, the wiki documents the procedures needed to setup and run the demo that was done at GEC 4. We expect to have a preliminary version of the wiki available to early users by the end of the first contract year.


In the short term, the SPP nodes will be managed through per-node resource reservation and allocation mechanisms. However, we are planning to implement a form of Rspec, and have worked out many of the details of how resources will be specified using Rspecs. To make this useful, the PLC will need to take a network-wide Rspec reservation and partition it into per-node Rspecs that can be propagated to the individual SPPs. This will allow users to reserve a consistent collection of resources across all SPPs in one step, rather than requiring that they reserve resources on individual SPPs.

Flow Monitoring

We are close to completion of the flow monitoring subsystem needed to keep track of outgoing Internet traffic. This subsystem will allow us to identify and disable experiments running on the SPPs that send unwanted traffic that triggers complaints. This involves data plane software in the egress side of the Line Card, data collection software in the Line Card's xScale management processor, and data aggregation and archiving software running in the CP. Once this is complete, it will be possible to access the outgoing flow statistics through standard PlanetLab management mechanisms.

Development of Second Version of Network Processor Datapath Software

We have a continuing effort to develop a more complete version of the datapath software for the Network Processor Engine (NPE) used to implement slice fastpaths. This new version is needed to support full 10 Gb/s operation and multicast packet delivery. We have not been able to devote a lot of resources to this effort so far, but expect to give it more attention after the initial deployment is complete.

Preliminary Work on New Fast Path Code Option

We have begun efforts to implement a new code option for slice fastpaths, that provides a simple VLAN-like service. We expect this to be directly useful for some applications, and that it can be a helpful starting point for more sophisticated code options in the future.

Bug Fixes

As with any substantial system project, this one has experienced a number of issues as we have gone through the system integration process. Many of these arise from our own software bugs, but others arise because of unexpected idiosyncracies and interactions among purchased components. We mention a few of the items that have been attended to in the last quarter.

One of our biggest concerns at the start of the quarter was an apparent interaction between the shelf manager of the ATCA chassis and the switch card that provides data connections among the various components, but also provides shelf management capabilities (which are disabled in our configuration). Further investigation revealed that the switch card was actually not involved in this problem. Rather, the GPEs were the source of the problem. Once the problem had been identified, we were able to work with the vendor to develop a solution, which involved upgrading the BIOS on the GPE boards.

During this quarter, we noticed that the ATCA chassis' cooling system was not working properly. Specifically, the shelf fans were running continuously, rather than cycling on and off as needed to maintain the proper chassis temperature. While not a serious problem, it was something that we felt needed to be addressed before deployment. After raising the issue with the manufacturer of the chassis and shelf manager, we were told that we needed to upgrade the Power Entry Module for the chassis by adding two capacitors. Remarkably, this took care of the problem, and the cooling system is now functioning properly.

Our ATCA chassis sometimes fails to properly initialize physical interfaces on the Line Card following a reboot. Specifically, interfaces which have nothing plugged into them at reboot time, often do not synchronize with the other endpoint, if a cable is plugged into them following reboot. Up until now, we have corrected this condition by rebooting the Line Card, but this is too disruptive for use in deployed systems. We have developed a utility that accesses a low level interface in the Line Card, so that we can enable and disable individual interfaces as needed.

We have found that the PLC (which is used to enable researchers to define slices) tends to crash after it has been running for more than a few days. We have been told that this is a common problem. For the time being, we are dealing with this by running a daily cron job to restart PLC in the middle of the night. While we don't consider this a great long term solution, we can live with it for now, since the PLC will remain deployed at Washington University, where it will be directly accessible by our staff.

Transition of Staff Responsibilities

During the last quarter, Fred Kuhns, one of our key staff members left our lab to take a position in Austin, where his wife was recently transferred. Fred has implemented most of the system level control software. His responsibilities have been transferred to another staff member, John DeHart, but there will be an ongoing learning curve, as John develops a more detailed understanding of the control software and as he re-balances his workload to accommodate his new responsibilities.

2. Milestones achieved

Develop initial component interface

GEC 4 demonstration

3. Deliverables made

none yet

4. Project participants

Jon Turner – PI
Patrick Crowley – PI
John Dehart – technical staff
Fred Kuhns – technical staff
Dave Zar – technical staff
Ken Wong – technical staff
Mike Wilson – graduate student
Mart Haitjema – graduate student
Ritun Patney – graduate student

5. Publications (individual and organizational)

None yet.

6. Outreach activities

None yet.

7. Collaborations

Have had some discussions with Larry Peterson of Princeton, on rspecs for the SPP.

Attachments (1)

Download all attachments as: .zip