Project Number

1709

Project Title

Data-Intensive Cloud Control for GENI
a.k.a. DICLOUD

Technical Contacts

Michael Zink, Principal Investigator University of Massachusetts, Amherst zink@cs.umass.edu http://www-net.cs.umass.edu/~zink/umasshome/pmwiki.php
Prashant Shenoy, Co-Principal Investigator University of Massachusetts, Amherst shenoy@cs.umass.edu http://www.cs.umass.edu/~shenoy/
Jim Kurose, Co-Principal Investigator University of Massachusetts, Amherst kurose@cs.umass.edu http://www-net.cs.umass.edu/personnel/kurose.html
David Irwin, Post-doctoral Research Associate University of Massachusetts, Amherst irwin@cs.umass.edu http://www.cs.umass.edu/~irwin/
Emmanuel Cecchet, Senior Research Fellow University of Massachusetts, Amherst Amherst, MA 01003-9264 http://www.cs.umass.edu/~cecchet/

Participating Organizations

Cloud Control Trac page hosted at UMass-Amherst. UMassAmherst, Amherst, MA

Related Projects

ViSE project

GPO Liaison System Engineer

Harry Mussman hmussman@geni.net

Scope

This project will develop a complete environment for researchers to conduct data-intensive experiments in GENI from start (the data collection point) to finish (processing and archiving).

To do so, this project will extend the GENI/ViSE sensor network (sensornet) testbed at UMass-Amherst and augment GENI Cluster D’s Orca control framework with capabilities for researchers to

(i) obtain data-centric slices that span core sensornet nodes, data center nodes, and, importantly, storage volumes “in the cloud,”
(ii) deploy popular cloud computing programming paradigms to enable simple, but powerful, distributed data processing, and
(iii) execute experiment workflows to explicitly control experiment data flow and resource allocation across a network of components/aggregates.

The project will build on existing software artifacts in the GENI “ecosystem” and tailor them to the distinct requirements of data-intensive experiments. While the enhanced software artifacts will generalize to any high-bandwidth data-intensive experiments, the GENI/ViSE sensornet testbed, which collects high-bandwidth data from multiple high-power (virtualized) sensor/actuators, will be the initial data source.

Our goal by year one is to incorporate commercial cloud computing services, including storage services, as GENI substrates available for researchers.

Our goal by year two is to enhance GENI’s usefulness by testing and hardening the capability for researchers to request (or load) distributed software platforms on commercial clouds. We will demonstrate the capability using both an MPI stack and Apache’s Hadoop framework, an open-source version of MapReduce? and Google File System (GFS).

Our goal by year three is to complete the integration of Gush to discover resources and deploy experiment workflows across data-centric slices in the Orca CF.

Operational Capabilities

DICloud Portal
DICloud User Guide

Milestones

Spiral 4

• DICLOUD: S4.a Tutorial at GEC12 and Experimenter Outreach (Due 11/04/11 (late))
• DICLOUD: S4.b Documentation and Code Release (Due 11/30/11 (late))
• DICLOUD: S4.c Intermediate (Due 01/15/12 (late))
• DICLOUD: S4.d Demonstration at GEC13 and Experimenter Outreach (Due 03/15/12 (late))
• DICLOUD: S4.e Documentation and Code Release (Due 03/31/12 (late))
• DICLOUD: S4.f Intermediate (Due 06/01/12 (late))
• DICLOUD: S4.g Demonstration at GEC14 and Experimenter Outreach (Due 07/31/12 (late))
• DICLOUD: S4.h Documentation and code release (Due 08/15/12 (late))

Status Reports and Demonstrations

4Q09 Status Report
1Q10 Status Report
GEC7 Demo Description
2Q10 Status Report
GEC8 Demo Description
Spiral 2 review slides

GEC9 Nowcasting Demo
Post GEC9 Status Report
Post-GEC10 Status Report

Technical Documents

Cloud Control Trac page hosted at UMass-Amherst.

Options and Cost Implications for GENI Network Connectivity
Orca-Amazon Cloud Handlers description
Orca-Amazon Cloud Handlers code
Broker Policy

DICloud Portal
DICloud User Guide
DICloud Software Architecture

Software Releases

Code release 11/01/10: (see attachments below)
Includes the AWS accounting service, updated Orca handlers for EC2, S3 and EBS, as well as the DiCloud?? Web Portal.
Documentation on the software installation and usage will be provided.
Expose through the web portal the capability to lease EBS volumes and EC2 servers independently, and bind them to EC2 servers as needed

Tutorial instructions for GEC12 are here. Example steps for using Gush with Orca are here.

Connectivity

Options and Cost Implications for GENI Network Connectivity