Data-Intensive Cloud Control for GENI
Michael Zink, Principal Investigator
University of Massachusetts, Amherst
Prashant Shenoy, Co-Principal Investigator University of Massachusetts, Amherst firstname.lastname@example.org http://www.cs.umass.edu/~shenoy/
Jim Kurose, Co-Principal Investigator University of Massachusetts, Amherst email@example.com http://www-net.cs.umass.edu/personnel/kurose.html
David Irwin, Post-doctoral Research Associate University of Massachusetts, Amherst firstname.lastname@example.org http://www.cs.umass.edu/~irwin/
Emmanuel Cecchet, Senior Research Fellow University of Massachusetts, Amherst Amherst, MA 01003-9264 http://www.cs.umass.edu/~cecchet/
GPO Liaison System Engineer
Harry Mussman email@example.com
This project will develop a complete environment for researchers to conduct data-intensive experiments in GENI from start (the data collection point) to finish (processing and archiving).
To do so, this project will extend the GENI/ViSE sensor network (sensornet) testbed at UMass-Amherst and augment GENI Cluster D’s Orca control framework with capabilities for researchers to
(i) obtain data-centric slices that span core sensornet nodes, data center nodes, and, importantly, storage volumes “in the cloud,”
(ii) deploy popular cloud computing programming paradigms to enable simple, but powerful, distributed data processing, and
(iii) execute experiment workflows to explicitly control experiment data flow and resource allocation across a network of components/aggregates.
The project will build on existing software artifacts in the GENI “ecosystem” and tailor them to the distinct requirements of data-intensive experiments. While the enhanced software artifacts will generalize to any high-bandwidth data-intensive experiments, the GENI/ViSE sensornet testbed, which collects high-bandwidth data from multiple high-power (virtualized) sensor/actuators, will be the initial data source.
Our goal by year one is to incorporate commercial cloud computing services, including storage services, as GENI substrates available for researchers.
Our goal by year two is to enhance GENI’s usefulness by testing and hardening the capability for researchers to request (or load) distributed software platforms on commercial clouds. We will demonstrate the capability using both an MPI stack and Apache’s Hadoop framework, an open-source version of MapReduce? and Google File System (GFS).
Our goal by year three is to complete the integration of Gush to discover resources and deploy experiment workflows across data-centric slices in the Orca CF.
- DICLOUD: S4.a Tutorial at GEC12 and Experimenter Outreach (Due 11/04/11 (late))
- DICLOUD: S4.b Documentation and Code Release (Due 11/30/11 (late))
- DICLOUD: S4.c Intermediate (Due 01/15/12 (late))
- DICLOUD: S4.d Demonstration at GEC13 and Experimenter Outreach (Due 03/15/12 (late))
- DICLOUD: S4.e Documentation and Code Release (Due 03/31/12 (late))
- DICLOUD: S4.f Intermediate (Due 06/01/12 (late))
- DICLOUD: S4.g Demonstration at GEC14 and Experimenter Outreach (Due 07/31/12 (late))
- DICLOUD: S4.h Documentation and code release (Due 08/15/12 (late))
Status Reports and Demonstrations
Cloud Control Trac page hosted at UMass-Amherst.
Code release 11/01/10: (see attachments below)
Includes the AWS accounting service, updated Orca handlers for EC2, S3 and EBS, as well as the DiCloud?? Web Portal.
Documentation on the software installation and usage will be provided.
Expose through the web portal the capability to lease EBS volumes and EC2 servers independently, and bind them to EC2 servers as needed
- 2009-12-31 GENI DiCloud q1 report final.pdf (36.6 kB) - added by firstname.lastname@example.org on 12/31/09 10:32:26.
- orca_amazon_handlers.tgz (37.7 kB) -
Orca Amazon Cloud Handlers, added by email@example.com on 01/29/10 11:57:28.
- 2009-01-29 S2c Orca-Amazon Cloud handlers.pdf (64.1 kB) -
Orca Amazon Handler Guide, added by firstname.lastname@example.org on 01/29/10 11:57:57.
- 2009-02-10 S2d Broker policy.pdf (72.9 kB) -
Broker Policy for Monitoring Cloud Usage, added by email@example.com on 02/11/10 11:12:51.
- GEC7 demo.pdf (143.6 kB) -
GEC7 demo description (missing the EC2 resource bridged through ViSE), added by firstname.lastname@example.org on 03/31/10 14:28:49.
- 2010-03-31 GENI DiCloud q2 report.pdf (41.4 kB) - added by email@example.com on 03/31/10 14:50:23.
- 2009-05-15 S2.f CloudWatch monitoring.pdf (0.8 MB) -
Deliverable S2.f, added by firstname.lastname@example.org on 05/14/10 14:28:54.
- 2010-06-31 GENI DiCloud q3 report.pdf (32.1 kB) - added by email@example.com on 07/01/10 23:41:22.
- 2010-07-20 GEC8 Cluster D Session - DiCloud.pdf (280.8 kB) -
Slides presented at the Cluster D session during GEC8, added by firstname.lastname@example.org on 07/26/10 09:48:30.
- 2010-07-20 GEC8 DiCloud demo.pdf (0.6 MB) -
Slides of the DiCloud? demo presented at GEC8, added by email@example.com on 07/26/10 09:48:57.
- 2010-08-06 DiCloud - Spiral 2 Project Review v2.pptx (146.8 kB) -
DiCloud? - Spiral 2 review slides, added by firstname.lastname@example.org on 08/05/10 17:08:59.
- dicloud-src-dist.tar.gz (27.2 kB) -
Updated handlers for Orca bella and aggregate manager demo-ed at GEC8 (source code), added by email@example.com on 08/13/10 00:55:15.
- dicloud-dist.tar.gz (3.5 MB) -
Updated handlers for Orca bella and aggregate manager demo-ed at GEC8 (binary distribution), added by firstname.lastname@example.org on 08/13/10 00:55:59.
- gec8_demo_scripts.tgz (5.1 kB) -
Scripts used for the GEC8 demo, added by email@example.com on 08/13/10 00:56:23.
- 2010-11-01 dicloud-web-portal.tar.gz (4.0 MB) -
DiCloud? Web portal source code, added by firstname.lastname@example.org on 11/01/10 12:17:32.
- 2010-11-01 dicloud-src-dist.tar.gz (45.6 kB) -
DiCloud? software source code (Orca handlers, console and server), added by email@example.com on 11/01/10 12:18:20.
- 2010-11-01 dicloud-dist.tar.gz (3.7 MB) -
DiCloud? software binary distribution (Orca handlers, console and server), added by firstname.lastname@example.org on 11/01/10 12:18:51.
- DiCloudUserguide.pdf (0.7 MB) -
DiCloud? User Guide (Oct 26, 2010 version), added by email@example.com on 11/01/10 12:21:02.
- 2010-10 DiCloud Software architecure.pdf (508.9 kB) -
DiCloud? software architecture and administration document, added by firstname.lastname@example.org on 11/03/10 17:14:01.
- 2010-11-05 GENI DiCloud GEC9 report.pdf (37.9 kB) -
DiCloud? progress report (GEC 9), added by email@example.com on 11/08/10 13:01:41.
- 2011-03-29 GENI DiCloud GEC10 report.pdf (16.0 kB) -
Quarterly progress report, added by firstname.lastname@example.org on 03/29/11 17:00:34.
- nowcasting gec9 demos.pptx (2.0 MB) - added by email@example.com on 04/15/11 17:24:11.
- 2011-07-29 GENI DiCloud GEC11 report.pdf (25.7 kB) -
DiCloud? report post GEC 11, added by firstname.lastname@example.org on 10/18/11 09:18:47.
- 2011-11-01 GENI DiCloud GEC12 report.doc (49.0 kB) -
Post-GEC12 Report, added by email@example.com on 11/17/11 15:53:26.
- GEC13-report.docx (130.7 kB) -
Post-GEC13 DiCloud? Report, added by firstname.lastname@example.org on 04/02/12 09:40:52.