[[PageOutline]] == Project Number == 1709 == Project Title == Data-Intensive Cloud Control for GENI [[BR]] a.k.a. DICLOUD Cloud Control [http://vise.cs.umass.edu/trac/wiki/CloudControl Trac page] hosted at UMass-Amherst. === Technical Contacts === Michael Zink, Principal Investigator University of Massachusetts, Amherst zink@cs.umass.edu http://www-net.cs.umass.edu/~zink/umasshome/pmwiki.php [[BR]] Prashant Shenoy, Co-Principal Investigator University of Massachusetts, Amherst shenoy@cs.umass.edu http://www.cs.umass.edu/~shenoy/ [[BR]] Jim Kurose, Co-Principal Investigator University of Massachusetts, Amherst kurose@cs.umass.edu http://www-net.cs.umass.edu/personnel/kurose.html [[BR]] David Irwin, Post-doctoral Research Associate University of Massachusetts, Amherst irwin@cs.umass.edu http://www.cs.umass.edu/~irwin/ [[BR]] Emmanuel Cecchet, Senior Research Fellow University of Massachusetts, Amherst Amherst, MA 01003-9264 http://www.cs.umass.edu/~cecchet/ [[BR]] === Participating Organizations === [http://www.cs.umass.edu/ UMassAmherst, Amherst, MA][[BR]] === GPO Liaison System Engineer === Harry Mussman hmussman@geni.net == Scope == This project will develop a complete environment for researchers to conduct data-intensive experiments in GENI from start (the data collection point) to finish (processing and archiving). [[BR]] To do so, this project will extend the GENI/ViSE sensor network (sensornet) testbed at UMass-Amherst and augment GENI Cluster D’s Orca control framework with capabilities for researchers to (i) obtain data-centric slices that span core sensornet nodes, data center nodes, and, importantly, storage volumes “in the cloud,” [[BR]] (ii) deploy popular cloud computing programming paradigms to enable simple, but powerful, distributed data processing, and [[BR]] (iii) execute experiment workflows to explicitly control experiment data flow and resource allocation across a network of components/aggregates. [[BR]] The project will build on existing software artifacts in the GENI “ecosystem” and tailor them to the distinct requirements of data-intensive experiments. While the enhanced software artifacts will generalize to any high-bandwidth data-intensive experiments, the GENI/ViSE sensornet testbed, which collects high-bandwidth data from multiple high-power (virtualized) sensor/actuators, will be the initial data source. [[BR]] Our goal by year one is to incorporate commercial cloud computing services, including storage services, as GENI substrates available for researchers. [[BR]] Our goal by year two is to enhance GENI’s usefulness by testing and hardening the capability for researchers to request (or load) distributed software platforms on commercial clouds. We will demonstrate the capability using both an MPI stack and Apache’s Hadoop framework, an open-source version of MapReduce and Google File System (GFS).[[BR]] Our goal by year three is to complete the integration of Gush to discover resources and deploy experiment workflows across data-centric slices in the Orca CF.[[BR]] === Current Capabilities === === Milestones === [[MilestoneDate(DICLOUD: S2.a Cluster plan for VLANs between testbeds)]] [[BR]] [[MilestoneDate(DICLOUD: S2.b Plan to connect to cloud)]] [[BR]] [[MilestoneDate(DICLOUD: S2.c Handlers to allocate cloud)]] [[BR]] [[MilestoneDate(DICLOUD: S2.d Policy to track usage)]] [[BR]] [[MilestoneDate(DICLOUD: S2.e Demo archiving sensor data)]] [[BR]] [[MilestoneDate(DICLOUD: S2.f Use CloudWatch to monitor usage)]] [[BR]] [[MilestoneDate(DICLOUD: S2.g Demo initial proxy aggregate manager)]] [[BR]] [[MilestoneDate(DICLOUD: S2.h Release initial proxy aggregate manager)]] [[BR]] [[MilestoneDate(DICLOUD: S2.i Extend ViSE web portal to include cloud)]] [[BR]] [[MilestoneDate(DICLOUD: S2.j Make available initial set of resources)]] [[BR]] [[MilestoneDate(DICLOUD: S2.k POC to GENI response team)]] [[BR]] [[MilestoneDate(DICLOUD: S2.l POC to GENI security team)]] [[BR]] [[MilestoneDate(DICLOUD: S2.m Contribution to GENI outreach)]] [[BR]] == Project Technical Documents == [http://vise.cs.umass.edu/trac/attachment/wiki/CloudControl/2009-12-23%20Options%20and%20cost%20implications%20for%20GENI%20network%20connectivity_final.pdf Options and Cost Implications for GENI Network Connectivity] [[BR]] [http://groups.geni.net/geni/attachment/wiki/DICLOUD/2009-01-29%20S2c%20Orca-Amazon%20Cloud%20handlers.pdf Orca-Amazon Cloud Handlers description] [[BR]] [http://vise.cs.umass.edu/trac/attachment/wiki/CloudControl/orca_amazon_handlers.tgz Orca-Amazon Cloud Handlers code][[BR]] [http://groups.geni.net/geni/attachment/wiki/DICLOUD/2009-02-10%20S2d%20Broker%20policy.pdf Broker Policy] [[BR]] [http://groups.geni.net/geni/attachment/wiki/DICLOUD/GEC7%20demo.pdf GEC7 Demo Description] [[BR]] === Quarterly Status Reports === [wiki:DICLOUD-4Q09-status 4Q09 Status Report][[BR]] [wiki:DICLOUD-1Q10-status 1Q10 Status Report] [[BR]] === Spiral 2 Connectivity === === Related Projects === [http://groups.geni.net/geni/wiki/ViSE ViSE project][[BR]]