cloudstack-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mc...@apache.org
Subject [35/50] [abbrv] git commit: updated refs/heads/object_store to 18aeef3
Date Wed, 12 Jun 2013 18:22:44 GMT
[GSoC]: Proposal from Dharmesh Kakadia


Project: http://git-wip-us.apache.org/repos/asf/cloudstack/repo
Commit: http://git-wip-us.apache.org/repos/asf/cloudstack/commit/e17a0c23
Tree: http://git-wip-us.apache.org/repos/asf/cloudstack/tree/e17a0c23
Diff: http://git-wip-us.apache.org/repos/asf/cloudstack/diff/e17a0c23

Branch: refs/heads/object_store
Commit: e17a0c23b8d4d7cbbcbf4a95cf545b0d8ae59aaa
Parents: 302741d
Author: Sebastien Goasguen <runseb@gmail.com>
Authored: Fri Jun 7 06:03:42 2013 -0400
Committer: Sebastien Goasguen <runseb@gmail.com>
Committed: Fri Jun 7 06:03:42 2013 -0400

----------------------------------------------------------------------
 docs/en-US/CloudStack_GSoC_Guide.xml         |   1 +
 docs/en-US/gsoc-dharmesh.xml                 | 149 ++++++++++++++++++++++
 docs/en-US/images/mesos-integration-arch.jpg |   0
 3 files changed, 150 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/cloudstack/blob/e17a0c23/docs/en-US/CloudStack_GSoC_Guide.xml
----------------------------------------------------------------------
diff --git a/docs/en-US/CloudStack_GSoC_Guide.xml b/docs/en-US/CloudStack_GSoC_Guide.xml
index b7ba61f..243a0ca 100644
--- a/docs/en-US/CloudStack_GSoC_Guide.xml
+++ b/docs/en-US/CloudStack_GSoC_Guide.xml
@@ -48,6 +48,7 @@
     </bookinfo>
     <xi:include href="gsoc-tuna.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
     <xi:include href="gsoc-imduffy15.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
+    <xi:include href="gsoc-dharmesh.xml" xmlns:xi="http://www.w3.org/2001/XInclude" />
 
 </book>
 

http://git-wip-us.apache.org/repos/asf/cloudstack/blob/e17a0c23/docs/en-US/gsoc-dharmesh.xml
----------------------------------------------------------------------
diff --git a/docs/en-US/gsoc-dharmesh.xml b/docs/en-US/gsoc-dharmesh.xml
new file mode 100644
index 0000000..5e2bf73
--- /dev/null
+++ b/docs/en-US/gsoc-dharmesh.xml
@@ -0,0 +1,149 @@
+<?xml version='1.0' encoding='utf-8' ?>
+<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"
[
+<!ENTITY % BOOK_ENTITIES SYSTEM "CloudStack_GSoC_Guide.ent">
+%BOOK_ENTITIES;
+]>
+
+<!-- Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements.  See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership.  The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License.  You may obtain a copy of the License at
+ 
+   http://www.apache.org/licenses/LICENSE-2.0
+ 
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied.  See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+<chapter id="gsoc-dharmesh">
+        <title>Dharmesh's 2013 GSoC Proposal</title>
+        <para>This chapter describes Dharmrsh's 2013 Google Summer of Code project
within the &PRODUCT; ASF project. It is a copy paste of the submitted proposal.</para>
+	<section id="abstract-dharmesh">
+		<title>Abstract</title>
+		<para>
+			The project aims to bring <ulink url="http://aws.amazon.com/cloudformation/"><citetitle>cloudformation</citetitle></ulink>
like service to cloudstack. One of the prime use-case is cluster computing frameworks on cloudstack.
A cloudformation service will give users and administrators of cloudstack ability to manage
and control a set of resources easily. The cloudformation will allow booting and configuring
a set of VMs and form a cluster. Simple example would be LAMP stack. More complex clusters
such as mesos or hadoop cluster requires a little more advanced configuration. There is already
some work done by Chiradeep Vittal at this front [5]. In this project, I will implement server
side cloudformation service for cloudstack and demonstrate how to run mesos cluster using
it.
+		</para>
+	</section>
+
+	<section id="mesos">
+		<title>Mesos</title>
+		<para>
+			<ulink url="http://incubator.apache.org/mesos/"><citetitle>Mesos</citetitle></ulink>
is a resource management platform for clusters. It aims to increase resource utilization of
clusters by sharing cluster resources among multiple processing frameworks(like MapReduce,
MPI, Graph Processing) or multiple instances of same framework. It provides efficient resource
isolation through use of containers. Uses zookeeper for state maintenance and fault tolerance.
+		</para>
+	</section>
+
+	<section id="mesos-use">
+		<title>What can run on mesos ?</title>
+		
+		<para><emphasis role="bold">Spark:</emphasis> A cluster computing framework
based on the Resilient Distributed Datasets (RDDs) abstraction. RDD is more generalized than
MapReduce and can support iterative and interactive computation while retaining fault tolerance,
scalability, data locality etc.</para>
+			
+		<para><emphasis role="bold">Hadoop:</emphasis>: Hadoop is fault tolerant
and scalable distributed computing framework based on MapReduce abstraction.</para>
+			
+		<para><emphasis role="bold">Begel:</emphasis>: A graph processing framework
based on pregel.</para>
+
+		<para>and other frameworks like MPI, Hypertable.</para>
+	</section>
+
+	<section id="mesos-deploy">
+		<title>How to deploy mesos ?</title>
+		
+		<para>Mesos provides cluster installation <ulink url="https://github.com/apache/mesos/blob/trunk/docs/Deploy-Scripts.textile"><citetitle>scripts</citetitle></ulink>
for cluster deployment. There are also scripts available to deploy a cluster on <ulink
url="https://github.com/apache/mesos/blob/trunk/docs/EC2-Scripts.textile"><citetitle>Amazon
EC2</citetitle></ulink>. It would be interesting to see if this scripts can be
leveraged in anyway.</para>
+	</section>
+
+	<section id="deliverables-dharmesh">
+		<title>Deliverables</title>
+		<orderedlist>
+			<listitem>
+				<para>Deploy CloudStack and understand instance configuration/contextualization</para>
+			</listitem>
+			<listitem>
+				<para>Test and deploy Mesos on a set of CloudStack based VM, manually. Design/propose
an automation framework</para>
+			</listitem>
+			<listitem>
+				<para>Test stackmate and engage chiradeep (report bugs, make suggestion, make pull
request)</para>
+			</listitem>
+			<listitem>
+				<para>Create cloudformation template to provision a Mesos Cluster</para>
+			</listitem>
+			<listitem>
+				<para>Compare with Apache Whirr or other cluster provisioning tools for server
side implementation of cloudformation service.</para>
+			</listitem>
+		</orderedlist>
+	</section>
+
+	<section id="arch-and-tools">
+		<title>Architecture and Tools</title>
+		
+		<para>The high level architecture is as follows:</para>
+		
+		<para>
+			<mediaobject>
+				<imageobject>
+					<imagedata fileref="images/mesos-integration-arch.jpg"/>
+				</imageobject>
+			</mediaobject>
+		</para>
+
+
+		<para>It includes following components:</para>
+
+		<orderedlist>
+			<listitem>
+				<para>CloudFormation Query API server:</para>
+				<para>This acts as a point of contact to and exposes CloudFormation functionality
as Query API. This can be accessed directly or through existing tools from Amazon AWS for
their cloudformation service. It will be easy to start as a module which resides outside cloudstack
at first and  I plan to use dropwizard [3] to start with. Later may be the API server can
be merged with cloudstack core. I plan to use mysql for storing details of clusters.</para>
+			</listitem>
+
+			<listitem>
+				<para>Provisioning:</para>
+
+				<para>Provisioning module is responsible for handling the booting process of the
VMs through cloudstack. This uses the cloudstack APIs for launching VMs. I plan to use preconfigured
templates/images with required dependencies installed, which will make cluster creation process
much faster even for large clusters. Error handling is very important part of this module.
For example, what you do if few VMs fail to boot in cluster ?</para>
+			</listitem>
+
+			<listitem>
+				<para>Configuration:</para>
+
+				<para>This module deals with configuring the VMs to form a cluster. This can be
done via manual scripts/code or via configuration management tools like chef/ironfan/knife.
Potentially workflow automation tools like rundeck [4] also can be used. Also Apache whirr
and Provisionr are options. I plan to explore this tools and select suitable ones.</para>
+			</listitem>
+
+		</orderedlist>
+	</section>
+
+	<section id="api">
+		<title>API</title>
+		
+		<para>Query <ulink url="http://docs.aws.amazon.com/AWSCloudFormation/latest/APIReference/API_Operations.html"><citetitle>API</citetitle></ulink>
will be based on Amazon AWS cloudformation service. This will allow leveraging existing <ulink
url="http://aws.amazon.com/developertools/AWS-CloudFormation"><citetitle>tools</citetitle></ulink>
for AWS.</para>
+	</section>
+
+	<section id="timeline">
+		<title>Timeline</title>
+		<para>1-1.5 week : project design. Architecture, tools selection, API design</para>
+		<para>1-1.5 week : getting familiar with cloudstack and stackmate codebase and architecture
details</para>
+		<para>1-1.5 week : getting familiar with mesos internals</para>
+		<para>1-1.5 week : setting up the dev environment and create mesos templates</para>
+		<para>2-3 week : build provisioning and configuration module</para>
+		<para>Midterm evaluation: provisioning module, configuration module</para>
+		<para>2-3 week : develope cloudformation server side implementation</para>
+		<para>2-3 week : test and integrate</para>
+	</section>
+
+	<section id="future-work">
+		<title>Future Work</title>
+		<orderedlist>
+			<listitem>
+				<para><emphasis role="bold">Auto Scaling:</emphasis></para>
+				<para>Automatically adding or removing VMs from mesos cluster based on various
conditions like utilization going above/below a static threshold. There can be more sophisticated
strategies based on prediction or fine grained metric collection with tight integration with
mesos framework.</para>
+			</listitem>
+			<listitem>
+				<para><emphasis role="bold">Cluster Simulator:</emphasis></para>
+				<para>Integrating with existing simulator to simulate mesos clusters. This can
be useful in various scenarios, for example while developing a new scheduling algorithm, testing
autoscaling etc.</para>
+			</listitem>
+		</orderedlist>
+	</section>
+</chapter>

http://git-wip-us.apache.org/repos/asf/cloudstack/blob/e17a0c23/docs/en-US/images/mesos-integration-arch.jpg
----------------------------------------------------------------------
diff --git a/docs/en-US/images/mesos-integration-arch.jpg b/docs/en-US/images/mesos-integration-arch.jpg
new file mode 100644
index 0000000..e69de29


Mime
View raw message