Mailing-List: contact dev-help@cloudstack.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cloudstack.apache.org
Received-SPF: pass (nike.apache.org: domain of runseb@gmail.com designates
 74.125.83.54 as permitted sender)
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\))
Subject: Re: [GSoC][Proposal] Integration project to deploy and use Mesos on a
 CloudStack based cloud
From: Sebastien Goasguen <runseb@gmail.com>
In-Reply-To: 
 <CALxDHn=A4zjr16X_qb-GbnBk2=KJUy9qF2M8qRCBPD22BYGwfA@mail.gmail.com>
Date: Tue, 7 May 2013 03:46:03 -0400
Content-Transfer-Encoding: quoted-printable
Message-Id: <24752387-3DA3-4418-AFBD-1B86906A8F87@gmail.com>
References: <CDA57E54.13BD2%chiradeep.vittal@citrix.com>
 <D15CA177-F530-4DC1-87F7-E9292D798256@gmail.com>
 <CALxDHn=A4zjr16X_qb-GbnBk2=KJUy9qF2M8qRCBPD22BYGwfA@mail.gmail.com>
To: dev@cloudstack.apache.org


On May 1, 2013, at 7:27 AM, Dharmesh Kakadia =
<dharmesh.kakadia@research.iiit.ac.in> wrote:

> Sebastien and Chiradeep, thanks for the comments !! That clarified a =
lot of
> things. I just read Chiradeep's blog (
> =
http://cloudierthanthou.wordpress.com/2013/04/26/stackmate-execute-cloudfo=
rmation-templates-on-cloudstack/)
> which details the service.
>=20
> I am proposing a server side implementation of cloudformation.
>=20
> I misunderstood the ReST and Query API. Thanks for correcting. =
Information
> here(http://gehrcke.de/2009/06/aws-about-api/) helped me. In case we =
want
> to use existing AWS tools for cloudformation, we also would be =
designing
> Query API, not ReST.
>=20
> Sorry for the confusion regarding cloudmonkey. I was proposing to =
integrate
> cloudformation API into cloudstack source code, directly and add
> corresponding support in cloudmonkey. But as you suggested, it might =
be
> easy to start with prototype decoupled from cloudstack (Uses =
cloudstack API
> and does not reside in cloudstack). I assume by existing =
cloudformation
> tools you mean AWS tools(
> http://aws.amazon.com/developertools/AWS-CloudFormation). Reusing them =
will
> be a really good idea.

Yes first step should be with cloud formation (stackmate) outside =
cloudstack. And I strongly suggest that you make sure it is compatible =
with the AWS tools.

>=20
> There are lot of options for configuration mgmt tools. I have used =
knife
> previously and good to know that it has cloudstack plugin based on fog =
(
> https://github.com/fifthecho/knife-cloudstack-fog). Reasons rundeck =
looked
> better was support for rollbacking and is full workflow execution =
engine.
> Finally rundeck can use chef/puppet. I have seen provisonr/whirr and =
they
> look promising. Definitely a lot to explore here !!

Yes, I have nothing against rundeck. But it would be one more =
dependency.

>=20
> Thanks for suggesting clear proposal.
>=20
> Thanks,
> Dharmesh
>=20
>=20
>=20
>=20
>=20
> On Wed, May 1, 2013 at 3:07 PM, Sebastien Goasguen <runseb@gmail.com> =
wrote:
>=20
>>=20
>> On Apr 30, 2013, at 4:59 PM, Chiradeep Vittal =
<Chiradeep.Vittal@citrix.com>
>> wrote:
>>=20
>>>=20
>>>=20
>>> On 4/30/13 5:01 AM, "Sebastien Goasguen" <runseb@gmail.com> wrote:
>>>=20
>>>> Dharmesh, see in-line
>>>>=20
>>>> On Apr 30, 2013, at 5:34 AM, Dharmesh Kakadia <dhkakadia@gmail.com>
>> wrote:
>>>>=20
>>>>> Hi,
>>>>>=20
>>>>> I am Dharmesh Kakdia and interested in project "Integration =
project to
>>>>> deploy and use Mesos on a CloudStack based cloud" (
>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-1784)
>>>>>=20
>>>>> I am working on proposal and want to get feedback. Please provide
>>>>> suggestions :)
>>>>>=20
>>>>> *
>>>>>=20
>>>>> Abstract:
>>>>>=20
>>>>> The project aims to bring cloudformation[1] like service to =
cloudstack.
>>>>> One
>>>>> of the prime use-case is cluster computing frameworks on =
cloudstack. A
>>>>> cloudformation service will give users and administrators of =
cloudstack
>>>>> ability to manage and control a set of resources easily. The
>>>>> cloudformation
>>>>> will allow booting and configuring a set of VMs and form a =
cluster.
>>>>> Simple
>>>>> example would be LAMP stack. More complex clusters such as mesos =
or
>>>>> hadoop
>>>>> cluster requires a little more advanced configuration. There is =
already
>>>>> some work done by Chiradeep Vittal at this front [5] using route =
and
>>>>=20
>>>> it's using ruote: http://ruote.rubyforge.org
>>>>=20
>>>>> sinatra. In this project, I will implement cloudformation service =
and
>>>>> demonstrate how to run mesos cluster using it.
>>>>=20
>>>> You will create cloud formation templates that describe a mesos =
cluster
>>>>=20
>>>>>=20
>>>>> Mesos:
>>>>>=20
>>>>> Mesos is a resource management platform for clusters [2]. It aims =
to
>>>>> increase resource utilization of clusters by sharing cluster =
resources
>>>>> among multiple processing frameworks(like MapReduce, MPI, Graph
>>>>> Processing)
>>>>> or multiple instances of same framework. It provides efficient =
resource
>>>>> isolation through use of containers. Uses zookeeper for state
>>>>> maintenance
>>>>> and fault tolerance.
>>>>>=20
>>>>> What can run on mesos ?
>>>>>=20
>>>>> Spark: A cluster computing framework based on the Resilient =
Distributed
>>>>> Datasets (RDDs) abstraction. RDD is more generalized than =
MapReduce and
>>>>> can
>>>>> support iterative and interactive computation while retaining =
fault
>>>>> tolerance, scalability, data locality etc.
>>>>>=20
>>>>> Hadoop: Hadoop is fault tolerant and scalable distributed =
computing
>>>>> framework based on MapReduce abstraction.
>>>>>=20
>>>>> Begel: A graph processing framework based on pregel.
>>>>>=20
>>>>> and other frameworks like MPI, Hypertable.
>>>>>=20
>>>>> How to deploy mesos
>>>>>=20
>>>>> Mesos provides cluster installation scripts [7] for cluster =
deployment.
>>>>> There are also scripts available to deploy a cluster on Amazon EC2 =
[8].
>>>>=20
>>>> It would be nice to see if these scripts can be used as is with the
>>>> CloudStack EC2 service.
>>>>=20
>>>>>=20
>>>>> Deliverables:
>>>>>=20
>>>>> 1. Cloudformation service implementation on cloudstack.
>>>>>=20
>>>>> 2. Integration of cloudformation with cloudmonkey, CLI tool.
>>>>=20
>>>> 2. is a little confusing. I believe that what Chiradeep prototype =
runs
>> on
>>>> the client side. What is needed is a server side implementation.
>>>> That way we could use existing cloudformation cli tools to talk to =
it.
>>>> I don't understand where cloudmonkey comes into play. CloudMonkey =
is a
>>>> cli for the CloudStack API. Unless you plan to integrate the
>>>> cloudformation API directly in the cloudstack source code, the
>>>> integration you propose is not clear to me.
>>>>=20
>>>=20
>>> Sebastien is correct. I intend to put in the query API server around =
the
>>> core of stack mate soon (as soon as I'm done helping on the internal
>>> loadbalancer). This will be written in Ruby.
>>>=20
>>>=20
>>=20
>> Dharmesh I suggest you propose the following:
>>=20
>> 1-Deploy CloudStack and understand instance =
configuration/contextualization
>> 2-Test and deploy Mesos on a set of CloudStack based VM, manually.
>> Design/propose an automation framework.
>> 3-Test stackmate and engage chiradeep (report bugs, make suggestion, =
make
>> pull request)
>> 4-Create cloud formation template to provision a Mesos Cluster
>> 5-Compare with Apache Whirr or other cluster provisioning tools.
>> 6-Potentially if you see a link with cloudmonkey, see how you could =
extend
>> it to talk to stackmate in a similar manner that it talks to =
CloudStack.
>>=20
>>=20
>> You are pretty close and this is a very exciting projects, so go =
ahead,
>> modify a bit your proposal and submit it.
>>=20
>> Deadline for applications is this Friday May 3rd.
>>=20
>> -sebastien
>>=20
>>=20
>>=20
>>=20