cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sebastien Goasguen <run...@gmail.com>
Subject Re: [GSoC][Proposal] Integration project to deploy and use Mesos on a CloudStack based cloud
Date Tue, 07 May 2013 07:46:03 GMT

On May 1, 2013, at 7:27 AM, Dharmesh Kakadia <dharmesh.kakadia@research.iiit.ac.in>
wrote:

> Sebastien and Chiradeep, thanks for the comments !! That clarified a lot of
> things. I just read Chiradeep's blog (
> http://cloudierthanthou.wordpress.com/2013/04/26/stackmate-execute-cloudformation-templates-on-cloudstack/)
> which details the service.
> 
> I am proposing a server side implementation of cloudformation.
> 
> I misunderstood the ReST and Query API. Thanks for correcting. Information
> here(http://gehrcke.de/2009/06/aws-about-api/) helped me. In case we want
> to use existing AWS tools for cloudformation, we also would be designing
> Query API, not ReST.
> 
> Sorry for the confusion regarding cloudmonkey. I was proposing to integrate
> cloudformation API into cloudstack source code, directly and add
> corresponding support in cloudmonkey. But as you suggested, it might be
> easy to start with prototype decoupled from cloudstack (Uses cloudstack API
> and does not reside in cloudstack). I assume by existing cloudformation
> tools you mean AWS tools(
> http://aws.amazon.com/developertools/AWS-CloudFormation). Reusing them will
> be a really good idea.

Yes first step should be with cloud formation (stackmate) outside cloudstack. And I strongly
suggest that you make sure it is compatible with the AWS tools.

> 
> There are lot of options for configuration mgmt tools. I have used knife
> previously and good to know that it has cloudstack plugin based on fog (
> https://github.com/fifthecho/knife-cloudstack-fog). Reasons rundeck looked
> better was support for rollbacking and is full workflow execution engine.
> Finally rundeck can use chef/puppet. I have seen provisonr/whirr and they
> look promising. Definitely a lot to explore here !!

Yes, I have nothing against rundeck. But it would be one more dependency.

> 
> Thanks for suggesting clear proposal.
> 
> Thanks,
> Dharmesh
> 
> 
> 
> 
> 
> On Wed, May 1, 2013 at 3:07 PM, Sebastien Goasguen <runseb@gmail.com> wrote:
> 
>> 
>> On Apr 30, 2013, at 4:59 PM, Chiradeep Vittal <Chiradeep.Vittal@citrix.com>
>> wrote:
>> 
>>> 
>>> 
>>> On 4/30/13 5:01 AM, "Sebastien Goasguen" <runseb@gmail.com> wrote:
>>> 
>>>> Dharmesh, see in-line
>>>> 
>>>> On Apr 30, 2013, at 5:34 AM, Dharmesh Kakadia <dhkakadia@gmail.com>
>> wrote:
>>>> 
>>>>> Hi,
>>>>> 
>>>>> I am Dharmesh Kakdia and interested in project "Integration project to
>>>>> deploy and use Mesos on a CloudStack based cloud" (
>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-1784)
>>>>> 
>>>>> I am working on proposal and want to get feedback. Please provide
>>>>> suggestions :)
>>>>> 
>>>>> *
>>>>> 
>>>>> Abstract:
>>>>> 
>>>>> The project aims to bring cloudformation[1] like service to cloudstack.
>>>>> One
>>>>> of the prime use-case is cluster computing frameworks on cloudstack.
A
>>>>> cloudformation service will give users and administrators of cloudstack
>>>>> ability to manage and control a set of resources easily. The
>>>>> cloudformation
>>>>> will allow booting and configuring a set of VMs and form a cluster.
>>>>> Simple
>>>>> example would be LAMP stack. More complex clusters such as mesos or
>>>>> hadoop
>>>>> cluster requires a little more advanced configuration. There is already
>>>>> some work done by Chiradeep Vittal at this front [5] using route and
>>>> 
>>>> it's using ruote: http://ruote.rubyforge.org
>>>> 
>>>>> sinatra. In this project, I will implement cloudformation service and
>>>>> demonstrate how to run mesos cluster using it.
>>>> 
>>>> You will create cloud formation templates that describe a mesos cluster
>>>> 
>>>>> 
>>>>> Mesos:
>>>>> 
>>>>> Mesos is a resource management platform for clusters [2]. It aims to
>>>>> increase resource utilization of clusters by sharing cluster resources
>>>>> among multiple processing frameworks(like MapReduce, MPI, Graph
>>>>> Processing)
>>>>> or multiple instances of same framework. It provides efficient resource
>>>>> isolation through use of containers. Uses zookeeper for state
>>>>> maintenance
>>>>> and fault tolerance.
>>>>> 
>>>>> What can run on mesos ?
>>>>> 
>>>>> Spark: A cluster computing framework based on the Resilient Distributed
>>>>> Datasets (RDDs) abstraction. RDD is more generalized than MapReduce and
>>>>> can
>>>>> support iterative and interactive computation while retaining fault
>>>>> tolerance, scalability, data locality etc.
>>>>> 
>>>>> Hadoop: Hadoop is fault tolerant and scalable distributed computing
>>>>> framework based on MapReduce abstraction.
>>>>> 
>>>>> Begel: A graph processing framework based on pregel.
>>>>> 
>>>>> and other frameworks like MPI, Hypertable.
>>>>> 
>>>>> How to deploy mesos
>>>>> 
>>>>> Mesos provides cluster installation scripts [7] for cluster deployment.
>>>>> There are also scripts available to deploy a cluster on Amazon EC2 [8].
>>>> 
>>>> It would be nice to see if these scripts can be used as is with the
>>>> CloudStack EC2 service.
>>>> 
>>>>> 
>>>>> Deliverables:
>>>>> 
>>>>> 1. Cloudformation service implementation on cloudstack.
>>>>> 
>>>>> 2. Integration of cloudformation with cloudmonkey, CLI tool.
>>>> 
>>>> 2. is a little confusing. I believe that what Chiradeep prototype runs
>> on
>>>> the client side. What is needed is a server side implementation.
>>>> That way we could use existing cloudformation cli tools to talk to it.
>>>> I don't understand where cloudmonkey comes into play. CloudMonkey is a
>>>> cli for the CloudStack API. Unless you plan to integrate the
>>>> cloudformation API directly in the cloudstack source code, the
>>>> integration you propose is not clear to me.
>>>> 
>>> 
>>> Sebastien is correct. I intend to put in the query API server around the
>>> core of stack mate soon (as soon as I'm done helping on the internal
>>> loadbalancer). This will be written in Ruby.
>>> 
>>> 
>> 
>> Dharmesh I suggest you propose the following:
>> 
>> 1-Deploy CloudStack and understand instance configuration/contextualization
>> 2-Test and deploy Mesos on a set of CloudStack based VM, manually.
>> Design/propose an automation framework.
>> 3-Test stackmate and engage chiradeep (report bugs, make suggestion, make
>> pull request)
>> 4-Create cloud formation template to provision a Mesos Cluster
>> 5-Compare with Apache Whirr or other cluster provisioning tools.
>> 6-Potentially if you see a link with cloudmonkey, see how you could extend
>> it to talk to stackmate in a similar manner that it talks to CloudStack.
>> 
>> 
>> You are pretty close and this is a very exciting projects, so go ahead,
>> modify a bit your proposal and submit it.
>> 
>> Deadline for applications is this Friday May 3rd.
>> 
>> -sebastien
>> 
>> 
>> 
>> 


Mime
View raw message