cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dharmesh Kakadia <dhkaka...@gmail.com>
Subject Re: [GSoC][Proposal] Integration project to deploy and use Mesos on a CloudStack based cloud
Date Thu, 02 May 2013 12:30:44 GMT
I am not sure wheather my earlier mail reached or was dropped (I got
failure notice!!). So resending. Sorry incase it is a duplicate.

------

Sebastien and Chiradeep, thanks for the comments !! That clarified a lot of
things. I just read Chiradeep's blog (
http://cloudierthanthou.wordpress.com/2013/04/26/stackmate-execute-cloudformation-templates-on-cloudstack/)
which details the service.

I am proposing a server side implementation of cloudformation.

I misunderstood the ReST and Query API. Thanks for correcting. Information
here(http://gehrcke.de/2009/06/aws-about-api/) helped me. In case we want
to use existing AWS tools for cloudformation, we also would be designing
Query API, not ReST.

Sorry for the confusion regarding cloudmonkey. I was proposing to integrate
cloudformation API into cloudstack source code, directly and add
corresponding support in cloudmonkey. But as you suggested, it might be
easy to start with prototype decoupled from cloudstack (Uses cloudstack API
and does not reside in cloudstack). I assume by existing cloudformation
tools you mean AWS tools(
http://aws.amazon.com/developertools/AWS-CloudFormation). Reusing them will
be a really good idea.

There are lot of options for configuration mgmt tools. I have used knife
previously and good to know that it has cloudstack plugin based on fog (
https://github.com/fifthecho/knife-cloudstack-fog). Reasons rundeck looked
better was support for rollbacking and is full workflow execution engine.
Finally rundeck can use chef/puppet. I have seen provisonr/whirr and they
look promising. Definitely a lot to explore here !!

Thanks for suggesting clear proposal. I have submitted the proposal.

Thanks,
Dharmesh


On Wed, May 1, 2013 at 3:07 PM, Sebastien Goasguen <runseb@gmail.com> wrote:

>
> On Apr 30, 2013, at 4:59 PM, Chiradeep Vittal <Chiradeep.Vittal@citrix.com>
> wrote:
>
> >
> >
> > On 4/30/13 5:01 AM, "Sebastien Goasguen" <runseb@gmail.com> wrote:
> >
> >> Dharmesh, see in-line
> >>
> >> On Apr 30, 2013, at 5:34 AM, Dharmesh Kakadia <dhkakadia@gmail.com>
> wrote:
> >>
> >>> Hi,
> >>>
> >>> I am Dharmesh Kakdia and interested in project "Integration project to
> >>> deploy and use Mesos on a CloudStack based cloud" (
> >>> https://issues.apache.org/jira/browse/CLOUDSTACK-1784)
> >>>
> >>> I am working on proposal and want to get feedback. Please provide
> >>> suggestions :)
> >>>
> >>> *
> >>>
> >>> Abstract:
> >>>
> >>> The project aims to bring cloudformation[1] like service to cloudstack.
> >>> One
> >>> of the prime use-case is cluster computing frameworks on cloudstack. A
> >>> cloudformation service will give users and administrators of cloudstack
> >>> ability to manage and control a set of resources easily. The
> >>> cloudformation
> >>> will allow booting and configuring a set of VMs and form a cluster.
> >>> Simple
> >>> example would be LAMP stack. More complex clusters such as mesos or
> >>> hadoop
> >>> cluster requires a little more advanced configuration. There is already
> >>> some work done by Chiradeep Vittal at this front [5] using route and
> >>
> >> it's using ruote: http://ruote.rubyforge.org
> >>
> >>> sinatra. In this project, I will implement cloudformation service and
> >>> demonstrate how to run mesos cluster using it.
> >>
> >> You will create cloud formation templates that describe a mesos cluster
> >>
> >>>
> >>> Mesos:
> >>>
> >>> Mesos is a resource management platform for clusters [2]. It aims to
> >>> increase resource utilization of clusters by sharing cluster resources
> >>> among multiple processing frameworks(like MapReduce, MPI, Graph
> >>> Processing)
> >>> or multiple instances of same framework. It provides efficient resource
> >>> isolation through use of containers. Uses zookeeper for state
> >>> maintenance
> >>> and fault tolerance.
> >>>
> >>> What can run on mesos ?
> >>>
> >>> Spark: A cluster computing framework based on the Resilient Distributed
> >>> Datasets (RDDs) abstraction. RDD is more generalized than MapReduce and
> >>> can
> >>> support iterative and interactive computation while retaining fault
> >>> tolerance, scalability, data locality etc.
> >>>
> >>> Hadoop: Hadoop is fault tolerant and scalable distributed computing
> >>> framework based on MapReduce abstraction.
> >>>
> >>> Begel: A graph processing framework based on pregel.
> >>>
> >>> and other frameworks like MPI, Hypertable.
> >>>
> >>> How to deploy mesos
> >>>
> >>> Mesos provides cluster installation scripts [7] for cluster deployment.
> >>> There are also scripts available to deploy a cluster on Amazon EC2 [8].
> >>
> >> It would be nice to see if these scripts can be used as is with the
> >> CloudStack EC2 service.
> >>
> >>>
> >>> Deliverables:
> >>>
> >>> 1. Cloudformation service implementation on cloudstack.
> >>>
> >>> 2. Integration of cloudformation with cloudmonkey, CLI tool.
> >>
> >> 2. is a little confusing. I believe that what Chiradeep prototype runs
> on
> >> the client side. What is needed is a server side implementation.
> >> That way we could use existing cloudformation cli tools to talk to it.
> >> I don't understand where cloudmonkey comes into play. CloudMonkey is a
> >> cli for the CloudStack API. Unless you plan to integrate the
> >> cloudformation API directly in the cloudstack source code, the
> >> integration you propose is not clear to me.
> >>
> >
> > Sebastien is correct. I intend to put in the query API server around the
> > core of stack mate soon (as soon as I'm done helping on the internal
> > loadbalancer). This will be written in Ruby.
> >
> >
>
> Dharmesh I suggest you propose the following:
>
> 1-Deploy CloudStack and understand instance configuration/contextualization
> 2-Test and deploy Mesos on a set of CloudStack based VM, manually.
> Design/propose an automation framework.
> 3-Test stackmate and engage chiradeep (report bugs, make suggestion, make
> pull request)
> 4-Create cloud formation template to provision a Mesos Cluster
> 5-Compare with Apache Whirr or other cluster provisioning tools.
> 6-Potentially if you see a link with cloudmonkey, see how you could extend
> it to talk to stackmate in a similar manner that it talks to CloudStack.
>
>
> You are pretty close and this is a very exciting projects, so go ahead,
> modify a bit your proposal and submit it.
>
> Deadline for applications is this Friday May 3rd.
>
> -sebastien
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message