whirr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Baclace <paul.bacl...@gmail.com>
Subject Re: Provisioning as a Dedicated Service
Date Sat, 09 Feb 2013 02:44:46 GMT
Andrei,

I keep watching the update stream of 
github.com/axemblr/axemblr-provisionr and still see plenty of activity.

Do you have any rough idea of state transition latency and throughput 
you get when using Activiti and how this compares to using Whirr/jclouds 
in a single process?

The reason I ask is that although Activiti has good support for 
designing processes and programmatic control of the engine, it is 
necessarily DB transaction limited. An obvious alternative design is to 
use something that is actor based which can run entirely in RAM. I admit 
that an actor control system would make it harder to trace what 
happened, compared to business process control which is very much 
oriented toward human-in-the-loop.

Regards,

Paul

For example, the

On 20121214 7:34 , Andrei Savu wrote:
> Hi guys,
>
> There is no secret that at Axemblr we are using Apache Whirr for
> provisioning and initial basic cluster configuration for Hadoop. As soon as
> the machines are running we configure Hadoop by leveraging APIs from
> existing tools like Cloudera Manager or Ambari.
>
> All the orchestration needed to make this happen is not trivial if you want
> the final system to be predictable, robust, restartable and easy to inspect
> while running.
>
> A few months ago we've realised that we need to re-work the machine
> provisioning layer from Whirr and build a system that has the following
> features:
>
> * should be able to provision 10s or 100s of virtual machines by doing a
> good job at handling API throttling and by using batch operations as much
> as possible
>
> * all the internal workflows should be persistent and as granular as
> possible and each step should be idempotent
>
> * it should be possible to restart the application server while starting
> virtual machines with no impact
>
> * it should have a modular architecture and provide enough flexibility to
> be able to work with a large number of public and private clouds just by
> replacing modules
>
> * it should hide all this complexity behind a simple REST API and a simple
> interactive shell
>
> * it should be able to automatically build gold base images and use the to
> spawn large clusters
>
> We've spent some time looking for existing products that do all this and in
> the end we've decided that it's better to start from scratch and build this
> system as a new project based on Activiti, Apache Karaf, jclouds and native
> sdks.
>
> The source code is now publicly available at:
>
> https://github.com/axemblr/axemblr-provisionr
>
> I would really like to know what you think about the work we've done so
> far. The project will improve a lot over the next couple of weeks / months
> so I encourage you to stay tunned.
>
> We want to bring this project to the Apache Foundation later on. I will
> give a talk in february at ApacheCon NA on this.
>
> Cheers,
>
> -- Andrei Savu / axemblr.com
>


Mime
View raw message