helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kanak Biscuitwala <kana...@hotmail.com>
Subject RE: Apache Helix
Date Thu, 06 Mar 2014 22:13:48 GMT
Hi Leonard,

We are currently actively developing a way for the Helix controller to do exactly that. Essentially,
working with YARN, Mesos, EC2, or another provisioner, the Helix controller will look at your
service constraints for how many machines should be up, and ensure that many containers are
always running for your service. We are also looking at ways to dynamically monitor CPU and
memory usage to determine the number of containers to have running. We have a working implementation
with YARN in the helix-provisioning branch of our source code. See http://helix.apache.org/sources.html for
instructions on how to access the source code.

Historically, Helix's goal has been to take the machines that are active, and intelligently
distribute your service across those machines. When a service goes down, Helix makes sure
that the cluster remains in a good state. Without provisioning, Helix would simply reallocate
your resources without any ability to start or stop containers. With provisioning, Helix can
now tell the provisioner to bring up or take down containers in addition to reassigning.

> Date: Thu, 6 Mar 2014 13:53:15 -0800 
> Subject: Re: Apache Helix 
> From: g.kishore@gmail.com 
> To: leonard.alexander.kramer@googlemail.com; user@helix.apache.org 
> + user 
> On Thu, Mar 6, 2014 at 5:24 AM, Leonard Kramer 
> <leonard.alexander.kramer@googlemail.com<mailto:leonard.alexander.kramer@googlemail.com>>

> wrote: 
> Hi Mr. Kishore, 
> my name is Leonard Kramer and I've found your mail-address in the 
> Apache Helix mailing list. I'm currently evaluating Apache Helix for 
> the cluster-management of ZooKeeper itself. The ultimate goal is to 
> create an autonomous zookeeper-service with its own 
> migration-strategies and self repair functions. 
> For the monitoring and reaction to outages of nodes I want to use 
> Apache Helix, because it already uses ZooKeeper as its primary 
> coordination & data-store. In the mailing-list you have answered a 
> thread for the differences between norbert and helix and mentioned that 
> "Failure: When a node fails, you have multiple options [...] #3 Start a 
> new node and assign the partitions to that new node. [...] #3 feature 
> is work in progress and is possible if the deployment system is 
> flexible and allows starting up process dynamically". 
> While studying Helix I couldn't find any more information regarding 
> this feature. All I could find was the blog post 
> "http://engineering.linkedin.com/cluster-management/auto-scaling-apache-helix-and-apache-yarn",

> which I think is too complex for my specific use-case. 
> Can you please provide me with more information regarding the self 
> repair of a helix cluster? Is the helix-controller capable of invoking 
> logic for starting additional nodes? My plan is to use Jclouds in 
> addition to the cloud-controller and start a new node when the 
> helix-controller notices a node's failure. Is this possible? 
> Thank you for your help and have nice day. 
> Greetings from Germany 
> Leo 
View raw message