helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Puneet Zaroo <puneetza...@gmail.com>
Subject Re: A few questions about helix.
Date Sun, 24 Feb 2013 03:22:06 GMT
Thanks for the detailed reply.
Please see further comments inline.

> 3) Spectator is informed of the changes due to each state transition.

OK. Will that not cause a lot of overhead if there are a lot of
Spectators in the system. Or was the rationale that there will be just
a few spectators in the system.

> 2) Yes it is possible to throttle the state transitions in a controlled
> manner. You can basically specify the max number of transitions that can
> occur at a resource, instance, instanceGroup, Cluster level. Helix will
> ensure that none of those constraints are violated.

What I had in mind was throttling based on time and not the number of
events. I.e. if a slave partition is lost, then the controller should
wait for some configurable time before selecting another slave
partition. This is to handle the case where a node is rebooting and we
do not want its partitions to be moved to a new node immediately.

> 1) Interpose Primary selection, yes it is possible  implement a custom
> primary selection algorithm. Here is how we achieve that in LinkedIn
> a) A separate entity watches the ExternalView and as soon as it finds out
> there is no primary for a partition, it can do the leader election and set
> the idealstate. You can do this using the CustomCodeInvoker option which
> ensures only one process watches the external view and computes the new
> primary and sets the idealstate.
> Your suggestion of LEADER_ELECTION state sounds interesting. Can you
> elaborate a bit more on the state machine ( states and transitions and
> constraints). How will they get into this state?.

Are there any examples of how to use the CustomCodeInvoker ?

Regarding a separate entity watching the ExternalView. Maybe I did not
follow this fully, but the external entity looks similar to the
controller; so I am not sure if this would solve the particular

We actually want the participants to take part in the decision of who
should become the next Primary or Master. I havent thought this
through completely, but one way could be to add a state
"LEADER_ELECTION" between the states "SLAVE" and "MASTER". In the
"LEADER_ELECTION" state the participants communicate with each other
and decide who should be the next Master, and the participant elected
as the next "Master" sets the IdealState.  This is fully auto mode,
except for one transition "LEADER_ELECTION" -> "MASTER" which is
Perhaps there are simpler ways of doing this.

- Puneet

> On Thu, Feb 21, 2013 at 5:22 PM, Puneet Zaroo <puneetzaroo@gmail.com> wrote:
>> I am a helix newbie. I have read the paper and the wiki pages and am
>> just starting to get familiar with the source code. I had a few
>> questions :
>> 1) Is it possible to interpose on Primary selection. I.e. instead of
>> relying completely on Helix to select a Primary, is it possible to
>> implement a voting based protocol, where the replicas have a say in
>> who becomes the next primary. One possible way would be to have a
>> state "LEADER_ELECTION", in which the replicas do the voting, and
>> finally just the winner sets the ideal state with itself as the
>> Are there any gotchas in what I outlined above, or is there a
>> completely alternative and better way of doing this ?
>> 2) Is it possible to throttle state transitions. E.g. If a node goes
>> offline, the replicas hosted on it should not be transferred to a new
>> node immediately; but in a throttled manner.
>> 3) When is a spectator informed of the new ExternalView ? Is it when
>> currentState becomes equal to the idealState, or are they informed on
>> all state changes due to each state transition.
>> thanks,
>> - Puneet

View raw message