helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore g <g.kish...@gmail.com>
Subject Re: State transitions of partitions
Date Thu, 28 Feb 2013 03:11:23 GMT
Its my pleasure and glad you are liking the design choices. Feel free to
suggest/contribute changes.

Good question on propagating the exception to controller. There are two
ways, if the transition throws an exception we put that partition in ERROR
state and depending on the mode of execution controller will select another
replica to satisfy the constraints. This partition will permanently be in
ERROR state until admin manually invokes a RESET partition api which will
put it back into OFFLINE or initial state.
Before getting to the next option, please note that you can actually have
time outs and multiple retires for a transition (analogous to task attempts
in hadoop), you can configure the number of retries before a transition is
considered as failed. In general, this might be a transient error either
due to configuration or some setup issue and in most cases all partitions
go into error state. This also is reset on node restart, that is controller
wont remember that you failed the previous transition.

In a more permanent failure, like disk crash or other catastrophic
scenario, the participant can request controller to disable itself either
for that partition or the entire node. This mean the node has permanent
issue and its disable across restarts unlike ERROR which gets reset when
you restart the node. The advantage of disable is you get the vm to be up
so that you can debug the issues without impacting the cluster stability.

Will this address your problem, we dont have distinct actions based on
ERROR codes that controller will understand and take different actions.
Were you looking for something like that ?

Good point on not differentiating if the partition once existed v/s newly
created.  We actually plan to modify the drop notification
behavior. Jason/Terence are discussing about this in another thread. Please
add your suggestion to that thread. We should probably have a create and
drop method(not transition) on the participants.

Kishore G

On Wed, Feb 27, 2013 at 4:42 PM, Vinayak Borkar <vborky@yahoo.com> wrote:

> Kishore,
> As always, thanks for the prompt response.
> On 2/27/13 10:53 AM, kishore g wrote:
>> Hi Vinayak,
>> By default, a transition is not time bound(it can be a short one or really
>> long), you can do the data movement as part of the transition and return
>> from the transition after its complete.
> It looks like I can run long-running transitions in this call itself
> unlike traditional event based systems where the event callback needs to
> hand over long running tasks to a separate worker. Furthermore, it looks
> like if the transition function throws an Exception, the server treats it
> as a transition failure so the controller can react to that. This is
> perfect! One question - what is the best way to propagate the exception to
> the controller so that the Controller can take different actions based on
> different kinds of problems (transient issues vs. more permanent errors,
> for e.g.).
>> Lets say you have some transition called STARTED-BOOTSTRAPPED where you
>> copy the data. And say you have 100 nodes in the cluster that needs to go
>> from STARTED-BOOTSTRAPED. You can simply set the end state of system like
>> ...
>> ....
> Talking about STARTED/BOOTSTRAPPED, I noticed that the builtin MasterSlave
> StateModel has a transition from OFFLINE to DROPPED, but there is no
> corresponding transition for bringing on a fresh dataset partition.
> For example, imagine the case where a new dataset is created and installed
> as a resource. Every participant who will store a partition of that dataset
> will see an OFFLINE->SLAVE transition and then possibly a SLAVE->MASTER
> transition for the masters. However, what is the best way to differentiate
> between the initial on-boarding of a dataset and the case where a dataset
> partition is being moved to a participant. In both cases the participant
> sees the same transition I think.
> Thanks,
> Vinayak

View raw message