helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Swaroop Jagadish <sjagad...@linkedin.com>
Subject Re: Need two kinds of DROPPED states?
Date Mon, 19 May 2014 18:25:10 GMT
If you have access to the ideal state for which the state transition was
sent, you would be able to tell if a partition was removed or moved. Is
the ideal state already exposed in the context object? A fresh read of
ideal state could lead to race conditions


On 5/19/14 10:23 AM, "Kanak Biscuitwala" <kanak.b@hotmail.com> wrote:

>If your meta-resource follows the CUSTOMIZED rebalance mode, then you
>won't get dropped transitions unless you explicitly remove the entry from
>the ideal state.
>An initial version of the task framework is in the helix-0.6.x branch.
>Unfortunately we don¹t have much documentation at this point, but you can
>take a look at TestTaskRebalancer for an example. There¹s still work to
>do from there, but this should be enough to hopefully get started.
>> Date: Mon, 19 May 2014 09:42:03 -0700
>> From: vinayakb@gmail.com
>> To: user@helix.apache.org
>> Subject: Re: Need two kinds of DROPPED states?
>> Yes, your interpretation is correct.
>> I thought about using your meta-resource idea, but it suffers from a
>> similar problem as what I am currently facing. The partitions of the
>> meta-resource will be "placed" on some participant by Helix. Now imagine
>> that participant fails. The partition of the meta-resource will go
>> through its DROPPED state to get placed elsewhere. So that DROPPED state
>> is still discernible from the partition being dropped to reflect
>> dropping of the resource it represents. Either I misunderstood your
>> solution (1) or I need to somehow have a participant that never fails.
>> I can see how to use the task framework to build a reliable workflow
>> with a sequence of steps where one of the steps is to do the final
>> cleanup. Is this already available in the codebase (if I was willing to
>> make my own build)?
>> Thanks,
>> Vinayak
>> On 5/19/14, 9:11 AM, Kanak Biscuitwala wrote:
>>> So if I understand correctly, you basically want to know if state
>>>should be kept in case the partition could move back, or not. Right
>>>now, Helix treats a dropped resource as if all its partitions have been
>>>dropped. Separately, Helix treats a moved partition as a dropped
>>>partition on one participant and an added partition on another
>>>participant. So they're currently very much linked.
>>> This requires some more thought, but here's what comes to mind:
>>> 1. Have a meta-resource whose partitions are simply the names of the
>>>other resources in the cluster. When you drop a resource, the operation
>>>would be to simultaneously drop the resource and drop the partition
>>>from the meta resource. Then you can get a separate transition for
>>>dropped resource. I haven't thought about the race conditions here, and
>>>there could be some impact depending on your app.
>>> 2. In the upcoming task framework, create a task that manages the drop
>>>resource scenario from beginning to end, for instance call
>>>helixadmin#dropresource, wait for external view to converge, issue
>>>cleanup requests to participants. Participants would implement a
>>>cleanup callback. This is something we're get out the door this quarter.
>>> 3. Something that works, but you would like to avoid: ask HelixAdmin
>>>if the resource exists
>>> Perhaps others can chime in with ideas.
>>> ----------------------------------------
>>>> Date: Sun, 18 May 2014 12:08:15 -0700
>>>> From: vinayakb@gmail.com
>>>> To: user@helix.apache.org
>>>> Subject: Need two kinds of DROPPED states?
>>>> Hi Guys,
>>>> It looks like when a partition that is on a participant (P1) is moved
>>>> another participant (P2), P1 is sent a transition request from OFFLINE
>>>> -> DROPPED.
>>>> In an other scenario, when a resource is dropped using HelixAdmin, the
>>>> partitions undergo a similar transition to DROPPED.
>>>> As an application, one might need to do different things in those two
>>>> cases. For example, in the first case its being dropped to become live
>>>> somewhere else and so any shared state for the resource should not be
>>>> lost. On the other hand, in the second scenario, the application might
>>>> want to clean up all state associated with the resource.
>>>> Is there a way for the application to distinguish between the first
>>>> of DROPPED and the second kind? I am looking to have the state machine
>>>> itself handle both the scenarios without the need for the application
>>>> trigger some special activity to perform the cleanup in the second
>>>> Thanks,
>>>> Vinayak

View raw message