helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Changgeng Li <li.changg...@gmail.com>
Subject Re: Customized rebalancer
Date Thu, 25 Jun 2015 18:20:01 GMT
It seems we don't have access to the event that triggers rebalance in
rebalancer.
Maybe we can add another method in HelixRebalancer similar to something:
shouldRebalance(ClusterEvent: event): boolean, and if this returns false
just don't rebalance.

On Thu, Jun 25, 2015 at 11:07 AM, kishore g <g.kishore@gmail.com> wrote:

> Hi Changgeng,
>
> Yes, prevAssignment will be preserved even if controller moves to another
> node. Agree, we can either have flag in resourceAssignment/marker
> interface/annotation on the computeAssignment method.
>
> Yeah using current state of each partition should work. One thing to note
> that is pending state is not guaranteed to reach (the transition might
> fail). In fact, all the in built rebalancers in Helix work off of current
> state.
>
> Yes, rebalancer is invoked on every change in the cluster. You can check
> for the event type and return the same assignment for some event types.
> Just basing on LiveNodes may not work if the participants fails to handle
> the transition.
>
> thanks,
> Kishore G
>
> On Thu, Jun 25, 2015 at 9:55 AM, Changgeng Li <li.changgeng@gmail.com>
> wrote:
>
>> I'm also trying to work around this for now by using currentState to
>> figure out the previous resource assignment, using pending state override
>> the current state of each partition. Do you think there is any potential
>> issue with this approach before prevResourceAssignment is persisted?
>>
>> I also have another question about when the rebalancer is invoked. My
>> test shows it's called a lot of times when add just one new node. Looks
>> like it's also called when the current state is changed. If so, we probably
>> need implement a shortcut that if livenodes are constraints are not
>> changed, just return the previous resource assignment.
>>
>>
>> On Thu, Jun 25, 2015 at 9:50 AM, Changgeng Li <li.changgeng@gmail.com>
>> wrote:
>>
>>> Actually I was going to ask whether prevAssignment is preserved if the
>>> controller moves to another node. A marker either for the rebalancer or for
>>> the returned resource assignment would work. It would be more flexible if
>>> there's a flag in the returned resource assignment.
>>>
>>>
>>>
>>> On Wed, Jun 24, 2015 at 11:44 PM, kishore g <g.kishore@gmail.com> wrote:
>>>
>>>> You are right. I think this is a bug. We need to preserve the previous
>>>> assignment and pass it on. Having said that we should not do this for all
>>>> types of rebalancers. Doing so will introduce additional latency(additional
>>>> writes to Zookeeper) and will impact fail over time. Most rebalancers also
>>>> tend to be idempotent, i.e they compute the same output given the same set
>>>> of inputs. This property in general is not error prone and makes debugging
>>>> easier.
>>>>
>>>> One idea is to have a marker interface Persistable or something and
>>>> only save it when the implementation implements this interface. The fix is
>>>> easy but love to hear your feedback.
>>>>
>>>> Also, the AutoRebalancer that comes with Helix, already provides the
>>>> features that you are planning to implement. May be you can clone it and
>>>> modify it to suite your needs.
>>>>
>>>>
>>>> https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/controller/rebalancer/FullAutoRebalancer.java
>>>>
>>>> https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/controller/strategy/AutoRebalanceStrategy.java
>>>>
>>>>
>>>> thanks,
>>>> Kishore G
>>>>
>>>> On Wed, Jun 24, 2015 at 11:13 PM, Changgeng Li <li.changgeng@gmail.com>
>>>> wrote:
>>>>
>>>>> yes, it's always null.
>>>>>
>>>>> actually I checked the code that parameter was never set.
>>>>>
>>>>> https://github.com/apache/helix/blob/acd902e2433f65e9864ccf49fcd1a04c36b1f206/helix-core/src/main/java/org/apache/helix/controller/stages/BestPossibleStateCalcStage.java#L222
>>>>>
>>>>> On Wed, Jun 24, 2015 at 10:41 PM, kishore g <g.kishore@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> I see that you haven't subscribed to mailing list yet. Please do
that
>>>>>> my sending an email to user-subscribe@helix.apache.org.
>>>>>>
>>>>>> I am also available on #apachehelix irc if you have additional
>>>>>> questions. http://helix.apache.org/IRC.html
>>>>>>
>>>>>>
>>>>>> ---------- Forwarded message ----------
>>>>>> From: kishore g <g.kishore@gmail.com>
>>>>>> Date: Wed, Jun 24, 2015 at 10:37 PM
>>>>>> Subject: Re: Customized rebalancer
>>>>>> To: "user@helix.apache.org" <user@helix.apache.org>
>>>>>>
>>>>>>
>>>>>> Hi Changgeng,
>>>>>>
>>>>>> I think the first invocation will always be null, subsequent
>>>>>> invocations should provide you the previous resource assignment.
Are you
>>>>>> saying its always null?.
>>>>>>
>>>>>> thanks,
>>>>>> Kishore G
>>>>>>
>>>>>> On Wed, Jun 24, 2015 at 3:42 PM, Changgeng Li <li.changgeng@gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> We have a use case that when adding or removing a new node, we
hope
>>>>>>> to minimize the shuffle of partitions between nodes. I'm trying
to
>>>>>>> implement a customized rebalancer calculating the new resource
assignment
>>>>>>> based on the previous resource assignment. When a new node is
added, just
>>>>>>> move some partitions from existing node to the new node, and
when a node is
>>>>>>> down, move the partitions on this node to the other nodes. Partitions
would
>>>>>>> not move between two nodes if the both status are not changed.
>>>>>>>
>>>>>>> The documentation says:
>>>>>>> In rebalance(), ... the third is the output of the previous
>>>>>>> invocation of this method (if supported) ...
>>>>>>>
>>>>>>> During my testing I found the previous resource assignment passed
in
>>>>>>> is always null. my question is what "if supported" means here?
Does the
>>>>>>> rebalancer need to support something?
>>>>>>>
>>>>>>>
>>>>>>> Any other suggestions are also welcome.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Changgeng
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message