helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore g <g.kish...@gmail.com>
Subject Re: Customized rebalancer
Date Thu, 25 Jun 2015 06:44:47 GMT
You are right. I think this is a bug. We need to preserve the previous
assignment and pass it on. Having said that we should not do this for all
types of rebalancers. Doing so will introduce additional latency(additional
writes to Zookeeper) and will impact fail over time. Most rebalancers also
tend to be idempotent, i.e they compute the same output given the same set
of inputs. This property in general is not error prone and makes debugging
easier.

One idea is to have a marker interface Persistable or something and only
save it when the implementation implements this interface. The fix is easy
but love to hear your feedback.

Also, the AutoRebalancer that comes with Helix, already provides the
features that you are planning to implement. May be you can clone it and
modify it to suite your needs.

https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/controller/rebalancer/FullAutoRebalancer.java
https://github.com/apache/helix/blob/master/helix-core/src/main/java/org/apache/helix/controller/strategy/AutoRebalanceStrategy.java


thanks,
Kishore G

On Wed, Jun 24, 2015 at 11:13 PM, Changgeng Li <li.changgeng@gmail.com>
wrote:

> yes, it's always null.
>
> actually I checked the code that parameter was never set.
>
> https://github.com/apache/helix/blob/acd902e2433f65e9864ccf49fcd1a04c36b1f206/helix-core/src/main/java/org/apache/helix/controller/stages/BestPossibleStateCalcStage.java#L222
>
> On Wed, Jun 24, 2015 at 10:41 PM, kishore g <g.kishore@gmail.com> wrote:
>
>> I see that you haven't subscribed to mailing list yet. Please do that my
>> sending an email to user-subscribe@helix.apache.org.
>>
>> I am also available on #apachehelix irc if you have additional questions.
>> http://helix.apache.org/IRC.html
>>
>>
>> ---------- Forwarded message ----------
>> From: kishore g <g.kishore@gmail.com>
>> Date: Wed, Jun 24, 2015 at 10:37 PM
>> Subject: Re: Customized rebalancer
>> To: "user@helix.apache.org" <user@helix.apache.org>
>>
>>
>> Hi Changgeng,
>>
>> I think the first invocation will always be null, subsequent invocations
>> should provide you the previous resource assignment. Are you saying its
>> always null?.
>>
>> thanks,
>> Kishore G
>>
>> On Wed, Jun 24, 2015 at 3:42 PM, Changgeng Li <li.changgeng@gmail.com>
>> wrote:
>>
>>> Hello,
>>>
>>> We have a use case that when adding or removing a new node, we hope to
>>> minimize the shuffle of partitions between nodes. I'm trying to implement a
>>> customized rebalancer calculating the new resource assignment based on the
>>> previous resource assignment. When a new node is added, just move some
>>> partitions from existing node to the new node, and when a node is down,
>>> move the partitions on this node to the other nodes. Partitions would not
>>> move between two nodes if the both status are not changed.
>>>
>>> The documentation says:
>>> In rebalance(), ... the third is the output of the previous invocation
>>> of this method (if supported) ...
>>>
>>> During my testing I found the previous resource assignment passed in is
>>> always null. my question is what "if supported" means here? Does the
>>> rebalancer need to support something?
>>>
>>>
>>> Any other suggestions are also welcome.
>>>
>>> Thanks,
>>> Changgeng
>>>
>>
>>
>>
>

Mime
View raw message