helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore g <g.kish...@gmail.com>
Subject Re: Balancing out skews in FULL_AUTO mode with built-in rebalancer
Date Tue, 15 Mar 2016 21:01:13 GMT
1) I am guessing it gets overriden by other logic in
computePartitionAssignment(..), the end assignment is still skewed.

What is the logic you are referring to?

Can you print the assignment count for your use case?


thanks,
Kishore G

On Tue, Mar 15, 2016 at 1:45 PM, Vinoth Chandar <vinoth@uber.com> wrote:

> Hi guys,
>
> We are hitting a fairly known issue where we have 100s of resource with <
> 8 resources spreading across 10 servers and the built-in assignment always
> assigns partitions from first to last, resulting in heavy skew for a few
> nodes.
>
> Chatted with Kishore offline and made a patch as here
> <https://gist.github.com/vinothchandar/e8837df301501f85e257>.Tested with
> 5 resources with 2 partitions each across 8 servers, logging out the
> nodeShift & ultimate index picked does indicate that we choose servers
> other than the first two, which is good
>
> But
> 1) I am guessing it gets overriden by other logic in
> computePartitionAssignment(..), the end assignment is still skewed.
> 2) Even with murmur hash, there is some skew on the nodeshift, which needs
> to ironed out.
>
> I will keep chipping at this.. Any feedback appreciated
>
> Thanks
> Vinoth
>

Mime
View raw message