helix-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore g <g.kish...@gmail.com>
Subject Re: Can Helix balance many small resources on a large cluster?
Date Fri, 11 Nov 2016 00:52:27 GMT
Hi Michael,

What you are seeing is right. Helix as of now does not consider balancing
across resources. This is something we would love to have but never got to

One workaround is the set max partitions per node constraint. For example,
if you set max partitions=1 then each node should have only one task. But
this has the drawback that if 1 node fails, it won't assign that task to
any of the remaining nodes unless you remove the constraint after the tasks
are evenly distributed.

Kishore G

On Thu, Nov 10, 2016 at 3:13 PM, Michael Craig <mcraig@box.com> wrote:

> This is related to a use case mentioned in a prior thread:
> > .. manage a set of ad-hoc tasks across a cluster of machines. Each task
> is a separate resource with 1 partition and 1 replica, intended to run on
> exactly 1 node.
> Can Helix balance many small resources like this? E.g. 3 resources {R1,
> R2, R3} each with 1 partition and 1 replica, on 3 nodes:
> N1: {R1}, N2: {R2}, N3: {R3}
> Or does Helix balance the resources independently, putting all partitions
> on the same node? I.e.
> N1: {R1, R2, R3}, N2: {}, N3: {}
> I've observed the latter behavior, though I'm wondering if I'm misusing
> Helix.

View raw message