Mailing-List: contact user-help@helix.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@helix.apache.org
Received-SPF: pass (athena.apache.org: domain of vlad.gm@gmail.com designates
 209.85.160.180 as permitted sender)
MIME-Version: 1.0
Date: Wed, 3 Dec 2014 10:57:23 -0800
Message-ID: 
 <CA+TL_nkEo__xK0X1sei-N1L4rOjCU3C2DgOiCFpyqvv4BWE7pg@mail.gmail.com>
Subject: managing a Kafka consumer group using Helix
From: "vlad.gm@gmail.com" <vlad.gm@gmail.com>
To: "user@helix.apache.org" <user@helix.apache.org>
Content-Type: multipart/alternative; boundary=001a113a57d2f54e280509546990

--001a113a57d2f54e280509546990
Content-Type: text/plain; charset=UTF-8

Dear all,

I am sure the following question appeared inside Linkedin before :)

We would like to manage a Kafka consumer group using Helix, that is have
multiple consumer instances and assign topics and their partitions among
the consumers automatically. The consumer group would use a whitelist to
select the topics, which means that the topic/partitions list is dynamic
and can change quite frequently. I can see each topic mapping to a Helix
resource or, alternatively, using a single Helix resource to handle all
topics. We are most likely to use a custom rebalancer in order to use
throughput metrics in order to balance traffic, not partition count.

Here are a few questions:
1) If we are to use a resource per topic, would we be able to later on
jointly rebalance multiple resources at once? The current rebalancer
callback seems to handle a single resource. Would we have to actually
manage the multiple resources in the background and just use the callback
when we are asked what to do with that resource?
2) If we are to put all topics and their partitions in a single resource we
are likely to quickly go over the amount of data that can be stored in a ZK
node. I remember that buckets can help with that problem. Can the number of
buckets increase dynamically with the number of partitions?
3) How big of a problem would it be to have an environment in which the
group of administered partitions changes quite often? I guess that with one
resource per topic this would not be a big issue, however it might be a
problem with a single resource for all topics.
4) Is there a limit on the number of resources that can be stored in a
single cluster, because of the amount of data that can be stored in a
single ZK node?

Regards,
Vlad

--001a113a57d2f54e280509546990
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><br><div>Dear all,</div><div><br></div><div>I am sure the =
following question appeared inside Linkedin before :)</div><div><br></div><=
div>We would like to manage a Kafka consumer group using Helix, that is hav=
e multiple consumer instances and assign topics and their partitions among =
the consumers automatically. The consumer group would use a whitelist to se=
lect the topics, which means that the topic/partitions list is dynamic and =
can change quite frequently. I can see each topic mapping to a Helix resour=
ce or, alternatively, using a single Helix resource to handle all topics. W=
e are most likely to use a custom rebalancer in order to use throughput met=
rics in order to balance traffic, not partition count.=C2=A0</div><div><br>=
</div><div>Here are a few questions:</div><div>1) If we are to use a resour=
ce per topic, would we be able to later on jointly rebalance multiple resou=
rces at once? The current rebalancer callback seems to handle a single reso=
urce. Would we have to actually manage the multiple resources in the backgr=
ound and just use the callback when we are asked what to do with that resou=
rce?</div><div>2) If we are to put all topics and their partitions in a sin=
gle resource we are likely to quickly go over the amount of data that can b=
e stored in a ZK node. I remember that buckets can help with that problem. =
Can the number of buckets increase dynamically with the number of partition=
s?</div><div>3) How big of a problem would it be to have an environment in =
which the group of administered partitions changes quite often? I guess tha=
t with one resource per topic this would not be a big issue, however it mig=
ht be a problem with a single resource for all topics.</div><div>4) Is ther=
e a limit on the number of resources that can be stored in a single cluster=
, because of the amount of data that can be stored in a single ZK node?</di=
v><div><br></div><div>Regards,</div><div>Vlad</div></div>

--001a113a57d2f54e280509546990--