Mailing-List: contact user-help@helix.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@helix.apache.org
Received-SPF: pass (nike.apache.org: domain of g.kishore@gmail.com designates
 74.125.82.49 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CA+TL_nkEo__xK0X1sei-N1L4rOjCU3C2DgOiCFpyqvv4BWE7pg@mail.gmail.com>
References: 
 <CA+TL_nkEo__xK0X1sei-N1L4rOjCU3C2DgOiCFpyqvv4BWE7pg@mail.gmail.com>
Date: Thu, 4 Dec 2014 14:50:34 -0800
Message-ID: 
 <CABaj-Qanw3N1zOtsoJb1WffiSrtH3Xw8urUCWbknjXd=otbUVw@mail.gmail.com>
Subject: Re: managing a Kafka consumer group using Helix
From: kishore g <g.kishore@gmail.com>
To: "user@helix.apache.org" <user@helix.apache.org>
Content-Type: multipart/alternative; boundary=047d7bfced1cbecdc705096bc959

--047d7bfced1cbecdc705096bc959
Content-Type: text/plain; charset=UTF-8

you bet :-)

Kafka had its own consumer group implementation before Helix was
implemented.

We have implemented a similar consumer group pattern for Databus, so its
definitely possible to extend it to Kafka.

Regarding your questions.

1. Yes, current rebalancer handles one resource at a time, we had some
discussion about having a composite rebalancer that rebalances multiple
resources at once. Even without this feature, its possible to read the
entire cluster data from the rebalancer and make the decision considering
other resources.
2. Probably not a good idea to put all topics in one resource even with
bucketing support.
3. Theoretically, group of changes does not matter if its within a resource
v/s across resource. Its the overall change that matters.
4. We have gone up to 2000 resources across the cluster. This is not using
one ZNode. I dont think its a good idea to have one ZNode for all resources.


thanks,
Kishore G


On Wed, Dec 3, 2014 at 10:57 AM, vlad.gm@gmail.com <vlad.gm@gmail.com>
wrote:

>
> Dear all,
>
> I am sure the following question appeared inside Linkedin before :)
>
> We would like to manage a Kafka consumer group using Helix, that is have
> multiple consumer instances and assign topics and their partitions among
> the consumers automatically. The consumer group would use a whitelist to
> select the topics, which means that the topic/partitions list is dynamic
> and can change quite frequently. I can see each topic mapping to a Helix
> resource or, alternatively, using a single Helix resource to handle all
> topics. We are most likely to use a custom rebalancer in order to use
> throughput metrics in order to balance traffic, not partition count.
>
> Here are a few questions:
> 1) If we are to use a resource per topic, would we be able to later on
> jointly rebalance multiple resources at once? The current rebalancer
> callback seems to handle a single resource. Would we have to actually
> manage the multiple resources in the background and just use the callback
> when we are asked what to do with that resource?
> 2) If we are to put all topics and their partitions in a single resource
> we are likely to quickly go over the amount of data that can be stored in a
> ZK node. I remember that buckets can help with that problem. Can the number
> of buckets increase dynamically with the number of partitions?
> 3) How big of a problem would it be to have an environment in which the
> group of administered partitions changes quite often? I guess that with one
> resource per topic this would not be a big issue, however it might be a
> problem with a single resource for all topics.
> 4) Is there a limit on the number of resources that can be stored in a
> single cluster, because of the amount of data that can be stored in a
> single ZK node?
>
> Regards,
> Vlad
>

--047d7bfced1cbecdc705096bc959
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div><div><div><div><div><div><div>you bet :-)<br><br></di=
v>Kafka had its own consumer group implementation before Helix was implemen=
ted. <br><br></div>We have implemented a similar consumer group pattern for=
 Databus, so its definitely possible to extend it to Kafka.<br><br></div>Re=
garding your questions.<br><br></div>1. Yes, current rebalancer handles one=
 resource at a time, we had some discussion about having a composite rebala=
ncer that rebalances multiple resources at once. Even without this feature,=
 its possible to read the entire cluster data from the rebalancer and make =
the decision considering other resources.<br></div>2. Probably not a good i=
dea to put all topics in one resource even with bucketing support.<br></div=
>3. Theoretically, group of changes does not matter if its within a resourc=
e v/s across resource. Its the overall change that matters.<br></div>4. We =
have gone up to 2000 resources across the cluster. This is not using one ZN=
ode. I dont think its a good idea to have one ZNode for all resources.<br><=
br><br>thanks,<br>Kishore G<br><div><div><div><div><div><div><div><div><br>=
<br></div></div></div></div></div></div></div></div></div><div class=3D"gma=
il_extra"><br><div class=3D"gmail_quote">On Wed, Dec 3, 2014 at 10:57 AM, <=
a href=3D"mailto:vlad.gm@gmail.com">vlad.gm@gmail.com</a> <span dir=3D"ltr"=
>&lt;<a href=3D"mailto:vlad.gm@gmail.com" target=3D"_blank">vlad.gm@gmail.c=
om</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" style=3D"marg=
in:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=3D"ltr"=
><br><div>Dear all,</div><div><br></div><div>I am sure the following questi=
on appeared inside Linkedin before :)</div><div><br></div><div>We would lik=
e to manage a Kafka consumer group using Helix, that is have multiple consu=
mer instances and assign topics and their partitions among the consumers au=
tomatically. The consumer group would use a whitelist to select the topics,=
 which means that the topic/partitions list is dynamic and can change quite=
 frequently. I can see each topic mapping to a Helix resource or, alternati=
vely, using a single Helix resource to handle all topics. We are most likel=
y to use a custom rebalancer in order to use throughput metrics in order to=
 balance traffic, not partition count.=C2=A0</div><div><br></div><div>Here =
are a few questions:</div><div>1) If we are to use a resource per topic, wo=
uld we be able to later on jointly rebalance multiple resources at once? Th=
e current rebalancer callback seems to handle a single resource. Would we h=
ave to actually manage the multiple resources in the background and just us=
e the callback when we are asked what to do with that resource?</div><div>2=
) If we are to put all topics and their partitions in a single resource we =
are likely to quickly go over the amount of data that can be stored in a ZK=
 node. I remember that buckets can help with that problem. Can the number o=
f buckets increase dynamically with the number of partitions?</div><div>3) =
How big of a problem would it be to have an environment in which the group =
of administered partitions changes quite often? I guess that with one resou=
rce per topic this would not be a big issue, however it might be a problem =
with a single resource for all topics.</div><div>4) Is there a limit on the=
 number of resources that can be stored in a single cluster, because of the=
 amount of data that can be stored in a single ZK node?</div><div><br></div=
><div>Regards,</div><div>Vlad</div></div>
</blockquote></div><br></div>

--047d7bfced1cbecdc705096bc959--