kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sriram Subramanian <srsubraman...@linkedin.com>
Subject Re: Question about auto-rebalancing
Date Sat, 12 Oct 2013 00:44:01 GMT
We already have a JIRA for auto rebalance. I would be working on this soon.

KAFKA-930 <https://issues.apache.org/jira/browse/KAFKA-930>



On 10/11/13 5:39 PM, "Guozhang Wang" <wangguoz@gmail.com> wrote:

>Hello Siyuan,
>
>For the automatic leader re-election, yes we are considering to make it
>work. Could you file a JIRA for this issue?
>
>For the high-level consumer's rebalancing logic, you can find it at
>
>https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-CanIpredictthere
>sultsoftheconsumerrebabalance%3F
>
>Guozhang
>
>
>On Fri, Oct 11, 2013 at 11:06 AM, hsy541@gmail.com <hsy541@gmail.com>
>wrote:
>
>> Hi Jun,
>>
>> Thanks for your reply, but in a real cluster, one broker could serve
>> different topics and different partitions, the simple consumer only has
>> knowledge of brokers that are available but it has no knowledge to
>>decide
>> which broker is best to pick up to consume messages.  If you don't
>>choose
>> carefully, multiple simple consumer might end up with reading from same
>> node which is definitely not good for performance.
>> Interesting thing is I find out there is
>> command kafka-preferred-replica-election.sh which will try to equally
>> distribute the leadership among different brokers, this is good that I
>>can
>> always let my simple consumer reads from leader broker(even it fails,
>>the
>> replica will pick up as leader which is fine).  But why don't kafka
>>cluster
>> run this command automatically when there is a broker change(up/down) in
>> the cluster so that the leadership can always be equally distributed
>>among
>> different brokers ASAP?  I think it's very good for simple consumer to
>> decide which broker is good to read from.
>>
>> Another question is I'm also curious how high-level consumer is
>>balanced. I
>> assume each high-level consumer know other consumers(int the same group)
>> which broker they read message from and it can try to avoid those
>>brokers
>> and to pick up a free one?  Is there a document for the balancing rule
>> among high-level consumer. Does it always guarantee that after several
>> leadership change/temporary broker fail, it can always equally
>>distribute
>> the read among the brokers. Basically I think it's nice to have a API to
>> let dev know which consumer reads from which broker otherwise I don't
>>know
>> anything behind the high-level consumer
>>
>> Thanks!
>>
>> Best,
>> Siyuan
>>
>
>
>
>-- 
>-- Guozhang


Mime
View raw message