incubator-kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: Adding a new broker behavior
Date Mon, 17 Oct 2011 16:21:03 GMT
Inder,

>From the producer side,VIP and the ZK Producer are mutually exclusive options.
You use either one or the other.

> 1. When a new broker comes in in the cluster, it would register in broker
> registry but won't be subscribed to any topics.

If you are using a VIP, then it will route some of the new requests to the new
broker automatically. If you are using the ZK Producer, it will get
notified once the
new broker registers itself in ZK, and it will start routing new
requests to the broker
according to the partitioner you use (Default is random partitioning,
which is similar to
VIP behavior)

> 3. If we are using a VIP to front brokers, how do they get all the
> information from ZK about what topics/partitions should go to which broker
> for appropriate routing.

As I said, if you are using VIP, then brokers are not "discovered"
through ZK watchers.
The VIP should have a TCP healthcheck on the Kafka port, and all
brokers that have
Kafka running on that port, will be involved in receiving Kafka data.
The VIP randomly
distributes produce requests amongst the live brokers.

Thanks,
Neha


On Mon, Oct 17, 2011 at 9:01 AM, Inder Pall <inder.pall@gmail.com> wrote:
> Jun,
>
> appreciate your prompt response.
>
> need some more inputs in understanding this better.
> Is the following understanding appropriate.
>
> 1. When a new broker comes in in the cluster, it would register in broker
> registry but won't be subscribed to any topics.
> 2. re-balance triggered by consumers will get some partitions of existing
> topics to this broker.
> 3. If we are using a VIP to front brokers, how do they get all the
> information from ZK about what topics/partitions should go to which broker
> for appropriate routing.
>
> Thanks,
> - Inder
> Yahoo India R&D.
>
> On Mon, Oct 17, 2011 at 8:21 PM, Jun Rao <junrao@gmail.com> wrote:
>
>> Inder,
>>
>> 1. Existing data doesn't get moved to the new broker. Some of the new data
>> will get routed to the new broker once the producer recognizes the new
>> broker (either through ZK-based producer or vip).
>>
>> 2. Yes. Once the new broker gets data for a new topic, consumers will be
>> notified and start rebalance.
>>
>> Thanks,
>>
>> Jun
>>
>> On Sun, Oct 16, 2011 at 10:29 PM, Inder Pall <inder.pall@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > I am trying to understand kafka by going through documents/wiki's for
>> > usage.
>> > Please pardon if the dev mailing list isn't appropriate to be in the TO
>> >  for
>> > this query.
>> >
>> > Couple of queries.
>> >
>> > 1. If you  add new broker boxes, how will existing topics and their
>> > partitions be redistributed on the new broker.
>> > 2. Does [1] trigger a re-balance process via watchers in which case if
>> you
>> > wanna redistribute only certain event types on the new broker, can it be
>> > achieved dynamically.
>> >
>> >
>> > -- Inder
>> >
>>
>
>
>
> --
> -- Inder
>

Mime
View raw message