Mailing-List: contact kafka-users-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: kafka-users@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of neha.narkhede@gmail.com
 designates 209.85.220.175 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAAR2Ka2d7CqzRDgoT8KB4r7uCbLYDrnHtQbGz3BPuw8HE2WQXg@mail.gmail.com>
References: 
 <CAAR2Ka3zEU4wt=gOtxHUv1CEPaaEFCeha4WC0_X8tkgtJjOAeA@mail.gmail.com>
	<CAFbh0Q367r27tNMeSBTmKNnt1POvh-ZWD_Bb1TVaSNEhT9Qa8Q@mail.gmail.com>
	<CAAR2Ka2d7CqzRDgoT8KB4r7uCbLYDrnHtQbGz3BPuw8HE2WQXg@mail.gmail.com>
Date: Mon, 17 Oct 2011 09:21:03 -0700
Message-ID: 
 <CAOG_4QaBgGgF+6rxwSjLv81KBOdtPVvs6+MudBSi8YEi_gk4qw@mail.gmail.com>
Subject: Re: Adding a new broker behavior
From: Neha Narkhede <neha.narkhede@gmail.com>
To: kafka-users@incubator.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Inder,

>From the producer side,VIP and the ZK Producer are mutually exclusive optio=
ns.
You use either one or the other.

> 1. When a new broker comes in in the cluster, it would register in broker
> registry but won't be subscribed to any topics.

If you are using a VIP, then it will route some of the new requests to the =
new
broker automatically. If you are using the ZK Producer, it will get
notified once the
new broker registers itself in ZK, and it will start routing new
requests to the broker
according to the partitioner you use (Default is random partitioning,
which is similar to
VIP behavior)

> 3. If we are using a VIP to front brokers, how do they get all the
> information from ZK about what topics/partitions should go to which broke=
r
> for appropriate routing.

As I said, if you are using VIP, then brokers are not "discovered"
through ZK watchers.
The VIP should have a TCP healthcheck on the Kafka port, and all
brokers that have
Kafka running on that port, will be involved in receiving Kafka data.
The VIP randomly
distributes produce requests amongst the live brokers.

Thanks,
Neha


On Mon, Oct 17, 2011 at 9:01 AM, Inder Pall <inder.pall@gmail.com> wrote:
> Jun,
>
> appreciate your prompt response.
>
> need some more inputs in understanding this better.
> Is the following understanding appropriate.
>
> 1. When a new broker comes in in the cluster, it would register in broker
> registry but won't be subscribed to any topics.
> 2. re-balance triggered by consumers will get some partitions of existing
> topics to this broker.
> 3. If we are using a VIP to front brokers, how do they get all the
> information from ZK about what topics/partitions should go to which broke=
r
> for appropriate routing.
>
> Thanks,
> - Inder
> Yahoo India R&D.
>
> On Mon, Oct 17, 2011 at 8:21 PM, Jun Rao <junrao@gmail.com> wrote:
>
>> Inder,
>>
>> 1. Existing data doesn't get moved to the new broker. Some of the new da=
ta
>> will get routed to the new broker once the producer recognizes the new
>> broker (either through ZK-based producer or vip).
>>
>> 2. Yes. Once the new broker gets data for a new topic, consumers will be
>> notified and start rebalance.
>>
>> Thanks,
>>
>> Jun
>>
>> On Sun, Oct 16, 2011 at 10:29 PM, Inder Pall <inder.pall@gmail.com> wrot=
e:
>>
>> > Hi,
>> >
>> > I am trying to understand kafka by going through documents/wiki's for
>> > usage.
>> > Please pardon if the dev mailing list isn't appropriate to be in the T=
O
>> > =A0for
>> > this query.
>> >
>> > Couple of queries.
>> >
>> > 1. If you =A0add new broker boxes, how will existing topics and their
>> > partitions be redistributed on the new broker.
>> > 2. Does [1] trigger a re-balance process via watchers in which case if
>> you
>> > wanna redistribute only certain event types on the new broker, can it =
be
>> > achieved dynamically.
>> >
>> >
>> > -- Inder
>> >
>>
>
>
>
> --
> -- Inder
>