Mailing-List: contact kafka-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: kafka-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of neha.narkhede@gmail.com
 designates 209.85.220.175 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAAR2Ka3RpeXwKrj11Egd99p3O6d21xtWBs6exwBA97BQT5eDkw@mail.gmail.com>
References: 
 <CALStshY02xT7b2n-=DjTF_QifEzjxWnCvmgRhSu8v8-ibbSGyQ@mail.gmail.com>
	<CAOG_4QaYB8HuCJb8gc9=JnQBMHKh8twT9ARktLFhTUA+E+3kOg@mail.gmail.com>
	<CALStshbQrniBMjQzX3hzBsyq+oEpbF_7JV24v0KX7kshZCsMcA@mail.gmail.com>
	<CAFbh0Q1AttuCOaR6k7NQXsUSi9wTmmqXX+XfFhMVXZyfTLMCpw@mail.gmail.com>
	<CAAR2Ka3RpeXwKrj11Egd99p3O6d21xtWBs6exwBA97BQT5eDkw@mail.gmail.com>
Date: Sun, 11 Dec 2011 10:27:13 -0800
Message-ID: 
 <CAOG_4QaWuz_xznJJp9M7oy1osW1EHFPTyYCVEXZZFZFa8if66w@mail.gmail.com>
Subject: Re: ZooKeeper-based Producer behavior before there are topics
From: Neha Narkhede <neha.narkhede@gmail.com>
To: kafka-dev@incubator.apache.org
Content-Type: multipart/alternative; boundary=20cf307d04dcc0c46d04b3d52a4f

--20cf307d04dcc0c46d04b3d52a4f
Content-Type: text/plain; charset=ISO-8859-1

Inder,

Zookeeper is used on the producer side for load balancing. It is used for
automatic detection of new and failed brokers.

Thanks,
Neha

On Sun, Dec 11, 2011 at 2:44 AM, Inder Pall <inder.pall@gmail.com> wrote:

> why do u want to integrate zk at producer side. can you please elaborate
> your usecase.
>
> is it for load balancing.
> On Dec 7, 2011 12:35 AM, "Jun Rao" <junrao@gmail.com> wrote:
> >
> > David,
> >
> > Could you also comment whether you want a programming api or a monitoring
> > api?
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Dec 6, 2011 at 10:03 AM, David Ormsbee <dave@datadoghq.com>
> wrote:
> >
> > > Jun, Neha:
> > >
> > > Thank you for pointing me the right way on the Producer issue. :-)
> > >
> > > As for the metadata requests, I've filed a ticket here:
> > >
> > >  https://issues.apache.org/jira/browse/KAFKA-223
> > >
> > > Take care.
> > >
> > > Dave
> > >
> > > On Tue, Dec 6, 2011 at 12:05 PM, Neha Narkhede <
> neha.narkhede@gmail.com
> > > >wrote:
> > >
> > > > Hi Dave,
> > > >
> > > > >> Or should I just make the initial assumption
> > > > that all brokers have one partition, send messages to those to create
> the
> > > > topic on the brokers, and then readjust when the the brokers publish
> > > their
> > > > partitions-per-topic information to ZK after the initial writes?
> > > >
> > > > That's exactly what we do. You can take a look at
> > > > bootstrapWithExistingBrokers() in
> kafka.producer.ZKBrokerPartitionInfo.
> > > >
> > > > >> As an aside, is there any way to query metadata about the broker
> > > > directly?
> > > > So that a simple consumer might be able to ask a broker for what
> topics
> > > > they have, how many partitions, what version it's running, etc?
> > > >
> > > > Today, we don't have such APIs. But I was thinking we can have
> something
> > > > similar to the 4-letter commands in zookeeper.
> > > > For example,
> > > >
> > > > echo topics | nc broker-host broker-port      (Returns all topics
> with
> > > > number of partitions on each broker)
> > > > echo srvr | nc broker-host broker-port         (Returns basic broker
> > > stats
> > > > like version, number of topics etc)
> > > >
> > > > If the above functionality is useful, lets file a JIRA for it.
> > > >
> > > > Thanks
> > > > Neha
> > > >
> > > > On Tue, Dec 6, 2011 at 6:34 AM, David Ormsbee <dave@datadoghq.com>
> > > wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > I'm implementing ZooKeeper support in our Python client, and I was
> > > > > wondering what the expected behavior of the Producer is when the
> > > brokers
> > > > > are completely empty. At this point, the brokers have registered
> > > > themselves
> > > > > in /brokers/ids/[0..N], but there are no topics registered under
> > > > > /brokers/topics. If someone wants to create a Producer at this
> point
> > > and
> > > > > start sending messages to a new topic, is that an error because no
> > > > brokers
> > > > > are registered for that topic? Or should I just make the initial
> > > > assumption
> > > > > that all brokers have one partition, send messages to those to
> create
> > > the
> > > > > topic on the brokers, and then readjust when the the brokers
> publish
> > > > their
> > > > > partitions-per-topic information to ZK after the initial writes?
> > > > >
> > > > > I'm sure I'm missing something obvious here, since we'd run into a
> > > > similar
> > > > > problem for new, empty brokers joining the pool.
> > > > >
> > > > > As an aside, is there any way to query metadata about the broker
> > > > directly?
> > > > > So that a simple consumer might be able to ask a broker for what
> topics
> > > > > they have, how many partitions, what version it's running, etc? Or
> is
> > > > that
> > > > > only ever published through ZooKeeper?
> > > > >
> > > > > Thank you.
> > > > >
> > > > > Dave
> > > > >
> > > >
> > >
>

--20cf307d04dcc0c46d04b3d52a4f--