incubator-kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: ZooKeeper-based Producer behavior before there are topics
Date Sun, 11 Dec 2011 18:27:13 GMT
Inder,

Zookeeper is used on the producer side for load balancing. It is used for
automatic detection of new and failed brokers.

Thanks,
Neha

On Sun, Dec 11, 2011 at 2:44 AM, Inder Pall <inder.pall@gmail.com> wrote:

> why do u want to integrate zk at producer side. can you please elaborate
> your usecase.
>
> is it for load balancing.
> On Dec 7, 2011 12:35 AM, "Jun Rao" <junrao@gmail.com> wrote:
> >
> > David,
> >
> > Could you also comment whether you want a programming api or a monitoring
> > api?
> >
> > Thanks,
> >
> > Jun
> >
> > On Tue, Dec 6, 2011 at 10:03 AM, David Ormsbee <dave@datadoghq.com>
> wrote:
> >
> > > Jun, Neha:
> > >
> > > Thank you for pointing me the right way on the Producer issue. :-)
> > >
> > > As for the metadata requests, I've filed a ticket here:
> > >
> > >  https://issues.apache.org/jira/browse/KAFKA-223
> > >
> > > Take care.
> > >
> > > Dave
> > >
> > > On Tue, Dec 6, 2011 at 12:05 PM, Neha Narkhede <
> neha.narkhede@gmail.com
> > > >wrote:
> > >
> > > > Hi Dave,
> > > >
> > > > >> Or should I just make the initial assumption
> > > > that all brokers have one partition, send messages to those to create
> the
> > > > topic on the brokers, and then readjust when the the brokers publish
> > > their
> > > > partitions-per-topic information to ZK after the initial writes?
> > > >
> > > > That's exactly what we do. You can take a look at
> > > > bootstrapWithExistingBrokers() in
> kafka.producer.ZKBrokerPartitionInfo.
> > > >
> > > > >> As an aside, is there any way to query metadata about the broker
> > > > directly?
> > > > So that a simple consumer might be able to ask a broker for what
> topics
> > > > they have, how many partitions, what version it's running, etc?
> > > >
> > > > Today, we don't have such APIs. But I was thinking we can have
> something
> > > > similar to the 4-letter commands in zookeeper.
> > > > For example,
> > > >
> > > > echo topics | nc broker-host broker-port      (Returns all topics
> with
> > > > number of partitions on each broker)
> > > > echo srvr | nc broker-host broker-port         (Returns basic broker
> > > stats
> > > > like version, number of topics etc)
> > > >
> > > > If the above functionality is useful, lets file a JIRA for it.
> > > >
> > > > Thanks
> > > > Neha
> > > >
> > > > On Tue, Dec 6, 2011 at 6:34 AM, David Ormsbee <dave@datadoghq.com>
> > > wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > I'm implementing ZooKeeper support in our Python client, and I was
> > > > > wondering what the expected behavior of the Producer is when the
> > > brokers
> > > > > are completely empty. At this point, the brokers have registered
> > > > themselves
> > > > > in /brokers/ids/[0..N], but there are no topics registered under
> > > > > /brokers/topics. If someone wants to create a Producer at this
> point
> > > and
> > > > > start sending messages to a new topic, is that an error because no
> > > > brokers
> > > > > are registered for that topic? Or should I just make the initial
> > > > assumption
> > > > > that all brokers have one partition, send messages to those to
> create
> > > the
> > > > > topic on the brokers, and then readjust when the the brokers
> publish
> > > > their
> > > > > partitions-per-topic information to ZK after the initial writes?
> > > > >
> > > > > I'm sure I'm missing something obvious here, since we'd run into
a
> > > > similar
> > > > > problem for new, empty brokers joining the pool.
> > > > >
> > > > > As an aside, is there any way to query metadata about the broker
> > > > directly?
> > > > > So that a simple consumer might be able to ask a broker for what
> topics
> > > > > they have, how many partitions, what version it's running, etc? Or
> is
> > > > that
> > > > > only ever published through ZooKeeper?
> > > > >
> > > > > Thank you.
> > > > >
> > > > > Dave
> > > > >
> > > >
> > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message