incubator-kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Inder Pall <inder.p...@gmail.com>
Subject Re: ZooKeeper-based Producer behavior before there are topics
Date Sun, 11 Dec 2011 10:44:12 GMT
why do u want to integrate zk at producer side. can you please elaborate
your usecase.

is it for load balancing.
On Dec 7, 2011 12:35 AM, "Jun Rao" <junrao@gmail.com> wrote:
>
> David,
>
> Could you also comment whether you want a programming api or a monitoring
> api?
>
> Thanks,
>
> Jun
>
> On Tue, Dec 6, 2011 at 10:03 AM, David Ormsbee <dave@datadoghq.com> wrote:
>
> > Jun, Neha:
> >
> > Thank you for pointing me the right way on the Producer issue. :-)
> >
> > As for the metadata requests, I've filed a ticket here:
> >
> >  https://issues.apache.org/jira/browse/KAFKA-223
> >
> > Take care.
> >
> > Dave
> >
> > On Tue, Dec 6, 2011 at 12:05 PM, Neha Narkhede <neha.narkhede@gmail.com
> > >wrote:
> >
> > > Hi Dave,
> > >
> > > >> Or should I just make the initial assumption
> > > that all brokers have one partition, send messages to those to create
the
> > > topic on the brokers, and then readjust when the the brokers publish
> > their
> > > partitions-per-topic information to ZK after the initial writes?
> > >
> > > That's exactly what we do. You can take a look at
> > > bootstrapWithExistingBrokers() in
kafka.producer.ZKBrokerPartitionInfo.
> > >
> > > >> As an aside, is there any way to query metadata about the broker
> > > directly?
> > > So that a simple consumer might be able to ask a broker for what
topics
> > > they have, how many partitions, what version it's running, etc?
> > >
> > > Today, we don't have such APIs. But I was thinking we can have
something
> > > similar to the 4-letter commands in zookeeper.
> > > For example,
> > >
> > > echo topics | nc broker-host broker-port      (Returns all topics with
> > > number of partitions on each broker)
> > > echo srvr | nc broker-host broker-port         (Returns basic broker
> > stats
> > > like version, number of topics etc)
> > >
> > > If the above functionality is useful, lets file a JIRA for it.
> > >
> > > Thanks
> > > Neha
> > >
> > > On Tue, Dec 6, 2011 at 6:34 AM, David Ormsbee <dave@datadoghq.com>
> > wrote:
> > >
> > > > Hi folks,
> > > >
> > > > I'm implementing ZooKeeper support in our Python client, and I was
> > > > wondering what the expected behavior of the Producer is when the
> > brokers
> > > > are completely empty. At this point, the brokers have registered
> > > themselves
> > > > in /brokers/ids/[0..N], but there are no topics registered under
> > > > /brokers/topics. If someone wants to create a Producer at this point
> > and
> > > > start sending messages to a new topic, is that an error because no
> > > brokers
> > > > are registered for that topic? Or should I just make the initial
> > > assumption
> > > > that all brokers have one partition, send messages to those to
create
> > the
> > > > topic on the brokers, and then readjust when the the brokers publish
> > > their
> > > > partitions-per-topic information to ZK after the initial writes?
> > > >
> > > > I'm sure I'm missing something obvious here, since we'd run into a
> > > similar
> > > > problem for new, empty brokers joining the pool.
> > > >
> > > > As an aside, is there any way to query metadata about the broker
> > > directly?
> > > > So that a simple consumer might be able to ask a broker for what
topics
> > > > they have, how many partitions, what version it's running, etc? Or
is
> > > that
> > > > only ever published through ZooKeeper?
> > > >
> > > > Thank you.
> > > >
> > > > Dave
> > > >
> > >
> >

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message