incubator-kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: ZooKeeper-based Producer behavior before there are topics
Date Tue, 06 Dec 2011 19:05:29 GMT
David,

Could you also comment whether you want a programming api or a monitoring
api?

Thanks,

Jun

On Tue, Dec 6, 2011 at 10:03 AM, David Ormsbee <dave@datadoghq.com> wrote:

> Jun, Neha:
>
> Thank you for pointing me the right way on the Producer issue. :-)
>
> As for the metadata requests, I've filed a ticket here:
>
>  https://issues.apache.org/jira/browse/KAFKA-223
>
> Take care.
>
> Dave
>
> On Tue, Dec 6, 2011 at 12:05 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > Hi Dave,
> >
> > >> Or should I just make the initial assumption
> > that all brokers have one partition, send messages to those to create the
> > topic on the brokers, and then readjust when the the brokers publish
> their
> > partitions-per-topic information to ZK after the initial writes?
> >
> > That's exactly what we do. You can take a look at
> > bootstrapWithExistingBrokers() in kafka.producer.ZKBrokerPartitionInfo.
> >
> > >> As an aside, is there any way to query metadata about the broker
> > directly?
> > So that a simple consumer might be able to ask a broker for what topics
> > they have, how many partitions, what version it's running, etc?
> >
> > Today, we don't have such APIs. But I was thinking we can have something
> > similar to the 4-letter commands in zookeeper.
> > For example,
> >
> > echo topics | nc broker-host broker-port      (Returns all topics with
> > number of partitions on each broker)
> > echo srvr | nc broker-host broker-port         (Returns basic broker
> stats
> > like version, number of topics etc)
> >
> > If the above functionality is useful, lets file a JIRA for it.
> >
> > Thanks
> > Neha
> >
> > On Tue, Dec 6, 2011 at 6:34 AM, David Ormsbee <dave@datadoghq.com>
> wrote:
> >
> > > Hi folks,
> > >
> > > I'm implementing ZooKeeper support in our Python client, and I was
> > > wondering what the expected behavior of the Producer is when the
> brokers
> > > are completely empty. At this point, the brokers have registered
> > themselves
> > > in /brokers/ids/[0..N], but there are no topics registered under
> > > /brokers/topics. If someone wants to create a Producer at this point
> and
> > > start sending messages to a new topic, is that an error because no
> > brokers
> > > are registered for that topic? Or should I just make the initial
> > assumption
> > > that all brokers have one partition, send messages to those to create
> the
> > > topic on the brokers, and then readjust when the the brokers publish
> > their
> > > partitions-per-topic information to ZK after the initial writes?
> > >
> > > I'm sure I'm missing something obvious here, since we'd run into a
> > similar
> > > problem for new, empty brokers joining the pool.
> > >
> > > As an aside, is there any way to query metadata about the broker
> > directly?
> > > So that a simple consumer might be able to ask a broker for what topics
> > > they have, how many partitions, what version it's running, etc? Or is
> > that
> > > only ever published through ZooKeeper?
> > >
> > > Thank you.
> > >
> > > Dave
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message