incubator-kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Ormsbee <d...@datadoghq.com>
Subject Re: ZooKeeper-based Producer behavior before there are topics
Date Tue, 06 Dec 2011 18:03:23 GMT
Jun, Neha:

Thank you for pointing me the right way on the Producer issue. :-)

As for the metadata requests, I've filed a ticket here:

  https://issues.apache.org/jira/browse/KAFKA-223

Take care.

Dave

On Tue, Dec 6, 2011 at 12:05 PM, Neha Narkhede <neha.narkhede@gmail.com>wrote:

> Hi Dave,
>
> >> Or should I just make the initial assumption
> that all brokers have one partition, send messages to those to create the
> topic on the brokers, and then readjust when the the brokers publish their
> partitions-per-topic information to ZK after the initial writes?
>
> That's exactly what we do. You can take a look at
> bootstrapWithExistingBrokers() in kafka.producer.ZKBrokerPartitionInfo.
>
> >> As an aside, is there any way to query metadata about the broker
> directly?
> So that a simple consumer might be able to ask a broker for what topics
> they have, how many partitions, what version it's running, etc?
>
> Today, we don't have such APIs. But I was thinking we can have something
> similar to the 4-letter commands in zookeeper.
> For example,
>
> echo topics | nc broker-host broker-port      (Returns all topics with
> number of partitions on each broker)
> echo srvr | nc broker-host broker-port         (Returns basic broker stats
> like version, number of topics etc)
>
> If the above functionality is useful, lets file a JIRA for it.
>
> Thanks
> Neha
>
> On Tue, Dec 6, 2011 at 6:34 AM, David Ormsbee <dave@datadoghq.com> wrote:
>
> > Hi folks,
> >
> > I'm implementing ZooKeeper support in our Python client, and I was
> > wondering what the expected behavior of the Producer is when the brokers
> > are completely empty. At this point, the brokers have registered
> themselves
> > in /brokers/ids/[0..N], but there are no topics registered under
> > /brokers/topics. If someone wants to create a Producer at this point and
> > start sending messages to a new topic, is that an error because no
> brokers
> > are registered for that topic? Or should I just make the initial
> assumption
> > that all brokers have one partition, send messages to those to create the
> > topic on the brokers, and then readjust when the the brokers publish
> their
> > partitions-per-topic information to ZK after the initial writes?
> >
> > I'm sure I'm missing something obvious here, since we'd run into a
> similar
> > problem for new, empty brokers joining the pool.
> >
> > As an aside, is there any way to query metadata about the broker
> directly?
> > So that a simple consumer might be able to ask a broker for what topics
> > they have, how many partitions, what version it's running, etc? Or is
> that
> > only ever published through ZooKeeper?
> >
> > Thank you.
> >
> > Dave
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message