incubator-kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Neha Narkhede <neha.narkh...@gmail.com>
Subject Re: ZooKeeper-based Producer behavior before there are topics
Date Tue, 06 Dec 2011 17:05:28 GMT
Hi Dave,

>> Or should I just make the initial assumption
that all brokers have one partition, send messages to those to create the
topic on the brokers, and then readjust when the the brokers publish their
partitions-per-topic information to ZK after the initial writes?

That's exactly what we do. You can take a look at
bootstrapWithExistingBrokers() in kafka.producer.ZKBrokerPartitionInfo.

>> As an aside, is there any way to query metadata about the broker
directly?
So that a simple consumer might be able to ask a broker for what topics
they have, how many partitions, what version it's running, etc?

Today, we don't have such APIs. But I was thinking we can have something
similar to the 4-letter commands in zookeeper.
For example,

echo topics | nc broker-host broker-port      (Returns all topics with
number of partitions on each broker)
echo srvr | nc broker-host broker-port         (Returns basic broker stats
like version, number of topics etc)

If the above functionality is useful, lets file a JIRA for it.

Thanks
Neha

On Tue, Dec 6, 2011 at 6:34 AM, David Ormsbee <dave@datadoghq.com> wrote:

> Hi folks,
>
> I'm implementing ZooKeeper support in our Python client, and I was
> wondering what the expected behavior of the Producer is when the brokers
> are completely empty. At this point, the brokers have registered themselves
> in /brokers/ids/[0..N], but there are no topics registered under
> /brokers/topics. If someone wants to create a Producer at this point and
> start sending messages to a new topic, is that an error because no brokers
> are registered for that topic? Or should I just make the initial assumption
> that all brokers have one partition, send messages to those to create the
> topic on the brokers, and then readjust when the the brokers publish their
> partitions-per-topic information to ZK after the initial writes?
>
> I'm sure I'm missing something obvious here, since we'd run into a similar
> problem for new, empty brokers joining the pool.
>
> As an aside, is there any way to query metadata about the broker directly?
> So that a simple consumer might be able to ask a broker for what topics
> they have, how many partitions, what version it's running, etc? Or is that
> only ever published through ZooKeeper?
>
> Thank you.
>
> Dave
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message