incubator-kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Inder Pall <inder.p...@gmail.com>
Subject Re: ZooKeeper-based Producer behavior before there are topics
Date Mon, 12 Dec 2011 05:20:18 GMT
Neha,

how would it be better than putting the new broker behind a VIP.
Is the ZK based producer equipped with more info to take a better call.



On Sun, Dec 11, 2011 at 11:57 PM, Neha Narkhede <neha.narkhede@gmail.com>wrote:

> Inder,
>
> Zookeeper is used on the producer side for load balancing. It is used for
> automatic detection of new and failed brokers.
>
> Thanks,
> Neha
>
> On Sun, Dec 11, 2011 at 2:44 AM, Inder Pall <inder.pall@gmail.com> wrote:
>
> > why do u want to integrate zk at producer side. can you please elaborate
> > your usecase.
> >
> > is it for load balancing.
> > On Dec 7, 2011 12:35 AM, "Jun Rao" <junrao@gmail.com> wrote:
> > >
> > > David,
> > >
> > > Could you also comment whether you want a programming api or a
> monitoring
> > > api?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Dec 6, 2011 at 10:03 AM, David Ormsbee <dave@datadoghq.com>
> > wrote:
> > >
> > > > Jun, Neha:
> > > >
> > > > Thank you for pointing me the right way on the Producer issue. :-)
> > > >
> > > > As for the metadata requests, I've filed a ticket here:
> > > >
> > > >  https://issues.apache.org/jira/browse/KAFKA-223
> > > >
> > > > Take care.
> > > >
> > > > Dave
> > > >
> > > > On Tue, Dec 6, 2011 at 12:05 PM, Neha Narkhede <
> > neha.narkhede@gmail.com
> > > > >wrote:
> > > >
> > > > > Hi Dave,
> > > > >
> > > > > >> Or should I just make the initial assumption
> > > > > that all brokers have one partition, send messages to those to
> create
> > the
> > > > > topic on the brokers, and then readjust when the the brokers
> publish
> > > > their
> > > > > partitions-per-topic information to ZK after the initial writes?
> > > > >
> > > > > That's exactly what we do. You can take a look at
> > > > > bootstrapWithExistingBrokers() in
> > kafka.producer.ZKBrokerPartitionInfo.
> > > > >
> > > > > >> As an aside, is there any way to query metadata about the
broker
> > > > > directly?
> > > > > So that a simple consumer might be able to ask a broker for what
> > topics
> > > > > they have, how many partitions, what version it's running, etc?
> > > > >
> > > > > Today, we don't have such APIs. But I was thinking we can have
> > something
> > > > > similar to the 4-letter commands in zookeeper.
> > > > > For example,
> > > > >
> > > > > echo topics | nc broker-host broker-port      (Returns all topics
> > with
> > > > > number of partitions on each broker)
> > > > > echo srvr | nc broker-host broker-port         (Returns basic
> broker
> > > > stats
> > > > > like version, number of topics etc)
> > > > >
> > > > > If the above functionality is useful, lets file a JIRA for it.
> > > > >
> > > > > Thanks
> > > > > Neha
> > > > >
> > > > > On Tue, Dec 6, 2011 at 6:34 AM, David Ormsbee <dave@datadoghq.com>
> > > > wrote:
> > > > >
> > > > > > Hi folks,
> > > > > >
> > > > > > I'm implementing ZooKeeper support in our Python client, and
I
> was
> > > > > > wondering what the expected behavior of the Producer is when
the
> > > > brokers
> > > > > > are completely empty. At this point, the brokers have registered
> > > > > themselves
> > > > > > in /brokers/ids/[0..N], but there are no topics registered under
> > > > > > /brokers/topics. If someone wants to create a Producer at this
> > point
> > > > and
> > > > > > start sending messages to a new topic, is that an error because
> no
> > > > > brokers
> > > > > > are registered for that topic? Or should I just make the initial
> > > > > assumption
> > > > > > that all brokers have one partition, send messages to those
to
> > create
> > > > the
> > > > > > topic on the brokers, and then readjust when the the brokers
> > publish
> > > > > their
> > > > > > partitions-per-topic information to ZK after the initial writes?
> > > > > >
> > > > > > I'm sure I'm missing something obvious here, since we'd run
into
> a
> > > > > similar
> > > > > > problem for new, empty brokers joining the pool.
> > > > > >
> > > > > > As an aside, is there any way to query metadata about the broker
> > > > > directly?
> > > > > > So that a simple consumer might be able to ask a broker for
what
> > topics
> > > > > > they have, how many partitions, what version it's running, etc?
> Or
> > is
> > > > > that
> > > > > > only ever published through ZooKeeper?
> > > > > >
> > > > > > Thank you.
> > > > > >
> > > > > > Dave
> > > > > >
> > > > >
> > > >
> >
>



-- 
Thanks,
- Inder
  Tech Platforms @Inmobi
  Linkedin - http://goo.gl/eR4Ub

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message