incubator-kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <jun...@gmail.com>
Subject Re: ZooKeeper-based Producer behavior before there are topics
Date Mon, 12 Dec 2011 17:16:54 GMT
Inder,

If you just want to do random distribution, ZK-based load balancing is not
too much different from VIP. We have experienced some subtle issues with
VIP. For example, it's not easy to figure out the real IP of a client
connected to a broker. Also, VIP timeout can be tricky (need to send RST to
the client if connection is killed by timeouts).

Further more, if you want to partition based on a key, then this is hard
for VIP to do.

Thanks,

Jun

On Sun, Dec 11, 2011 at 9:20 PM, Inder Pall <inder.pall@gmail.com> wrote:

> Neha,
>
> how would it be better than putting the new broker behind a VIP.
> Is the ZK based producer equipped with more info to take a better call.
>
>
>
> On Sun, Dec 11, 2011 at 11:57 PM, Neha Narkhede <neha.narkhede@gmail.com
> >wrote:
>
> > Inder,
> >
> > Zookeeper is used on the producer side for load balancing. It is used for
> > automatic detection of new and failed brokers.
> >
> > Thanks,
> > Neha
> >
> > On Sun, Dec 11, 2011 at 2:44 AM, Inder Pall <inder.pall@gmail.com>
> wrote:
> >
> > > why do u want to integrate zk at producer side. can you please
> elaborate
> > > your usecase.
> > >
> > > is it for load balancing.
> > > On Dec 7, 2011 12:35 AM, "Jun Rao" <junrao@gmail.com> wrote:
> > > >
> > > > David,
> > > >
> > > > Could you also comment whether you want a programming api or a
> > monitoring
> > > > api?
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > > On Tue, Dec 6, 2011 at 10:03 AM, David Ormsbee <dave@datadoghq.com>
> > > wrote:
> > > >
> > > > > Jun, Neha:
> > > > >
> > > > > Thank you for pointing me the right way on the Producer issue. :-)
> > > > >
> > > > > As for the metadata requests, I've filed a ticket here:
> > > > >
> > > > >  https://issues.apache.org/jira/browse/KAFKA-223
> > > > >
> > > > > Take care.
> > > > >
> > > > > Dave
> > > > >
> > > > > On Tue, Dec 6, 2011 at 12:05 PM, Neha Narkhede <
> > > neha.narkhede@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Hi Dave,
> > > > > >
> > > > > > >> Or should I just make the initial assumption
> > > > > > that all brokers have one partition, send messages to those
to
> > create
> > > the
> > > > > > topic on the brokers, and then readjust when the the brokers
> > publish
> > > > > their
> > > > > > partitions-per-topic information to ZK after the initial writes?
> > > > > >
> > > > > > That's exactly what we do. You can take a look at
> > > > > > bootstrapWithExistingBrokers() in
> > > kafka.producer.ZKBrokerPartitionInfo.
> > > > > >
> > > > > > >> As an aside, is there any way to query metadata about
the
> broker
> > > > > > directly?
> > > > > > So that a simple consumer might be able to ask a broker for
what
> > > topics
> > > > > > they have, how many partitions, what version it's running, etc?
> > > > > >
> > > > > > Today, we don't have such APIs. But I was thinking we can have
> > > something
> > > > > > similar to the 4-letter commands in zookeeper.
> > > > > > For example,
> > > > > >
> > > > > > echo topics | nc broker-host broker-port      (Returns all topics
> > > with
> > > > > > number of partitions on each broker)
> > > > > > echo srvr | nc broker-host broker-port         (Returns basic
> > broker
> > > > > stats
> > > > > > like version, number of topics etc)
> > > > > >
> > > > > > If the above functionality is useful, lets file a JIRA for it.
> > > > > >
> > > > > > Thanks
> > > > > > Neha
> > > > > >
> > > > > > On Tue, Dec 6, 2011 at 6:34 AM, David Ormsbee <
> dave@datadoghq.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hi folks,
> > > > > > >
> > > > > > > I'm implementing ZooKeeper support in our Python client,
and I
> > was
> > > > > > > wondering what the expected behavior of the Producer is
when
> the
> > > > > brokers
> > > > > > > are completely empty. At this point, the brokers have
> registered
> > > > > > themselves
> > > > > > > in /brokers/ids/[0..N], but there are no topics registered
> under
> > > > > > > /brokers/topics. If someone wants to create a Producer
at this
> > > point
> > > > > and
> > > > > > > start sending messages to a new topic, is that an error
because
> > no
> > > > > > brokers
> > > > > > > are registered for that topic? Or should I just make the
> initial
> > > > > > assumption
> > > > > > > that all brokers have one partition, send messages to those
to
> > > create
> > > > > the
> > > > > > > topic on the brokers, and then readjust when the the brokers
> > > publish
> > > > > > their
> > > > > > > partitions-per-topic information to ZK after the initial
> writes?
> > > > > > >
> > > > > > > I'm sure I'm missing something obvious here, since we'd
run
> into
> > a
> > > > > > similar
> > > > > > > problem for new, empty brokers joining the pool.
> > > > > > >
> > > > > > > As an aside, is there any way to query metadata about the
> broker
> > > > > > directly?
> > > > > > > So that a simple consumer might be able to ask a broker
for
> what
> > > topics
> > > > > > > they have, how many partitions, what version it's running,
etc?
> > Or
> > > is
> > > > > > that
> > > > > > > only ever published through ZooKeeper?
> > > > > > >
> > > > > > > Thank you.
> > > > > > >
> > > > > > > Dave
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>
>
>
> --
> Thanks,
> - Inder
>  Tech Platforms @Inmobi
>  Linkedin - http://goo.gl/eR4Ub
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message