kafka-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "hsy541@gmail.com" <hsy...@gmail.com>
Subject Re: Is there a way to pull out kafka metadata from zookeeper?
Date Fri, 11 Oct 2013 22:20:27 GMT
What if the broker it tries to get the TopicMetadata from is down?
I assume a broker is more like to be down than a zookeeper.

Thanks,
Siyuan


On Fri, Oct 11, 2013 at 2:39 PM, Neha Narkhede <neha.narkhede@gmail.com>wrote:

> >> For each consumer consumes different
> topic/replica I have to specify those 20 brokers and go over all of them to
> know which broker is alive. And even worse how about I dynamically add new
> broker into the cluster and remove the old one
>
> TopicMetadataRequest is a batch API and you can get metadata information
> for either a list of all topics or all topics in the cluster, if you
> specify an empty list of topics. Adding a broker is not a problem since the
> metadata request also returns the list of brokers in a cluster. The reason
> this is better than reading from zookeeper is because the same operation
> would require multiple zookeeper roundtrips, instead of a single
> TopicMetadataRequest roundtrip to some kafka broker.
>
> Thanks,
> Neha
>
>
> On Fri, Oct 11, 2013 at 11:30 AM, hsy541@gmail.com <hsy541@gmail.com>
> wrote:
>
> > Thanks guys!
> > But I feel weird. Assume I have 20 brokers for 10 different topics with 2
> > partitions and  2 replicas for each. For each consumer consumes different
> > topic/replica I have to specify those 20 brokers and go over all of them
> to
> > know which broker is alive. And even worse how about I dynamically add
> new
> > broker into the cluster and remove the old one. I think it's nice to
> have a
> > way to get metadata from zookeeper(centralized coordinator?) directly.
> >
> > Best,
> > Siyuan
> >
> >
> > On Fri, Oct 11, 2013 at 9:12 AM, Neha Narkhede <neha.narkhede@gmail.com
> > >wrote:
> >
> > > If, for some reason, you don't have access to a virtual IP or load
> > > balancer, you need to round robin once through all the brokers before
> > > failing a TopicMetadataRequest. So unless all the brokers in your
> cluster
> > > are down, this should not be a problem.
> > >
> > > Thanks,
> > > Neha
> > >
> > >
> > > On Thu, Oct 10, 2013 at 10:50 PM, hsy541@gmail.com <hsy541@gmail.com>
> > > wrote:
> > >
> > > > Hi guys,
> > > >
> > > > I'm trying to maintain a bunch of simple kafka consumer to consume
> > > messages
> > > > from brokers. I know there is a way to send TopicMetadataRequest to
> > > broker
> > > > and get the response from the broker. But you have to specify the
> > broker
> > > > list to query the information. But broker might not be available
> > because
> > > of
> > > > some failure. My question is is there any api I can call and query
> > broker
> > > > metadata for topic/partition directly from zookeeper? I know I can
> > query
> > > > that information using zookeeper API. But that's not friendly
> > > datastructure
> > > >  like the TopicMetadata/PartitionMetadata.  Thank you!
> > > >
> > > > Best,
> > > > Siyuan
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message