Mailing-List: contact users-help@kafka.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@kafka.apache.org
Received-SPF: pass (athena.apache.org: domain of hsy541@gmail.com designates
 209.85.214.169 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <5133607134025723777@unknownmsgid>
References: 
 <CAG2CJAcJ4F0NiWO6e8LGFtbzG+M6kLmiO9rsSpSVnNTRKyrKEQ@mail.gmail.com>
	<CAOG_4QYbzM82prNeB65LU5ZSWMNaDRkcLp8fTa7fAf0yf2+wjQ@mail.gmail.com>
	<CAG2CJAcszyG2vq0GwyFHTeZLCiqySd1G-QkvxnK1XpxeOoX-5w@mail.gmail.com>
	<CAOG_4QYrct-wDrDogwjG6y0YJ8pfqgRop+7kgRc46hZ0AFGHnw@mail.gmail.com>
	<5133607134025723777@unknownmsgid>
Date: Fri, 11 Oct 2013 17:16:16 -0700
Message-ID: 
 <CAG2CJAcjNwrBsV+YQSZhFoSJK5pfE6WygWHAP6pUs--gT6mJYA@mail.gmail.com>
Subject: Re: Is there a way to pull out kafka metadata from zookeeper?
From: "hsy541@gmail.com" <hsy541@gmail.com>
To: users@kafka.apache.org
Content-Type: multipart/alternative; boundary=e89a8fb1fde6bc260404e880249d

--e89a8fb1fde6bc260404e880249d
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

That's why I'm asking, I would like to see a kafka zookeeper client api to
get TopicMetadata instead of my own hacky way to query the zookeeper

Thanks!
Best,
Siyuan


On Fri, Oct 11, 2013 at 4:00 PM, Bruno D. Rodrigues <
bruno.rodrigues@litux.org> wrote:

> Why not ask zookeeper for the list of brokers and then ask a random
> broker for the metadata (and repeat if the broker is down), even if
> it's two calls.
>
> Heck it already does unnecessary connections. It connects to a broker,
> gets the metadata, disconnects, and then connects again for the data.
> If it's already assumed a producer or consumer will take some seconds
> until ready, what is another call gonna prejudice the flow.
>
> Then producers and consumers would then be consistently configured. Or
> allow the producers to also go to a broker instead of zookeeper.
>
> This way the consumer needs to know and hardcode at least one node.
> The node can fail. It can be changed.
>
> I thought zookeeper served to abstract this kind of complexity
>
>
>
>
>
>
>
> --
> Bruno Rodrigues
> Sent from my iPhone
>
> No dia 11/10/2013, =E0s 22:40, Neha Narkhede <neha.narkhede@gmail.com>
> escreveu:
>
> >>> For each consumer consumes different
> > topic/replica I have to specify those 20 brokers and go over all of the=
m
> to
> > know which broker is alive. And even worse how about I dynamically add
> new
> > broker into the cluster and remove the old one
> >
> > TopicMetadataRequest is a batch API and you can get metadata informatio=
n
> > for either a list of all topics or all topics in the cluster, if you
> > specify an empty list of topics. Adding a broker is not a problem since
> the
> > metadata request also returns the list of brokers in a cluster. The
> reason
> > this is better than reading from zookeeper is because the same operatio=
n
> > would require multiple zookeeper roundtrips, instead of a single
> > TopicMetadataRequest roundtrip to some kafka broker.
> >
> > Thanks,
> > Neha
> >
> >
> >> On Fri, Oct 11, 2013 at 11:30 AM, hsy541@gmail.com <hsy541@gmail.com>
> wrote:
> >>
> >> Thanks guys!
> >> But I feel weird. Assume I have 20 brokers for 10 different topics wit=
h
> 2
> >> partitions and  2 replicas for each. For each consumer consumes
> different
> >> topic/replica I have to specify those 20 brokers and go over all of
> them to
> >> know which broker is alive. And even worse how about I dynamically add
> new
> >> broker into the cluster and remove the old one. I think it's nice to
> have a
> >> way to get metadata from zookeeper(centralized coordinator?) directly.
> >>
> >> Best,
> >> Siyuan
> >>
> >>
> >> On Fri, Oct 11, 2013 at 9:12 AM, Neha Narkhede <neha.narkhede@gmail.co=
m
> >>> wrote:
> >>
> >>> If, for some reason, you don't have access to a virtual IP or load
> >>> balancer, you need to round robin once through all the brokers before
> >>> failing a TopicMetadataRequest. So unless all the brokers in your
> cluster
> >>> are down, this should not be a problem.
> >>>
> >>> Thanks,
> >>> Neha
> >>>
> >>>
> >>> On Thu, Oct 10, 2013 at 10:50 PM, hsy541@gmail.com <hsy541@gmail.com>
> >>> wrote:
> >>>
> >>>> Hi guys,
> >>>>
> >>>> I'm trying to maintain a bunch of simple kafka consumer to consume
> >>> messages
> >>>> from brokers. I know there is a way to send TopicMetadataRequest to
> >>> broker
> >>>> and get the response from the broker. But you have to specify the
> >> broker
> >>>> list to query the information. But broker might not be available
> >> because
> >>> of
> >>>> some failure. My question is is there any api I can call and query
> >> broker
> >>>> metadata for topic/partition directly from zookeeper? I know I can
> >> query
> >>>> that information using zookeeper API. But that's not friendly
> >>> datastructure
> >>>> like the TopicMetadata/PartitionMetadata.  Thank you!
> >>>>
> >>>> Best,
> >>>> Siyuan
> >>
>

--e89a8fb1fde6bc260404e880249d--