kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colin McCabe <cmcc...@apache.org>
Subject Re: KIP-327: Add describe all topics API to AdminClient
Date Mon, 16 Jul 2018 03:08:40 GMT
Thanks, Manikumar.  I've been meaning to bring up KIP-142 again.  It would definitely be a
nice improvement.

best,
Colin


On Sat, Jul 14, 2018, at 08:51, Manikumar wrote:
> Hi Jason and Colin,
> 
> Thanks for the feedback. I agree that having filtering support to the
> Metadata API would be useful and solves
> the scalability issues.
> 
> But to implement specific use case of "describe all topics", regex 
> support
> won't help. In any case user needs to
> call listTopics() to get topic list, and then make describeTopics() 
> calls
> with a subset of the topics set.
> This leads to improving existing listTopics() API performance. Colin
> already raised a KIP for this: KIP-142
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-142%3A+Add+ListTopicsRequest+to+efficiently+list+all+the+topics+in+a+cluster>
>  .
> May be we should consider implementing KIP-142.
> 
> Since we have support wildcard ACLs, Initially, I can explore
> prefixed/wildcards patterns support to Metadata API.
> We can later extend support for regular expressions.
> 
> Thanks
> 
> 
> 
> On Sat, Jul 14, 2018 at 2:42 PM Ted Yu <yuzhihong@gmail.com> wrote:
> 
> > What if broker crashes before all the pages can be returned ?
> >
> > Cheers
> >
> > On Sat, Jul 14, 2018 at 1:07 AM Stephane Maarek <
> > stephane@simplemachines.com.au> wrote:
> >
> > > Why not paginate ? Then one can retrieve as many topics as desired ?
> > >
> > > On Sat., 14 Jul. 2018, 4:15 pm Colin McCabe, <cmccabe@apache.org> wrote:
> > >
> > > > Good point.  We should probably have a maximum number of results like
> > > > 1000 or something.  That can go in the request RPC as well...
> > > > Cheers,
> > > > Colin
> > > >
> > > > On Fri, Jul 13, 2018, at 18:15, Ted Yu wrote:
> > > > > bq. describe topics by a regular expression on the server side
> > > > >
> > > > > Should caution be taken if the regex doesn't filter ("*") ?
> > > > >
> > > > > Cheers
> > > > >
> > > > > On Fri, Jul 13, 2018 at 6:02 PM Colin McCabe
> > > > > <cmccabe@apache.org> wrote:>
> > > > > > As Jason wrote, this won't scale as the number of partitions
> > > > > > increases.> > We already have users who have tens of thousands
of
> > > > topics, or
> > > > > > more.  If> > you multiply that by 100x over the next few
years, you
> > > > end up with
> > > > > > this API> > returning full information about millions
of topics,
> > > which
> > > > clearly
> > > > > > doesn't> > work.
> > > > > >
> > > > > > We discussed this a lot in the original KIP-117 DISCUSS thread
> > > > > > which added> > the Java AdminClient.  ListTopics and DescribeTopics
> > > > were
> > > > > > deliberately kept> > separate because we understood that
> > eventually a
> > > > single RPC would
> > > > > > not be> > able to return information about all the topics
in the
> > > > cluster.  So
> > > > > > I have> > to vote -1 for this proposal as it stands.
> > > > > >
> > > > > > I do agree that adding a way to describe topics by a regular
> > > > > > expression on> > the server side would be very useful.
 This would
> > > > also fix a major
> > > > > > scalability problem we have now, which is that when
> > > > > > subscribing via a> > regular expression, clients need
to fetch the
> > > > full list of all
> > > > > > topics in> > the cluster and filter locally.
> > > > > >
> > > > > > I think a regular expression library like re2 would be ideal
> > > > > > for this> > purpose.  re2 is standardized and language-agnostic
> > (it's
> > > > not tied
> > > > > > only to> > Java).  In contrast, Java regular expression
change with
> > > > different
> > > > > > releases> > of the JDK (there were some changes in java
8, for
> > > > example).
> > > > > > Also, re2> > regular expressions are linear time, never
exponential
> > > > time.  See
> > > > > > https://github.com/google/re2j
> > > > > >
> > > > > > regards,
> > > > > > Colin
> > > > > >
> > > > > >
> > > > > > On Fri, Jul 13, 2018, at 05:00, Andras Beni wrote:
> > > > > > > The KIP looks good to me.
> > > > > > > However, if there is willingness in the community to work
on
> > > > > > > metadata> > > request with patterns, the feature
proposed here
> > and
> > > > filtering by
> > > > > > > '*' or> > > '.*' would be redundant.
> > > > > > >
> > > > > > > Andras
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Jul 13, 2018 at 12:38 AM Jason Gustafson
> > > > > > > <jason@confluent.io>> > wrote:
> > > > > > >
> > > > > > > > Hey Manikumar,
> > > > > > > >
> > > > > > > > As Kafka begins to scale to larger and larger numbers
of
> > > > > > topics/partitions,
> > > > > > > > I'm a little concerned about the scalability of APIs
such as
> > > > > > > > this. The> > API
> > > > > > > > looks benign, but imagine you have have a few million
> > > > > > > > partitions. We> > > > already expose similar
APIs in the
> > producer
> > > > and consumer, so
> > > > > > > > probably> > not
> > > > > > > > much additional harm to expose it in the AdminClient,
but it
> > > > > > > > would be> > nice
> > > > > > > > to put a little thought into some longer term options.
We
> > should
> > > > > > > > be> > giving
> > > > > > > > users an efficient way to select a smaller set of
the topics
> > > > > > > > they are> > > > interested in. We have
always discussed adding
> > > > some filtering
> > > > > > > > support> > to
> > > > > > > > the Metadata API. Perhaps now is a good time to reconsider
> > this?
> > > > > > > > We now> > > > have a convention for wildcard
ACLs, so perhaps
> > we
> > > > can do
> > > > > > > > something> > > > similar. Full regex support
might be ideal
> > given
> > > > the consumer's> > > > subscription API, but that is more challenging.
> > > What
> > > > do you
> > > > > > > > think?> > > >
> > > > > > > > Thanks,
> > > > > > > > Jason
> > > > > > > >
> > > > > > > > On Thu, Jul 12, 2018 at 2:35 PM, Harsha <kafka@harsha.io>
> > > wrote:>
> > > > > > >
> > > > > > > > > Very useful. LGTM.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Harsha
> > > > > > > > >
> > > > > > > > > On Thu, Jul 12, 2018, at 9:56 AM, Manikumar wrote:
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I have created a KIP to add describe all
topics API to
> > > > > > > > > > AdminClient> > .
> > > > > > > > > >
> > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > > > > > 327%3A+Add+describe+all+topics+API+to+AdminClient
> > > > > > > > > >
> > > > > > > > > > Please take a look.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > > >
> > >
> >

Mime
View raw message