kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <...@confluent.io>
Subject Re: KIP-4 Wiki Update
Date Mon, 04 Apr 2016 00:04:19 GMT
Ismael,

The reason that we only apply the regex for topics on the client side is
that regex is java specific. So, for non-java clients, it would be a bit
weird for them to include java specific stuff in the wire protocol.

Thanks,

Jun

On Sun, Apr 3, 2016 at 12:36 AM, Ismael Juma <ismael@juma.me.uk> wrote:

> Hi Grant,
>
> One question that occurred to me is whether we want to take the chance to
> make it possible to pass a regex pattern for the desired topics in the
> metadata request. This would potentially improve the efficiency of
> `KafkaConsumer.subscribe` significantly for cases where there are large
> number of topics and partitions (we currently get the metadata for _all_
> topics and apply the regular expression on the client).
>
> Ismael
>
> On Fri, Apr 1, 2016 at 3:29 AM, Grant Henke <ghenke@cloudera.com> wrote:
>
> > I have update the wiki and patch based on the feedback on the metadata
> > changes. Please take a look and let me know if there are any concerns or
> > issues.
> >
> >
> >    - Wiki:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-MetadataSchema
> >    - PR: https://github.com/apache/kafka/pull/1095
> >
> > I will hold a vote on the metadata change soon if no major issues are
> > raised.
> >
> > On Thu, Mar 31, 2016 at 6:33 PM, Jason Gustafson <jason@confluent.io>
> > wrote:
> >
> > > I also prefer B. In addition to being intuitive, it seems less
> > error-prone
> > > in the long term, though it might be a little annoying for clients
> > > maintaining backwards compatibility.
> > >
> > > -Jason
> > >
> > > On Thu, Mar 31, 2016 at 1:24 PM, Ismael Juma <ismael@juma.me.uk>
> wrote:
> > >
> > > > I prefer B, the fact that we version the protocol means that we can
> fix
> > > > mistakes instead of living with them forever. We should take
> advantage
> > of
> > > > that.
> > > >
> > > > Ismael
> > > >
> > > > On Thu, Mar 31, 2016 at 9:15 PM, Grant Henke <ghenke@cloudera.com>
> > > wrote:
> > > >
> > > > > Looking for some resolution on the "No Topics" change.
> > > > >
> > > > > I am thinking that using null in the protocol isn't that complex,
> and
> > > it
> > > > > avoids various edge cases with having multiple fields. That leaves
> us
> > > > with
> > > > > 2 options:
> > > > >
> > > > >    - A: null = no topics, empty = all topics
> > > > >    - B: null = all topics, empty = no topics
> > > > >
> > > > > A is nice because it just adds new functionality, existing logic
> > > doesn't
> > > > > change
> > > > > B is nice because its more "intuitive", but has the drawback of
> > > changing
> > > > > what empty means from request v0
> > > > >
> > > > > I do not have a strong opinion on the approach taken, which makes
> me
> > > lean
> > > > > towards option A. Keep in mind at the user level, the apis in the
> > > various
> > > > > clients can map this however they like.
> > > > >
> > > > > Does anyone feel strongly about the choice?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Mar 31, 2016 at 9:21 AM, Grant Henke <ghenke@cloudera.com>
> > > > wrote:
> > > > >
> > > > > > I had a second look at the proposed changes to Metadata Request
> and
> > > > > >> Response and it seems to me that having a `controller_id`
field
> > > would
> > > > be
> > > > > >> more efficient for non-trivial cases than having a
> `is_controller`
> > > > field
> > > > > >>  for each broker (which would be false for all but 1 case).
> > > > > >
> > > > > >
> > > > > > I agree this is better. I will update it.
> > > > > >
> > > > > > Similar, but less clear is the best way to encode
> > > `marked_for_deletion`
> > > > > and
> > > > > >> `is_internal`. These will also be false for most topics
(there
> is
> > > only
> > > > > one
> > > > > >> internal topic at the moment, for example), so it may make
sense
> > to
> > > > > have a
> > > > > >> topics_marked_for_deletion and internal_topics in the response.
> > > > Because
> > > > > >> topics are identified by strings, it is not as clear-cut
as the
> > > > > >> controller_id case, but it still seems like it would be
a win
> for
> > > when
> > > > > it
> > > > > >> matters most (when the number of topics is large).
> > > > > >>
> > > > > >
> > > > > > Thats an interesting idea. I can try making this change to see
> what
> > > it
> > > > > > would look like.
> > > > > >
> > > > > > Thanks,
> > > > > > Grant
> > > > > >
> > > > > > On Thu, Mar 31, 2016 at 8:59 AM, Ismael Juma <ismael@juma.me.uk>
> > > > wrote:
> > > > > >
> > > > > >> Hi Grant,
> > > > > >>
> > > > > >> I had a second look at the proposed changes to Metadata
Request
> > and
> > > > > >> Response and it seems to me that having a `controller_id`
field
> > > would
> > > > be
> > > > > >> more efficient for non-trivial cases than having a
> `is_controller`
> > > > field
> > > > > >>  for each broker (which would be false for all but 1 case).
> > > > > >>
> > > > > >> Similar, but less clear is the best way to encode
> > > > `marked_for_deletion`
> > > > > >> and
> > > > > >> `is_internal`. These will also be false for most topics
(there
> is
> > > only
> > > > > one
> > > > > >> internal topic at the moment, for example), so it may make
sense
> > to
> > > > > have a
> > > > > >> topics_marked_for_deletion and internal_topics in the response.
> > > > Because
> > > > > >> topics are identified by strings, it is not as clear-cut
as the
> > > > > >> controller_id case, but it still seems like it would be
a win
> for
> > > when
> > > > > it
> > > > > >> matters most (when the number of topics is large).
> > > > > >
> > > > > >
> > > > > >> Ismael
> > > > > >>
> > > > > >> On Mon, Mar 14, 2016 at 10:07 PM, Grant Henke <
> > ghenke@cloudera.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > I have been updating the KIP-4 wiki page based on the
last KIP
> > > call
> > > > > and
> > > > > >> > wanted to get some review and discussion around the
server
> side
> > > > > >> > implementation for admin requests. Both the "ideal"
> > functionality
> > > > and
> > > > > >> the
> > > > > >> > "intermediated" functionality. The updates are still
in
> > progress,
> > > > but
> > > > > >> this
> > > > > >> > section is the most critical and will likely have the
most
> > > > discussion.
> > > > > >> This
> > > > > >> > topic has had a few shifts in perspective and various
> > discussions
> > > on
> > > > > >> > synchronous vs asynchronous server support. The wiki
contains
> my
> > > > > current
> > > > > >> > perspective on the challenges and approach.
> > > > > >> >
> > > > > >> > If you have any thoughts or feedback on the "Server-side
Admin
> > > > Request
> > > > > >> > handlers" section here
> > > > > >> > <
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-2.Server-sideAdminRequesthandlers
> > > > > >> > >.
> > > > > >> > Lets discuss them in this thread.
> > > > > >> >
> > > > > >> > For reference the last KIP discussion can be viewed
here:
> > > > > >> > https://youtu.be/rFW0-zJqg5I?t=12m30s
> > > > > >> >
> > > > > >> > Thank you,
> > > > > >> > Grant
> > > > > >> > --
> > > > > >> > Grant Henke
> > > > > >> > Software Engineer | Cloudera
> > > > > >> > grant@cloudera.com | twitter.com/gchenke |
> > > > linkedin.com/in/granthenke
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Grant Henke
> > > > > > Software Engineer | Cloudera
> > > > > > grant@cloudera.com | twitter.com/gchenke |
> > > linkedin.com/in/granthenke
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Grant Henke
> > > > > Software Engineer | Cloudera
> > > > > grant@cloudera.com | twitter.com/gchenke |
> > linkedin.com/in/granthenke
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Grant Henke
> > Software Engineer | Cloudera
> > grant@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message