kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Byrne <bby...@confluent.io>
Subject Re: [VOTE] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics
Date Fri, 08 Nov 2019 16:50:06 GMT
Hi Guozhang,

Regarding metadata expiry, no access times other than the initial lookup[1]
are used for determining when to expire producer metadata. Therefore,
frequently used topics' metadata will be aged out and subsequently
refreshed (in a blocking manner) every five minutes, and infrequently used
topics will be retained for a minimum of five minutes and currently
refetched on every metadata update during that time period. The sentence is
suggesting that we could reduce the expiry time to improve the latter
without affecting (rather slightly improving) the former.

Keep in mind that in most all cases, I wouldn't anticipate much of a
difference with producer behavior, and the extra logic can be implemented
to have insignificant cost. It's the large/dynamic topic corner cases that
we're trying to improve.

It'd be convenient if the KIP is no longer necessary. You're right in that
there's no public API changes and the behavioral changes are entirely
internal. I'd be happy to continue the discussion around the KIP, but
unless otherwise objected, it can be retired.

[1] Not entirely accurate, it's actually the first time when the client
calculates whether to retain the topic in its metadata.


On Thu, Nov 7, 2019 at 4:48 PM Guozhang Wang <wangguoz@gmail.com> wrote:

> Hello Brian,
> Could you elaborate a bit more on this sentence: "This logic can be made
> more intelligent by managing the expiry from when the topic was last used,
> enabling the expiry duration to be reduced to improve cases where a large
> number of topics are touched intermittently." Not sure I fully understand
> the proposal.
> Also since now this KIP did not make any public API changes and the
> behavioral changes are not considered a public API contract (i.e. how we
> maintain the topic metadata in producer cache is never committed to users),
> I wonder if we still need a KIP for the proposed change any more?
> Guozhang
> On Thu, Nov 7, 2019 at 12:43 PM Brian Byrne <bbyrne@confluent.io> wrote:
> > Hello all,
> >
> > I'd like to propose a vote for a producer change to improve producer
> > behavior when dealing with a large number of topics, in part by reducing
> > the amount of metadata fetching performed.
> >
> > The full KIP is provided here:
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-526%3A+Reduce+Producer+Metadata+Lookups+for+Large+Number+of+Topics
> >
> > And the discussion thread:
> >
> >
> https://lists.apache.org/thread.html/b2f8f830ef04587144cf0840c7d4811bbf0a14f3c459723dbc5acf9e@%3Cdev.kafka.apache.org%3E
> >
> > Thanks,
> > Brian
> >
> --
> -- Guozhang

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message