kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jun Rao <...@confluent.io>
Subject Re: [DISCUSS] KIP-74: Add Fetch Response Size Limit in Bytes
Date Fri, 12 Aug 2016 15:32:45 GMT
Hi, Andrey,

Why shouldn't the client library do reordering? It seems that if
ReplicaFetcher thread does round-robin, the consumer client should do that
too?

Thanks,

Jun

On Fri, Aug 12, 2016 at 3:56 AM, Andrey L. Neporada <
aneporada@yandex-team.ru> wrote:

> Hi!
>
> > On 12 Aug 2016, at 04:29, Jun Rao <jun@confluent.io> wrote:
> >
> > Hi, Andrey,
> >
> > One potential benefit of keeping the per partition limit is for Kafka
> > stream. When reading messages from different partitions, KStream prefers
> to
> > read from partitions with smaller timestamps first and only advances the
> > KStream timestamp when it sees at least one message from every partition.
> > Being able to fill up multiple partitions in a single fetch response can
> > help KStream advance the timestamp quicker when there is backlog from the
> > input. So, it's probably better if we just add a new response limit while
> > keeping the per partition limit.
> >
>
> Yes, this makes sense for me.
>
> > Also, for fairness across partitions, you mentioned "The solution is to
> > start fetching from first empty partition in round-robin fashion or to
> > perform random shuffle of partitions.".  It seems the former is more
> > deterministic. Did you use that in your implementation and should we
> > recommend that for non-java clients as well?
> >
>
> In my initial implementation I just did random shuffling on server side.
> Now I plan to use deterministic approach - do round-robin in
> ReplicaFetcher thread - I believe the client library itself shouldn’t do
> any form of reordering.
>
> But we should definitely recommend some form of shuffling if client wants
> to limit response size.
> Not sure which shuffling method is better.
>
>
> > Thanks,
> >
> > Jun
> >
>
> Thanks,
> Andrey.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message