kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Falko <afa...@salesforce.com>
Subject Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability
Date Tue, 09 Jan 2018 05:43:24 GMT
On Sat, Jan 6, 2018 at 5:57 PM, Colin McCabe <cmccabe@apache.org> wrote:
> On Thu, Jan 4, 2018, at 10:37, Jun Rao wrote:
>>
>> 4. The process to mark partition as dirty requires updating every fetch
>> session having the partition. This may add some overhead. An alternative
>> approach is to check the difference btw cached fetch offset and HW (or LEO)
>> when serving the fetch request.
>
> That's a good point.  The caching approach avoids needing to update every fetch session
when one of those numbers changes.  I think an even more important advantage is that it's
simpler to implement -- we don't have to worry about forgetting to update a fetch session
when one of those numbers changes.  The disadvantage is some extra memory consumption per
partition per fetch session.
>
> I think the advantage, especially in terms of simplicity, might override the memory concern.
 My initial implementation uses the caching approach.  I will update the KIP once I have this
working.
>

We're very interested in this KIP because it might improve one of our
topic-heavy clusters. I have a stress-test generating topics across a
number of kafka brokers; if you'd like early and quick feedback on
your implementation let me know!

The discussion thread is very long, so hopefully I'm not asking
something that was asked before: does Kafka already expose
`FetchRequest` size for monitoring purposes? It might improve the KIP
to track the before-and-after behavior.

-Andrey

Mime
View raw message