kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ysysberserk (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (KAFKA-4405) Kafka consumer improperly send prefetch request
Date Wed, 30 Nov 2016 01:52:58 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15707215#comment-15707215
] 

ysysberserk edited comment on KAFKA-4405 at 11/30/16 1:52 AM:
--------------------------------------------------------------

Oh, I am sorry that I only read the code and did not notice that in fetchablePartitions()
the partition which has records is removed before sent a prefetch.

So there is no such a big problem and current work flow is ok.

But we can still add check if total number of retained records is less than max.poll.records
 before we call 

fetcher.sendFetches().It will prevent useless call and save a lot of time and make code eaiser
to understand.


was (Author: ysysberserk):
Oh, I am sorry that I only read the code and did not notice that in fetchablePartitions()
the partition which has records is removed before sent a prefetch.

So there is no such a big problem and current work flow is ok.

But we can still add check if total number of retained records is less than max.poll.records
 before we call 

fetcher.sendFetches().It will prevent useless call and save a lot of time.

> Kafka consumer improperly send prefetch request
> -----------------------------------------------
>
>                 Key: KAFKA-4405
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4405
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.10.0.1
>            Reporter: ysysberserk
>
> Now kafka consumer has added max.poll.records to limit the count of messages return by
poll().
> According to KIP-41, to implement  max.poll.records, the prefetch request should only
be sent when the total number of retained records is less than max.poll.records.
> But in the code of 0.10.0.1 , the consumer will send a prefetch request if it retained
any records and never check if total number of retained records is less than max.poll.records..
> If max.poll.records is set to a count much less than the count of message fetched , the
poll() loop will send a lot of requests than expected and will have more and more records
fetched and stored in memory before they can be consumed.
> So before sending a  prefetch request , the consumer must check if total number of retained
records is less than max.poll.records.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message