kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4405) Kafka consumer improperly send prefetch request
Date Wed, 30 Nov 2016 09:30:58 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15708050#comment-15708050

ASF GitHub Bot commented on KAFKA-4405:

GitHub user enothereska opened a pull request:


    KAFKA-4405: Check max.poll.records before prefetching


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/enothereska/kafka KAFKA-4405-prefetch

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2193
commit 592482e0c41b14f66bd78f2f4f6247a686743280
Author: Eno Thereska <eno.thereska@gmail.com>
Date:   2016-11-30T09:29:55Z

    Check max.poll.records before prefetching


> Kafka consumer improperly send prefetch request
> -----------------------------------------------
>                 Key: KAFKA-4405
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4405
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions:
>            Reporter: ysysberserk
> Now kafka consumer has added max.poll.records to limit the count of messages return by
> According to KIP-41, to implement  max.poll.records, the prefetch request should only
be sent when the total number of retained records is less than max.poll.records.
> But in the code of , the consumer will send a prefetch request if it retained
any records and never check if total number of retained records is less than max.poll.records..
> If max.poll.records is set to a count much less than the count of message fetched , the
poll() loop will send a lot of requests than expected and will have more and more records
fetched and stored in memory before they can be consumed.
> So before sending a  prefetch request , the consumer must check if total number of retained
records is less than max.poll.records.

This message was sent by Atlassian JIRA

View raw message