kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jiangjie Qin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-5630) Consumer poll loop over the same record after a CorruptRecordException
Date Mon, 24 Jul 2017 19:09:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16098990#comment-16098990
] 

Jiangjie Qin commented on KAFKA-5630:
-------------------------------------

[~vmaurin_glispa] Thanks for reporting the issue. It looks that the issue here is that we
assumed the InvalidRecordException will only be thrown when we explicitly validate the record
in the Fetcher.PartitionRecords. But it can actually be thrown from the iterator as well.
I'll fix that.

> Consumer poll loop over the same record after a CorruptRecordException
> ----------------------------------------------------------------------
>
>                 Key: KAFKA-5630
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5630
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>    Affects Versions: 0.11.0.0
>            Reporter: Vincent Maurin
>            Priority: Critical
>              Labels: regression, reliability
>             Fix For: 0.11.0.1
>
>
> Hello
> While consuming a topic with log compaction enabled, I am getting an infinite consumption
loop of the same record, i.e, each call to poll is returning to me 500 times one record (500
is my max.poll.records). I am using the java client 0.11.0.0.
> Running the code with the debugger, the initial problem come from `Fetcher.PartitionRecords,fetchRecords()`.
> Here I get a `org.apache.kafka.common.errors.CorruptRecordException: Record size is less
than the minimum record overhead (14)`
> Then the boolean `hasExceptionInLastFetch` is set to true, resulting the test block in
`Fetcher.PartitionRecords.nextFetchedRecord()` to always return the last record.
> I guess the corruption problem is similar too https://issues.apache.org/jira/browse/KAFKA-5582
but this behavior of the client is probably not the expected one



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message