kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Gustafson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-5211) KafkaConsumer should not skip a corrupted record after throwing an exception.
Date Thu, 25 May 2017 02:22:04 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024066#comment-16024066

Jason Gustafson commented on KAFKA-5211:

[~enothereska] Looking toward the future, it would be good to get your take on what the ideal
behavior would be for Kafka streams. Thanks to some of the improvements in this release cycle,
the current "intended" semantics may now actually be usable (which wasn't the case before--as
is apparent from the discussion in KAFKA-4740). When we raise an exception from parsing or
record corruption, the consumer's position should now be pointing to the offset of the record
that failed parsing, which means the user can seek past it if they wish. Because of the reasons
mentioned by Becket in the description, I don't think it will be safe to proactively skip
past a corrupt record--at least not without refetching the data--but it may be possible for
parsing errors. It has also been proposed to move parsing out of the consumer entirely (see
KAFKA-1895), which would neatly solve part of the problem, but would likely require a KIP.
Given the short horizon for this release, however, Becket's patch is probably the way to go
for now.

> KafkaConsumer should not skip a corrupted record after throwing an exception.
> -----------------------------------------------------------------------------
>                 Key: KAFKA-5211
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5211
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Jiangjie Qin
>            Assignee: Jiangjie Qin
>              Labels: clients, consumer
>             Fix For:
> In 0.10.2, when there is a corrupted record, KafkaConsumer.poll() will throw an exception
and block on that corrupted record. In the latest trunk this behavior has changed to skip
the corrupted record (which is the old consumer behavior). With KIP-98, skipping corrupted
messages would be a little dangerous as the message could be a control message for a transaction.
We should fix the issue to let the KafkaConsumer block on the corrupted messages.

This message was sent by Atlassian JIRA

View raw message