crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Micah Whitacre (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (CRUNCH-630) KafkaRecordReader keeps retrying to poll data when the offset is reset to latest offset
Date Thu, 12 Jan 2017 02:04:16 GMT

     [ https://issues.apache.org/jira/browse/CRUNCH-630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Micah Whitacre resolved CRUNCH-630.
-----------------------------------
       Resolution: Fixed
    Fix Version/s: 0.15.0

Code has been pushed to master.

> KafkaRecordReader keeps retrying to poll data when the offset is reset to latest offset
> ---------------------------------------------------------------------------------------
>
>                 Key: CRUNCH-630
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-630
>             Project: Crunch
>          Issue Type: Bug
>            Reporter: Pooja Dhondge
>            Assignee: Micah Whitacre
>             Fix For: 0.15.0
>
>         Attachments: CRUNCH-630.patch
>
>
> We recently saw this behavior where, if the offset it is trying to read from doesn't
exist on Kafka due to retention policy, the offset gets reset to latest(default) and the KafkaRecordReader
keeps retrying beyond KAFKA_EMPTY_RETRY_ATTEMPTS_KEY
> {noformat}
> ...crunch.kafka.inputformat.KafkaRecordReader: No records retrieved but pending offsets
to consume therefore polling again. Attempt 17/10
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message