kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Gustafson (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-2500) Expose fetch response high watermark in ConsumerRecords
Date Fri, 23 Sep 2016 23:06:20 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-2500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Jason Gustafson updated KAFKA-2500:
    Fix Version/s:

> Expose fetch response high watermark in ConsumerRecords
> -------------------------------------------------------
>                 Key: KAFKA-2500
>                 URL: https://issues.apache.org/jira/browse/KAFKA-2500
>             Project: Kafka
>          Issue Type: Sub-task
>          Components: consumer
>    Affects Versions:
>            Reporter: Will Funnell
>            Assignee: Jason Gustafson
>            Priority: Critical
>             Fix For:
> Originally created in the old consumer here: https://issues.apache.org/jira/browse/KAFKA-1977
> The requirement is to create a snapshot from the Kafka topic but NOT do continual reads
after that point. For example you might be creating a backup of the data to a file.
> This ticket covers the addition of the functionality to the new consumer.
> In order to achieve that, a recommended solution by Joel Koshy and Jay Kreps was to expose
the high watermark, as maxEndOffset, from the FetchResponse object through to each MessageAndMetadata
object in order to be aware when the consumer has reached the end of each partition.
> The submitted patch achieves this by adding the maxEndOffset to the PartitionTopicInfo,
which is updated when a new message arrives in the ConsumerFetcherThread and then exposed
in MessageAndMetadata.
> See here for discussion:
> http://search-hadoop.com/m/4TaT4TpJy71

This message was sent by Atlassian JIRA

View raw message