flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jens Kat (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3294) KafkaConsumer (0.8) commit offsets using SimpleConsumer.commitOffsets()
Date Wed, 29 Jun 2016 10:16:37 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15354983#comment-15354983

Jens Kat commented on FLINK-3294:

I am interested in using 'option 2' committing into a consumer_offset topic of kafka, instead
of using Zookeeper as offset-store.
What is the status of your code? Is it far from complete or does it need some tidying up,
I can try to help with that.  

> KafkaConsumer (0.8) commit offsets using SimpleConsumer.commitOffsets()
> -----------------------------------------------------------------------
>                 Key: FLINK-3294
>                 URL: https://issues.apache.org/jira/browse/FLINK-3294
>             Project: Flink
>          Issue Type: Bug
>          Components: Kafka Connector
>            Reporter: Robert Metzger
>            Assignee: Robert Metzger
> Currently, the 0.8 consumer for Kafka is committing the offsets manually into Zookeeper
so that users can track the lag using external tools.
> The 0.8 consumer has a pluggable design, and this component is easily pluggable.
> Since OffsetCommitRequest version=1 (supported in 0.8.2 or later), users can choose between
two offset commit modes:
> a) Let the broker commit into ZK (this is  what we are doing from the consumer
> b) Let the broker commit the offset into a special topic.
> By adding a different "OffsetHandler" backend, users can commit offsets from the brokers
(reducing the total number of ZK connections) or into the broker's offset topic.

This message was sent by Atlassian JIRA

View raw message