flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Metzger (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-3294) KafkaConsumer (0.8) commit offsets using SimpleConsumer.commitOffsets()
Date Wed, 27 Jan 2016 15:23:39 GMT
Robert Metzger created FLINK-3294:

             Summary: KafkaConsumer (0.8) commit offsets using SimpleConsumer.commitOffsets()
                 Key: FLINK-3294
                 URL: https://issues.apache.org/jira/browse/FLINK-3294
             Project: Flink
          Issue Type: Bug
          Components: Kafka Connector
            Reporter: Robert Metzger

Currently, the 0.8 consumer for Kafka is committing the offsets manually into Zookeeper so
that users can track the lag using external tools.

The 0.8 consumer has a pluggable design, and this component is easily pluggable.
Since OffsetCommitRequest version=1 (supported in 0.8.2 or later), users can choose between
two offset commit modes:
a) Let the broker commit into ZK (this is  what we are doing from the consumer
b) Let the broker commit the offset into a special topic.

By adding a different "OffsetHandler" backend, users can commit offsets from the brokers (reducing
the total number of ZK connections) or into the broker's offset topic.

This message was sent by Atlassian JIRA

View raw message