flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-5704) Deprecate FlinkKafkaConsumer constructors in favor of improvements to decoupling from Kafka offset committing
Date Mon, 31 Jul 2017 09:54:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107058#comment-16107058

Tzu-Li (Gordon) Tai commented on FLINK-5704:

Setting  "Fix Version" to 1.4.0 for this issue. With a big new partition discovery feature
in 1.4, it would be a good opportunity to incorporate that as a feature available only when
using the new constructors.

> Deprecate FlinkKafkaConsumer constructors in favor of improvements to decoupling from
Kafka offset committing
> -------------------------------------------------------------------------------------------------------------
>                 Key: FLINK-5704
>                 URL: https://issues.apache.org/jira/browse/FLINK-5704
>             Project: Flink
>          Issue Type: Improvement
>          Components: Kafka Connector, Streaming Connectors
>            Reporter: Tzu-Li (Gordon) Tai
>             Fix For: 1.4.0
> With FLINK-3398 and FLINK-4280, the {{FlinkKafkaConsumer}} will be able to completely
operate independently of committed offsets in Kafka.
> I.e.,
> (1) *Starting position*: when starting, the consumer can choose to not use any committed
offsets in Kafka as the starting position
> (2) *Committing offsets back to Kafka*: the consumer can completely opt-out of committing
offsets back to Kafka
> However, our current default behaviour for (1) is to respect committed offsets, and (2)
is to always have offset committing. Users still have to call the respective setter configuration
methods to change this.
> I think we should deprecate the current constructors in favor of new ones with default
behaviours (1) start from the latest record, without respecting Kafka offsets, and (2) don't
commit offsets.
> With this change, users explicitly call the config methods of FLINK-3398 and FLINK-4280
to *enable* respecting committed offsets for Kafka, instead of _disabling_ it. They would
want to / need to enable it, only when perhaps to migrate from a non-Flink consuming application,
or they wish to expose the internal checkpointed offsets to measure consumer lag using Kafka
> The main advantage for this change is that the API of {{FlinkKafkaConsumer}} can speak
for itself that it does not depend on committed offsets in Kafka (this is a misconception
that users frequently have), and that exactly-once depends solely on offsets checkpointed
internally using Flink's checkpointing mechanics.

This message was sent by Atlassian JIRA

View raw message