flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephan Ewen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3398) Flink Kafka consumer should support auto-commit opt-outs
Date Fri, 29 Jul 2016 16:04:20 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399575#comment-15399575

Stephan Ewen commented on FLINK-3398:

Let's reboot the discussion and work on this issue. Some observations:

  1. Many cases do not not need committing of offsets externally. Users often specify some
random "group.id" if no meaningful group can be specified.
  2. In some cases, committing is even harmful, as the picked up start offsets override the
reset strategies like (earliest offset). To avoid that, people use a random "group.id".
  3. If one wants to use external offsets, one typically also wants to update them, and vice
versa. Both is tied to a "group.id"

With these observations, it seems we can tie this to the existence of a group:

  - If one sets a "group.id", we use it for start offset and commit offsets.
  - If no "group.id" exists, no start offsets are obtained and no offsets are committed.

Does that make sense?

> Flink Kafka consumer should support auto-commit opt-outs
> --------------------------------------------------------
>                 Key: FLINK-3398
>                 URL: https://issues.apache.org/jira/browse/FLINK-3398
>             Project: Flink
>          Issue Type: Improvement
>          Components: Kafka Connector
>            Reporter: Shikhar Bhushan
>             Fix For: 1.2.0
> Currently the Kafka source will commit consumer offsets to Zookeeper, either upon a checkpoint
if checkpointing is enabled, otherwise periodically based on {{auto.commit.interval.ms}}
> It should be possible to opt-out of committing consumer offsets to Zookeeper. Kafka has
this config as {{auto.commit.enable}} (0.8) and {{enable.auto.commit}} (0.9).

This message was sent by Atlassian JIRA

View raw message