Mailing-List: contact issues-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Date: Thu, 2 Feb 2017 09:55:54 +0000 (UTC)
From: "Tzu-Li (Gordon) Tai (JIRA)" <jira@apache.org>
To: issues@flink.apache.org
Message-ID: <JIRA.12939176.1455506081000.15900.1486029354987@Atlassian.JIRA>
In-Reply-To: <JIRA.12939176.1455506081000@Atlassian.JIRA>
References: <JIRA.12939176.1455506081000@Atlassian.JIRA> <JIRA.12939176.1455506081591@jira-lw-us.apache.org>
Subject: [jira] [Comment Edited] (FLINK-3398) Flink Kafka consumer should
 support auto-commit opt-outs
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Thu, 02 Feb 2017 09:56:03 -0000


    [ https://issues.apache.org/jira/browse/FLINK-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15849733#comment-15849733 ] 

Tzu-Li (Gordon) Tai edited comment on FLINK-3398 at 2/2/17 9:55 AM:
--------------------------------------------------------------------

I think we should keep this JIRA instead of merging it into FLINK-4280.
They try to address different new features: FLINK-4280 is for more flexible start position configuration independent from Kafka offsets.
This JIRA is for opt-out from offset committing on Flink's checkpoints.

I'll move the respective proposed API for commit opt-out here:

{code}
Properties props = new Properties();
...
FlinkKafkaConsumer09 kafka = new FlinkKafkaConsumer09("topic", schema, props);
kafka.setEnableOffsetCommittingOnCheckpoints(boolean); // if true (checkpointing should be enabled), overrides periodic checkpointing if "enable.auto.commit" is set in props.
{code}

So the scenarios breaks down into:

1. {{props.put("auto.commit.enable", "true")}} & {{setEnableOffsetCommittingOnCheckpoints(false)}}:
Perform auto committing with the internal client (or in 0.8 case our own periodic committer).

2. {{props.put("auto.commit.enable", "true")}} & {{setEnableOffsetCommittingOnCheckpoints(true)}}:
Overrides whatever value in props for "auto.commit.enable", only commit on checkpoints. If checkpointing isn't enabled, the consumer fails due to conflicting configuration.

3. {{props.put("auto.commit.enable", "false")}} & {{setEnableOffsetCommittingOnCheckpoints(false)}}: No offset committing at all.


was (Author: tzulitai):
I think we should keep this JIRA instead of merging it into FLINK-4280.
They try to address different new features: FLINK-4280 is for more flexible start position configuration independent from Kafka offsets.
This JIRA is for opt-out from offset committing on Flink's checkpoints.

I'll move the respective proposed API for commit opt-out here:

{code}
Properties props = new Properties();
...
FlinkKafkaConsumer09 kafka = new FlinkKafkaConsumer09("topic", schema, props);
kafka.setEnableOffsetCommittingOnCheckpoints(boolean); // if true (checkpointing should be enabled), overrides periodic checkpointing if "enable.auto.commit" is set in props.
{code}

So the scenarios breaks down into:

1. {{props.put("auto.commit.enable", "true")}} & {{setEnableOffsetCommittingOnCheckpoints(false)}}:
Perform auto committing with the internal client (or in 0.8 case our own periodic committer).

2. {{props.put("auto.commit.enable", "true")}} & {{setEnableOffsetCommittingOnCheckpoints(true)}}:
Overrides whatever value in props for "auto.commit.enable", only commit on checkpoints. If checkpointing isn't enabled, the consumer fails due to conflicting configuration.


> Flink Kafka consumer should support auto-commit opt-outs
> --------------------------------------------------------
>
>                 Key: FLINK-3398
>                 URL: https://issues.apache.org/jira/browse/FLINK-3398
>             Project: Flink
>          Issue Type: Improvement
>          Components: Kafka Connector
>            Reporter: Shikhar Bhushan
>            Assignee: Tzu-Li (Gordon) Tai
>            Priority: Critical
>             Fix For: 1.3.0
>
>
> Currently the Kafka source will commit consumer offsets to Zookeeper, either upon a checkpoint if checkpointing is enabled, otherwise periodically based on {{auto.commit.interval.ms}}
> It should be possible to opt-out of committing consumer offsets to Zookeeper. Kafka has this config as {{auto.commit.enable}} (0.8) and {{enable.auto.commit}} (0.9).


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)