flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3398) Flink Kafka consumer should support auto-commit opt-outs
Date Thu, 07 Apr 2016 19:53:25 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15230927#comment-15230927
] 

ASF GitHub Bot commented on FLINK-3398:
---------------------------------------

Github user wanderingbort commented on the pull request:

    https://github.com/apache/flink/pull/1690#issuecomment-207065222
  
    I just wanted to echo @StephanEwen 's sentiment.  
    
    Flink's management of offsets seems to supersede the Kafka concept of committed offsets
(whether to ZK or an 0.9 broker) leaving me to think there are only 2 scenarios where offset
commitment is useful in the context of flink :
    
    - coordinating with delivery with consumers outside the flink ecosystem (which seems conceptually
dubious)
    - reporting to monitoring software that understands 0.8- zookeeper stored offsets or 0.9+
broker stored offsets
    
    I would consider both of these to be optional features and not required features. 
    
    For my particular use case, we are on Kafka 0.8 and requiring write access to the zookeeper
cluster that is coordinating kafka in order to access read-only message data is problematic.



> Flink Kafka consumer should support auto-commit opt-outs
> --------------------------------------------------------
>
>                 Key: FLINK-3398
>                 URL: https://issues.apache.org/jira/browse/FLINK-3398
>             Project: Flink
>          Issue Type: Bug
>            Reporter: Shikhar Bhushan
>
> Currently the Kafka source will commit consumer offsets to Zookeeper, either upon a checkpoint
if checkpointing is enabled, otherwise periodically based on {{auto.commit.interval.ms}}
> It should be possible to opt-out of committing consumer offsets to Zookeeper. Kafka has
this config as {{auto.commit.enable}} (0.8) and {{enable.auto.commit}} (0.9).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message