kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4161) Decouple flush and offset commits
Date Thu, 01 Dec 2016 23:02:58 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15713356#comment-15713356
] 

ASF GitHub Bot commented on KAFKA-4161:
---------------------------------------

Github user asfgit closed the pull request at:

    https://github.com/apache/kafka/pull/2139


> Decouple flush and offset commits
> ---------------------------------
>
>                 Key: KAFKA-4161
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4161
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>            Reporter: Shikhar Bhushan
>            Assignee: Shikhar Bhushan
>              Labels: needs-kip
>             Fix For: 0.10.2.0
>
>
> It is desirable to have, in addition to the time-based flush interval, volume or size-based
commits. E.g. a sink connector which is buffering in terms of number of records may want to
request a flush when the buffer is full, or when sufficient amount of data has been buffered
in a file.
> Having a method like say {{requestFlush()}} on the {{SinkTaskContext}} would allow for
connectors to have flexible policies around flushes. This would be in addition to the time
interval based flushes that are controlled with {{offset.flush.interval.ms}}, for which the
clock should be reset when any kind of flush happens.
> We should probably also support requesting flushes via the {{SourceTaskContext}} for
consistency though a use-case doesn't come to mind off the bat.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message