kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Noll (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-3478) Finer Stream Flow Control
Date Tue, 05 Jul 2016 12:34:11 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-3478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Michael Noll updated KAFKA-3478:
--------------------------------
    Description: 
Today we have a event-time based flow control mechanism in order to synchronize multiple input
streams in a best effort manner:

http://docs.confluent.io/3.0.0/streams/architecture.html#flow-control-with-timestamps

However, there are some use cases where users would like to have finer control of the input
streams, for example, with two input streams, one of them always reading from offset 0 upon
(re)-starting, and the other reading for log end offset.

Today we only have one consumer config "offset.auto.reset" to control that behavior, which
means all streams are read either from "earliest" or "latest".

We should consider how to improve this settings to allow users have finer control over these
frameworks.

  was:
Today we have a event-time based flow control mechanism in order to synchronize multiple input
streams in a best effort manner:

http://docs.confluent.io/2.1.0-alpha1/streams/architecture.html#flow-control-with-timestamps

However, there are some use cases where users would like to have finer control of the input
streams, for example, with two input streams, one of them always reading from offset 0 upon
(re)-starting, and the other reading for log end offset.

Today we only have one consumer config "offset.auto.reset" to control that behavior, which
means all streams are read either from "earliest" or "latest".

We should consider how to improve this settings to allow users have finer control over these
frameworks.


> Finer Stream Flow Control
> -------------------------
>
>                 Key: KAFKA-3478
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3478
>             Project: Kafka
>          Issue Type: Sub-task
>          Components: streams
>            Reporter: Guozhang Wang
>              Labels: user-experience
>             Fix For: 0.10.1.0
>
>
> Today we have a event-time based flow control mechanism in order to synchronize multiple
input streams in a best effort manner:
> http://docs.confluent.io/3.0.0/streams/architecture.html#flow-control-with-timestamps
> However, there are some use cases where users would like to have finer control of the
input streams, for example, with two input streams, one of them always reading from offset
0 upon (re)-starting, and the other reading for log end offset.
> Today we only have one consumer config "offset.auto.reset" to control that behavior,
which means all streams are read either from "earliest" or "latest".
> We should consider how to improve this settings to allow users have finer control over
these frameworks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message