kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-4144) Allow per stream/table timestamp extractor
Date Fri, 14 Apr 2017 22:52:41 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Matthias J. Sax updated KAFKA-4144:
-----------------------------------
    Description: 
At the moment the timestamp extractor is configured via a {{StreamConfig}} value to {{KafkaStreams}}.
That means you can only have a single timestamp extractor per app, even though you may be
joining multiple streams/tables that require different timestamp extraction methods.

You should be able to specify a timestamp extractor via {{KStreamBuilder.stream()/table()}},
just like you can specify key and value serdes that override the StreamConfig defaults.

KIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788

Specifying a per-stream extractor should only be possible for sources, but not for intermediate
topics. For PAPI we cannot enforce this, but for DSL {{through()}} should not allow to set
a custom extractor by the user. In contrast, with regard to KAFKA-4785, is must internally
set an extractor that returns the record's metadata timestamp in order to overwrite the global
extractor from {{StreamsConfig}} (ie, set {{FailOnInvalidTimestampExtractor}}). This change
should be done in KAFKA-4785 though.

  was:
At the moment the timestamp extractor is configured via a {{StreamConfig}} value to {{KafkaStreams}}.
That means you can only have a single timestamp extractor per app, even though you may be
joining multiple streams/tables that require different timestamp extraction methods.

You should be able to specify a timestamp extractor via {{KStreamBuilder.stream()/table()}},
just like you can specify key and value serdes that override the StreamConfig defaults.

Specifying a per-stream extractor should only be possible for sources, but not for intermediate
topics. For PAPI we cannot enforce this, but for DSL {{through()}} should not allow to set
a custom extractor by the user. In contrast, with regard to KAFKA-4785, is must internally
set an extractor that returns the record's metadata timestamp in order to overwrite the global
extractor from {{StreamsConfig}} (ie, set {{FailOnInvalidTimestampExtractor}}). This change
should be done in KAFKA-4785 though.


> Allow per stream/table timestamp extractor
> ------------------------------------------
>
>                 Key: KAFKA-4144
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4144
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>    Affects Versions: 0.10.0.1
>            Reporter: Elias Levy
>            Assignee: Jeyhun Karimov
>              Labels: api, kip
>
> At the moment the timestamp extractor is configured via a {{StreamConfig}} value to {{KafkaStreams}}.
That means you can only have a single timestamp extractor per app, even though you may be
joining multiple streams/tables that require different timestamp extraction methods.
> You should be able to specify a timestamp extractor via {{KStreamBuilder.stream()/table()}},
just like you can specify key and value serdes that override the StreamConfig defaults.
> KIP: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68714788
> Specifying a per-stream extractor should only be possible for sources, but not for intermediate
topics. For PAPI we cannot enforce this, but for DSL {{through()}} should not allow to set
a custom extractor by the user. In contrast, with regard to KAFKA-4785, is must internally
set an extractor that returns the record's metadata timestamp in order to overwrite the global
extractor from {{StreamsConfig}} (ie, set {{FailOnInvalidTimestampExtractor}}). This change
should be done in KAFKA-4785 though.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message