flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xingcan Cui (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7548) Support watermark generation for TableSource
Date Tue, 17 Oct 2017 15:28:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16207782#comment-16207782

Xingcan Cui commented on FLINK-7548:

Hi [~fhueske], sorry for the late reply. I'll first share some ideas that struck me and then
keep the focus on this issue.
# Now that the rowtime can be extracted by expressions, shall we consider allowing more data
types for the original fields? For instance, the users may want to extract the timestamps
from a {{String}} field or even from multiple fields by a UDF. 
# Since the stream API do not support multiple watermarks, maybe it's not necessary to define
a list of {{RowtimeAttributeDescriptor}}.
# Sometimes the processing time could be inapplicable, while the streams are lack of timestamps.
I think it would be convenient to provide an inner mechanism, which can materialize the ingestion
time as the rowtime.
# Using Table API Expression sounds like a good idea. However, I just wonder whether it will
make the DDL harder to implement.

Best, Xingcan

> Support watermark generation for TableSource
> --------------------------------------------
>                 Key: FLINK-7548
>                 URL: https://issues.apache.org/jira/browse/FLINK-7548
>             Project: Flink
>          Issue Type: Bug
>          Components: Table API & SQL
>            Reporter: Jark Wu
>            Assignee: Fabian Hueske
>            Priority: Blocker
>             Fix For: 1.4.0
> As discussed in FLINK-7446, currently the TableSource only support to define rowtime
field, but not support to extract watermarks from the rowtime field. We can provide a new
interface called {{DefinedWatermark}}, which has two methods {{getRowtimeAttribute}} (can
only be an existing field) and {{getWatermarkGenerator}}. The {{DefinedRowtimeAttribute}}
will be marked deprecated.
> How to support periodic and punctuated watermarks and support some built-in strategies
needs further discussion.

This message was sent by Atlassian JIRA

View raw message