flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-4329) Fix Streaming File Source Timestamps/Watermarks Handling
Date Thu, 11 Aug 2016 12:33:22 GMT

    [ https://issues.apache.org/jira/browse/FLINK-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417133#comment-15417133
] 

ASF GitHub Bot commented on FLINK-4329:
---------------------------------------

Github user kl0u commented on the issue:

    https://github.com/apache/flink/pull/2350
  
    The way it works is that now the reader gets a ReaderContext and emits its own watermarks
depending on which timeCharacteristic we are operating on. If it is on IngestionTime, which
was the original problem, we emit periodically. In addition, in this case, it assigns timestamps
to the emitted elements.


> Fix Streaming File Source Timestamps/Watermarks Handling
> --------------------------------------------------------
>
>                 Key: FLINK-4329
>                 URL: https://issues.apache.org/jira/browse/FLINK-4329
>             Project: Flink
>          Issue Type: Bug
>          Components: Streaming Connectors
>    Affects Versions: 1.1.0
>            Reporter: Aljoscha Krettek
>            Assignee: Kostas Kloudas
>             Fix For: 1.1.1
>
>
> The {{ContinuousFileReaderOperator}} does not correctly deal with watermarks, i.e. they
are just passed through. This means that when the {{ContinuousFileMonitoringFunction}} closes
and emits a {{Long.MAX_VALUE}} that watermark can "overtake" the records that are to be emitted
in the {{ContinuousFileReaderOperator}}. Together with the new "allowed lateness" setting
in window operator this can lead to elements being dropped as late.
> Also, {{ContinuousFileReaderOperator}} does not correctly assign ingestion timestamps
since it is not technically a source but looks like one to the user.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message