Mailing-List: contact issues-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Date: Wed, 28 Sep 2016 13:27:20 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: issues@flink.apache.org
Message-ID: <JIRA.12995632.1470657618000.687070.1475069240635@Atlassian.JIRA>
In-Reply-To: <JIRA.12995632.1470657618000@Atlassian.JIRA>
References: <JIRA.12995632.1470657618000@Atlassian.JIRA> <JIRA.12995632.1470657618377@arcas>
Subject: [jira] [Commented] (FLINK-4329) Fix Streaming File Source
 Timestamps/Watermarks Handling
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 28 Sep 2016 13:27:22 -0000


    [ https://issues.apache.org/jira/browse/FLINK-4329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15529592#comment-15529592 ] 

ASF GitHub Bot commented on FLINK-4329:
---------------------------------------

Github user kl0u commented on the issue:

    https://github.com/apache/flink/pull/2546
  
    Hi @StephanEwen , thanks for the review!
    
    The watermarks/timestamps are now generated by the Reader, and not the operator that creates the splits. The same holds for the LongMax watermark, which is created at the close() of the ContinuousFileReaderOperator. 
    
    As for tests, it is the testFileReadingOperatorWithIngestionTime() in the ContinuousFileMonitoringTest which checks if the last Watermark is the LongMax.
    
    The original problem was that there were no timestamps assigned to the elements for Ingestion time and watermarks were emitted (I think it was a Process_once case).
    

> Fix Streaming File Source Timestamps/Watermarks Handling
> --------------------------------------------------------
>
>                 Key: FLINK-4329
>                 URL: https://issues.apache.org/jira/browse/FLINK-4329
>             Project: Flink
>          Issue Type: Bug
>          Components: Streaming Connectors
>    Affects Versions: 1.1.0
>            Reporter: Aljoscha Krettek
>            Assignee: Kostas Kloudas
>             Fix For: 1.2.0, 1.1.3
>
>
> The {{ContinuousFileReaderOperator}} does not correctly deal with watermarks, i.e. they are just passed through. This means that when the {{ContinuousFileMonitoringFunction}} closes and emits a {{Long.MAX_VALUE}} that watermark can "overtake" the records that are to be emitted in the {{ContinuousFileReaderOperator}}. Together with the new "allowed lateness" setting in window operator this can lead to elements being dropped as late.
> Also, {{ContinuousFileReaderOperator}} does not correctly assign ingestion timestamps since it is not technically a source but looks like one to the user.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)