Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Wed, 29 Nov 2017 01:13:00 +0000 (UTC)
From: "Eugene Kirpichov (JIRA)" <jira@apache.org>
To: commits@beam.apache.org
Message-ID: <JIRA.13107636.1507326477000.340443.1511917980087@Atlassian.JIRA>
In-Reply-To: <JIRA.13107636.1507326477000@Atlassian.JIRA>
References: <JIRA.13107636.1507326477000@Atlassian.JIRA> <JIRA.13107636.1507326477522@jira-lw-us.apache.org>
Subject: [jira] [Commented] (BEAM-3030) watchForNewFiles() can emit a file
 multiple times if it's growing
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 29 Nov 2017 01:13:04 -0000


    [ https://issues.apache.org/jira/browse/BEAM-3030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269863#comment-16269863 ] 

Eugene Kirpichov commented on BEAM-3030:
----------------------------------------

Fix in https://github.com/apache/beam/pull/4190

> watchForNewFiles() can emit a file multiple times if it's growing
> -----------------------------------------------------------------
>
>                 Key: BEAM-3030
>                 URL: https://issues.apache.org/jira/browse/BEAM-3030
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-core
>            Reporter: Eugene Kirpichov
>            Assignee: Eugene Kirpichov
>             Fix For: 2.3.0
>
>
> TextIO and AvroIO watchForNewFiles(), as well as FileIO.match().continuously(), use Watch transform under the hood, and watch the set of Metadata matching a filepattern.
> Two Metadata's with the same filename but different size are not considered equal, so if these transforms observe the same file multiple times with different sizes, they'll read the file multiple times.
> This is likely not yet a problem for production users, because these features require SDF, it's supported only in Dataflow runner, and users of the Dataflow runner are likely to use only files on GCS which doesn't support appends. However, this needs to be fixed still.


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)