beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Halperin (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (BEAM-57) FileBasedSink should write intermediate output to a different folder
Date Thu, 03 Nov 2016 19:53:58 GMT

     [ https://issues.apache.org/jira/browse/BEAM-57?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Daniel Halperin resolved BEAM-57.
---------------------------------
       Resolution: Fixed
    Fix Version/s: 0.4.0-incubating

> FileBasedSink should write intermediate output to a different folder
> --------------------------------------------------------------------
>
>                 Key: BEAM-57
>                 URL: https://issues.apache.org/jira/browse/BEAM-57
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-java-core
>            Reporter: Daniel Halperin
>            Assignee: Eugene Kirpichov
>            Priority: Minor
>             Fix For: 0.4.0-incubating
>
>
> Currently, FileBasedSink writes intermediate files and final output files to the same
folder.  Since deletion of intermediate files is prone to eventual consistency issues with
GCS,S3,other systems and users may use globs in subsequent pipelines, intermediate files should
be written to a different folder.
> This may be a subdirectory or a pipeline-defined temp directory (with probably some additional
subdirectories for this source).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message