beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Halperin (JIRA)" <j...@apache.org>
Subject [jira] [Created] (BEAM-57) FileBasedSink should write intermediate output to a different folder
Date Thu, 25 Feb 2016 15:45:18 GMT
Daniel Halperin created BEAM-57:
-----------------------------------

             Summary: FileBasedSink should write intermediate output to a different folder
                 Key: BEAM-57
                 URL: https://issues.apache.org/jira/browse/BEAM-57
             Project: Beam
          Issue Type: New Feature
          Components: sdk-java-core
            Reporter: Daniel Halperin
            Assignee: Davor Bonaci
            Priority: Minor


Currently, FileBasedSink writes intermediate files and final output files to the same folder.
 Since deletion of intermediate files is prone to eventual consistency issues with GCS,S3,other
systems and users may use globs in subsequent pipelines, intermediate files should be written
to a different folder.

This may be a subdirectory or a pipeline-defined temp directory (with probably some additional
subdirectories for this source).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message