Mailing-List: contact commits-help@beam.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.incubator.apache.org
Date: Thu, 25 Feb 2016 16:08:18 +0000 (UTC)
From: "Daniel Halperin (JIRA)" <jira@apache.org>
To: commits@beam.incubator.apache.org
Message-ID: <JIRA.12944537.1456416476000.145309.1456416498193@Atlassian.JIRA>
In-Reply-To: <JIRA.12944537.1456416476000@Atlassian.JIRA>
References: <JIRA.12944537.1456416476000@Atlassian.JIRA>
 <JIRA.12944537.1456416476863@arcas>
Subject: [jira] [Created] (BEAM-60) FileBasedSource/IOChannelFactory: Custom
 glob expansion
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Daniel Halperin created BEAM-60:
-----------------------------------

             Summary: FileBasedSource/IOChannelFactory: Custom glob expansion
                 Key: BEAM-60
                 URL: https://issues.apache.org/jira/browse/BEAM-60
             Project: Beam
          Issue Type: New Feature
          Components: sdk-java-core
            Reporter: Daniel Halperin
            Assignee: Davor Bonaci


Many cloud and distributed filesystems are eventually consistent, for instance Amazon s3 and Google Cloud Storage.

To work around this, many systems that produce files such as Beam's FileBasedSinks, or Google BigQuery will provide methods to determine the number and set of files produced. E.g.,

* Beam FileBasedSink uses -00000-of-NNNNN
* BigQuery export jobs uses -000000 -000001 -000002 ... until an empty file is produced
* Another system may produce a .filelist suffix that contains a list of all files.

Users should be able to supply a glob to FileBasedSource but additionally supply a "glob expander" that can provide a custom implementation for file expansion. That way, e.g., Beam pipelines can be run back-to-back-to-back where each consumes the output of the previous, on an inconsistent filesystem, without data loss.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)