flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Artiom Darie (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-6417) Wildcard support for read text file
Date Fri, 28 Apr 2017 16:06:04 GMT
Artiom Darie created FLINK-6417:

             Summary: Wildcard support for read text file
                 Key: FLINK-6417
                 URL: https://issues.apache.org/jira/browse/FLINK-6417
             Project: Flink
          Issue Type: New Feature
          Components: Core
            Reporter: Artiom Darie
            Priority: Minor

Add wildcard support while reading from s3://, hdfs://, file://, etc.

h6. Examples:
# {code} s3://bucket-name/*.gz {code}
# {code} hdfs://path/*file-name*.csv {code}
# {code} file://tmp/**/*.* {code}

h6. Proposal
# Use the existing method: {code}environment.readFile(...){code}
# List all the files in the directories
# Read files using existing: {code}ContinuousFileReaderOperator{code}

h6. Concerns (Open for discussions)
# Have multiple DataSource(s) created for each each file and then to join them into a single
# Have all the files into the same DataSource

This message was sent by Atlassian JIRA

View raw message