flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michele Bertoni <michele1.bert...@mail.polimi.it>
Subject Re: open multiple file from list of uri
Date Fri, 26 Jun 2015 10:19:51 GMT
Hi Stephan, thanks for answering,
right now I am using an extension of the DelimitedInputFormat, is there a way to merge it
with the option 2?



Il giorno 26/giu/2015, alle ore 12:17, Stephan Ewen <sewen@apache.org<mailto:sewen@apache.org>>
ha scritto:

There are two ways you can realize that:

1) Create multiple sources and union them. This is easy, but probably a bit less efficient.

2) Override the FileInputFormat's createInputSplits method to take a union of the paths to
create a list of all files and fils splits that will be read.

Stephan


On Fri, Jun 26, 2015 at 12:12 PM, Michele Bertoni <michele1.bertoni@mail.polimi.it<mailto:michele1.bertoni@mail.polimi.it>>
wrote:
Hi everybody,
is there a way to specify a list of URI (“hdfs://file1”,”hdfs://file2”,…) and open
them as different files?
I know i may open the entire directory, but i want to be able to select a subset of files
in the directory

thanks


Mime
View raw message