flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: open multiple file from list of uri
Date Fri, 26 Jun 2015 10:21:57 GMT
Sure, just override the "createInputSplits()" method. Call for each of your
file paths "super.createInputSplits()" and then combine the results into
one array that you return.

That should do it...

On Fri, Jun 26, 2015 at 12:19 PM, Michele Bertoni <
michele1.bertoni@mail.polimi.it> wrote:

>  Hi Stephan, thanks for answering,
> right now I am using an extension of the DelimitedInputFormat, is there a
> way to merge it with the option 2?
>
>
>
>  Il giorno 26/giu/2015, alle ore 12:17, Stephan Ewen <sewen@apache.org>
> ha scritto:
>
>  There are two ways you can realize that:
>
>  1) Create multiple sources and union them. This is easy, but probably a
> bit less efficient.
>
>  2) Override the FileInputFormat's createInputSplits method to take a
> union of the paths to create a list of all files and fils splits that will
> be read.
>
>  Stephan
>
>
> On Fri, Jun 26, 2015 at 12:12 PM, Michele Bertoni <
> michele1.bertoni@mail.polimi.it> wrote:
>
>> Hi everybody,
>> is there a way to specify a list of URI (“hdfs://file1”,”hdfs://file2”,…)
>> and open them as different files?
>> I know i may open the entire directory, but i want to be able to select a
>> subset of files in the directory
>>
>> thanks
>
>
>
>

Mime
View raw message