flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sourigna Phetsarath <gna.phetsar...@teamaol.com>
Subject Re: Flink 1.0.0 reading files from multiple directory with wildcards
Date Mon, 21 Mar 2016 14:01:43 GMT
Thanks Ufuk, I'm already using the recursive traversal feature.

On Mon, Mar 21, 2016 at 8:39 AM, Ufuk Celebi <uce@apache.org> wrote:

> If you want all sub directories under data/2016/01, then this could help:
> https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/index.html#recursive-traversal-of-the-input-path-directory
>
> On Mon, Mar 21, 2016 at 11:35 AM, Fabian Hueske <fhueske@gmail.com> wrote:
>
>> Hi,
>>
>> no, this is currently not supported. However, I agree this would be a
>> very valuable addition to the FileInputFormat.
>> Would you mind opening a JIRA issue with your suggestions?
>>
>> Until this is added to Flink, it can be implemented as a custom
>> InputFormat based on FileInputFormat by overriding the createInputSplits()
>> method.
>>
>> Best, Fabian
>>
>> 2016-03-21 0:11 GMT+01:00 Sourigna Phetsarath <gna.phetsarath@teamaol.com
>> >:
>>
>>> All,
>>>
>>> Do any of the Flink Data Sources support comma separated directories
>>> with wildcards?
>>>
>>> For example:
>>>
>>> env.readFile("/data/2016/01/01/*/*,/data/2016/01/02/*/*,
>>> /data/2016/01/03/*/*")
>>>
>>>
>>> Thanks in advance for any help that you can provide.
>>> --
>>>
>>>
>>> *Gna Phetsarath*System Architect // AOL Platforms // Data Services //
>>> Applied Research Chapter
>>> 770 Broadway, 5th Floor, New York, NY 10003
>>> o: 212.402.4871 // m: 917.373.7363
>>> vvmr: 8890237 aim: sphetsarath20 t: @sourigna
>>>
>>> * <http://www.aolplatforms.com>*
>>>
>>
>>
>


-- 


*Gna Phetsarath*System Architect // AOL Platforms // Data Services //
Applied Research Chapter
770 Broadway, 5th Floor, New York, NY 10003
o: 212.402.4871 // m: 917.373.7363
vvmr: 8890237 aim: sphetsarath20 t: @sourigna

* <http://www.aolplatforms.com>*

Mime
View raw message