hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <qwertyman...@gmail.com>
Subject Re: Recusrive file search for FileInputPath
Date Thu, 11 Nov 2010 13:02:24 GMT

On Thu, Nov 11, 2010 at 4:25 PM, Jaydeep Ayachit
<jaydeep_ayachit@persistent.co.in> wrote:
> Can mapreduce job recursively browse through all files and select them for processing
when higher level folder is set in FileInputPath?
> For example,
> Dir-1
> |___      Dir-2
>                |____   Dir-3
> If dir-1 is given in fileInput path, does it includes files from dir-2 and dir-3?

Not directly, no. You need to implement the logic for this yourself,
see what happens in FileInputFormat.listStatus method and override
that functionality to recurse as you need it.

In the next release, this will be given by FileInputFormat itself,
controllable by a Configuration-settable property. See Zheng Shao's
patch for that feature at MAPREDUCE-1501 :)

Harsh J

View raw message