hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zheng Shao (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1501) FileInputFormat to support multi-level/recursive directory listing
Date Mon, 22 Feb 2010 22:32:28 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12836963#action_12836963
] 

Zheng Shao commented on MAPREDUCE-1501:
---------------------------------------

Thanks Dhruba. I missed the part "and other hidden directories". We do call PathFilter on
the sub directories as well (see addInputPathRecursively(...)). Is that good enough or we
want to split the PathFilters for files and the PathFilters for directories?


> FileInputFormat to support multi-level/recursive directory listing
> ------------------------------------------------------------------
>
>                 Key: MAPREDUCE-1501
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1501
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>            Reporter: Zheng Shao
>            Assignee: Zheng Shao
>         Attachments: MAPREDUCE-1501.1.branch-0.20.patch, MAPREDUCE-1501.1.trunk.patch
>
>
> As we have seen multiple times in the mailing list, users want to have the capability
of getting all files out of a multi-level directory structure.
> 4/1/2008: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200804.mbox/%3Ce75c02ef0804011433x144813e6x2450da7883de3aca@mail.gmail.com%3E
> 2/3/2009: http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200902.mbox/%3C7F80089C-3E7F-4330-90BA-6F1C5B0B0F3F@nist.gov%3E
> 6/2/2009: http://mail-archives.apache.org/mod_mbox/hadoop-common-user/200906.mbox/%3C4A258A16.8050300@darose.net%3E
> One solution that our users had is to write a new FileInputFormat, but that means all
existing FileInputFormat subclasses need to be changed in order to support this feature.
> We can easily provide a JobConf option (which defaults to false) to {{FileInputFormat.listStatus(...)}}
to recursively go into directory structure.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message