hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5029) Recursively take all files in the directories of a root directory
Date Tue, 26 Feb 2013 10:32:12 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13586994#comment-13586994

Steve Loughran commented on MAPREDUCE-5029:

Please can you see if this problem still exists on Hadoop trunk. There have been changes to
do iterative enumeration of directory contents rather than the ls * -see {{FileSystem.listFiles()}}.
If this method isn't being used in MR jobs, maybe it could be.
> Recursively take all files in the directories of a root directory
> -----------------------------------------------------------------
>                 Key: MAPREDUCE-5029
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5029
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>    Affects Versions: 0.20.2
>            Reporter: Abhilash S R
> Suppose we have a root directories with 1000's of sub directories and in each directory
there can be 100's of files.So while specifying the root directory in the input path in map-reduce
the program crashes due to sub directories in the root directory.So if this feature is includes
in latest version it will be great helpful for programers.  

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message