hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5029) Recursively take all files in the directories of a root directory
Date Tue, 26 Feb 2013 10:32:12 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13586994#comment-13586994
] 

Steve Loughran commented on MAPREDUCE-5029:
-------------------------------------------

Please can you see if this problem still exists on Hadoop trunk. There have been changes to
do iterative enumeration of directory contents rather than the ls * -see {{FileSystem.listFiles()}}.
If this method isn't being used in MR jobs, maybe it could be.
                
> Recursively take all files in the directories of a root directory
> -----------------------------------------------------------------
>
>                 Key: MAPREDUCE-5029
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5029
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>    Affects Versions: 0.20.2
>            Reporter: Abhilash S R
>
> Suppose we have a root directories with 1000's of sub directories and in each directory
there can be 100's of files.So while specifying the root directory in the input path in map-reduce
the program crashes due to sub directories in the root directory.So if this feature is includes
in latest version it will be great helpful for programers.  

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message