hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-6963) Fix FileUtil.getDU. It should not include the size of the directory or follow symbolic links
Date Thu, 05 Apr 2012 18:38:28 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13247491#comment-13247491

Hudson commented on HADOOP-6963:

Integrated in Hadoop-Common-trunk-Commit #2005 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2005/])
    HADOOP-6963. In FileUtil.getDU(..), neither include the size of directories nor follow
symbolic links.  Contributed by Ravi Prakash (Revision 1309994)

     Result = SUCCESS
szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1309994
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/pom.xml
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java

> Fix FileUtil.getDU. It should not include the size of the directory or follow symbolic
> --------------------------------------------------------------------------------------------
>                 Key: HADOOP-6963
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6963
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions:, 0.23.1
>            Reporter: Owen O'Malley
>            Assignee: Ravi Prakash
>            Priority: Critical
>             Fix For: 2.0.0, 3.0.0
>         Attachments: HADOOP-6963.branch-1.0.2.patch, HADOOP-6963.branch-1.0.2.patch,
HADOOP-6963.branch-23.patch, HADOOP-6963.branch-23.patch, HADOOP-6963.branch-23.patch
> The getDU method should not include the size of the directory. The Java interface says
that the value is undefined and in Linux/Sun it gets the 4096 for the inode. Clearly this
isn't useful.
> It also recursively calls itself. In case the directory has a symbolic link forming a
cycle, getDU keeps spinning in the cycle. In our case, we saw this in the org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCacheObjects
call. This prevented other tasks on the same node from committing, causing the TT to become
effectively useless (because the JT thinks it already has enough tasks running)

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message