hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo (Nicholas), SZE (Resolved) (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (HADOOP-6963) Fix FileUtil.getDU. It should not include the size of the directory or follow symbolic links
Date Thu, 05 Apr 2012 19:52:24 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-6963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Tsz Wo (Nicholas), SZE resolved HADOOP-6963.

          Resolution: Fixed
       Fix Version/s:     (was: 3.0.0)
                          (was: 2.0.0)
    Target Version/s: 1.0.3, 0.23.3  (was: 0.23.3, 1.0.3)

I also committed the patch to branch-1 and branch-1.0.
> Fix FileUtil.getDU. It should not include the size of the directory or follow symbolic
> --------------------------------------------------------------------------------------------
>                 Key: HADOOP-6963
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6963
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions:, 0.23.1
>            Reporter: Owen O'Malley
>            Assignee: Ravi Prakash
>            Priority: Critical
>             Fix For: 1.0.3, 0.23.3
>         Attachments: HADOOP-6963.branch-1.0.2.patch, HADOOP-6963.branch-1.0.2.patch,
HADOOP-6963.branch-1.patch, HADOOP-6963.branch-23.patch, HADOOP-6963.branch-23.patch, HADOOP-6963.branch-23.patch
> The getDU method should not include the size of the directory. The Java interface says
that the value is undefined and in Linux/Sun it gets the 4096 for the inode. Clearly this
isn't useful.
> It also recursively calls itself. In case the directory has a symbolic link forming a
cycle, getDU keeps spinning in the cycle. In our case, we saw this in the org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCacheObjects
call. This prevented other tasks on the same node from committing, causing the TT to become
effectively useless (because the JT thinks it already has enough tasks running)

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message