hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Nauroth (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9038) Reserved space is erroneously counted towards non-DFS used.
Date Wed, 02 Dec 2015 01:24:11 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15035067#comment-15035067

Chris Nauroth commented on HDFS-9038:

bq. Brahma changed this at my suggestion. Exposing reserved bytes through nonDfsUsed was inadvertent
side effect of HDFS-6898.

Sorry, I missed that earlier comment.  I understand the rationale now, so I'll withdraw my
comment about changing it.

Regarding {{File#getUsableSpace}} vs. {{File#getFreeSpace}}, I've been approaching this as
a regression introduced by HDFS-5215, with the goal of restoring the pre-HDFS-5215 behavior
as closely as possible.  Before HDFS-5215, non-DFS used was based on {{File#getUsableSpace}},
indirectly because of use of {{getAvailable}} in the calculation.  This meant that "free but
unusable" space would have counted towards non-DFS used.

However, that would then lead me to expect seeing a non-zero non-DFS used for the typical
5% reserved for privileged users.  In practice, I haven't seen that happen though.  I'm not
sure why not.

I tried digging through revision history for past decisions.  The use of {{File#getUsableSpace}}
traces back to HADOOP-5958, when the code switched from forking to {{df}} to using then-new
JDK 1.6 APIs.  There is a question posted about which method to use, but no specific discussion
about why {{File#getUsableSpace}} was chosen.

I don't think switching to {{File#getFreeSpace}} is necessarily wrong, just different from
the pre-HDFS-5215 calculation, which might cause some surprises.

> Reserved space is erroneously counted towards non-DFS used.
> -----------------------------------------------------------
>                 Key: HDFS-9038
>                 URL: https://issues.apache.org/jira/browse/HDFS-9038
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 2.7.1
>            Reporter: Chris Nauroth
>            Assignee: Brahma Reddy Battula
>         Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch, HDFS-9038-004.patch, HDFS-9038-005.patch,
> HDFS-5215 changed the DataNode volume available space calculation to consider the reserved
space held by the {{dfs.datanode.du.reserved}} configuration property.  As a side effect,
reserved space is now counted towards non-DFS used.  I don't believe it was intentional to
change the definition of non-DFS used.  This issue proposes restoring the prior behavior:
do not count reserved space towards non-DFS used.

This message was sent by Atlassian JIRA

View raw message