hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
Date Tue, 15 Dec 2015 00:45:46 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15057077#comment-15057077

Arpit Agarwal commented on HDFS-9038:

[~brahma], thanks for your patience as we work through the math. It may be useful to describe
your proposed derivation as a Jira comment before you post another patch.

[~cnauroth] I think you are right in conclusion but there is a misstep in the equations in
[this comment|https://issues.apache.org/jira/browse/HDFS-9038?focusedCommentId=15051894&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15051894].
{{reserved}} _should_ cancel out and it should not factor in the final computation.

The problem with the v005 patch we missed earlier is that getCapacity() subtracts reserved
space. We should use the raw capacity.

One way to fix it is:
public long getNonDfsUsed() throws IOException {
  long totalFreeSpace = currentDir.getUsableSpace();
  long nonDfsUsed = getCapacity() + reserved - getDfsUsed() - totalFreeSpace;
  return (nonDfsUsed >= 0) ? nonDfsUsed : 0;

Then the derivation becomes:
1: non-DFS used = getCapacity() + reserved - getDfsUsed() - totalFreeSpace
2:              = usage.getCapacity() - reserved + reserved - getDfsUsed() - totalFreeSpace
3:              = usage.getCapacity() - getDfsUsed() - totalFreeSpace
4:              = File#getTotalSpace - getDfsUsed() - File#getFreeSpace

Hope that makes sense.

> DFS reserved space is erroneously counted towards non-DFS used.
> ---------------------------------------------------------------
>                 Key: HDFS-9038
>                 URL: https://issues.apache.org/jira/browse/HDFS-9038
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 2.7.1
>            Reporter: Chris Nauroth
>            Assignee: Brahma Reddy Battula
>         Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch, HDFS-9038-004.patch, HDFS-9038-005.patch,
HDFS-9038-006.patch, HDFS-9038-007.patch, HDFS-9038.patch
> HDFS-5215 changed the DataNode volume available space calculation to consider the reserved
space held by the {{dfs.datanode.du.reserved}} configuration property.  As a side effect,
reserved space is now counted towards non-DFS used.  I don't believe it was intentional to
change the definition of non-DFS used.  This issue proposes restoring the prior behavior:
do not count reserved space towards non-DFS used.

This message was sent by Atlassian JIRA

View raw message