hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinayakumar B (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
Date Sat, 12 Dec 2015 02:27:47 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15053971#comment-15053971

Vinayakumar B commented on HDFS-9038:

bq. dfs.datanode.du.reserved means that the administrator intentionally wants to hold back
a portion of the volume, essentially making it invisible to HDFS. This is the same reason
that dfs.datanode.du.reserved gets subtracted from the capacity calculation. This is why I
think 3 GB is incorrect in the example above. Non-DFS usage is really "unexpected non-DFS
usage". By setting dfs.datanode.du.reserved to 1 GB, you have stated that up to 1 GB of usage
by something other than HDFS is expected, so it's incorrect to count it as part of "unexpected
non-DFS usage".
So you mean, when 1GB is configured as reserved, 'Non-DFS' usage metric should show 2GB, even
though actual usage by files other than HDFS is 3GB.?
I was thinking 'NonDfsUsage' should show actual non-dfs usage. Not only unexpected part of
it. So if actual non-dfs usage is within this reserved limit, metric should show 0. right?

In that case, wouldn't it be better to rename the 'NonDfsUsage' metric to 'UnexpectedNonDfsUsage'
to make it more clear? Or mention somewhere to clear confusion in people like me.

> DFS reserved space is erroneously counted towards non-DFS used.
> ---------------------------------------------------------------
>                 Key: HDFS-9038
>                 URL: https://issues.apache.org/jira/browse/HDFS-9038
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 2.7.1
>            Reporter: Chris Nauroth
>            Assignee: Brahma Reddy Battula
>         Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch, HDFS-9038-004.patch, HDFS-9038-005.patch,
HDFS-9038-006.patch, HDFS-9038-007.patch, HDFS-9038.patch
> HDFS-5215 changed the DataNode volume available space calculation to consider the reserved
space held by the {{dfs.datanode.du.reserved}} configuration property.  As a side effect,
reserved space is now counted towards non-DFS used.  I don't believe it was intentional to
change the definition of non-DFS used.  This issue proposes restoring the prior behavior:
do not count reserved space towards non-DFS used.

This message was sent by Atlassian JIRA

View raw message