hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brahma Reddy Battula (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9038) Reserved space is erroneously counted towards non-DFS used.
Date Thu, 03 Dec 2015 12:41:11 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037733#comment-15037733

Brahma Reddy Battula commented on HDFS-9038:

bq. I don't think switching to File#getFreeSpace is necessarily wrong, just different from
the pre-HDFS-5215 calculation, which might cause some surprises.
Yes, its surprising. 
Difference in HDFS user about that {{~5%}} space, 
before patch --> It was taken as already used by someone (non-dfs) because its not usable
by HDFS.
after patch --> It will display actual usage of someone (non-dfs) even though its not usable.

Only matter is, value of metric. Which doesnt affect HDFS functionality anyway.
But assumption in calculattion in tests {{capacityUsed + capacityRemaining + capacityUsedNonDFS
== capacityTotal}} might need to change, as {{capacityRemaining}} doesnt include, that {{~5%}}.

>From the above Test Failure of {{TestNameNodeMetrics}}, below values have got
|CapacityUsed |147456|
|CapacityRemaining |20273827930112|
|CapacityUsedNonDFS |1859806781440|
|| Difference || ~5% ||

 *Now we have two options to go ahead with the patch.* 
* Update the tests with assumption of extra {{~5-6%}} space for privileged users
* fallback to {{File.getUsablespace()}}

Which one do you prefer [~cnauroth],[~arpitagarwal]..?

> Reserved space is erroneously counted towards non-DFS used.
> -----------------------------------------------------------
>                 Key: HDFS-9038
>                 URL: https://issues.apache.org/jira/browse/HDFS-9038
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 2.7.1
>            Reporter: Chris Nauroth
>            Assignee: Brahma Reddy Battula
>         Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch, HDFS-9038-004.patch, HDFS-9038-005.patch,
> HDFS-5215 changed the DataNode volume available space calculation to consider the reserved
space held by the {{dfs.datanode.du.reserved}} configuration property.  As a side effect,
reserved space is now counted towards non-DFS used.  I don't believe it was intentional to
change the definition of non-DFS used.  This issue proposes restoring the prior behavior:
do not count reserved space towards non-DFS used.

This message was sent by Atlassian JIRA

View raw message