hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinayakumar B (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
Date Fri, 11 Dec 2015 02:25:11 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15052039#comment-15052039

Vinayakumar B commented on HDFS-9038:

Thanks [~cnauroth] for the explanation.

Lets go through one simple example.
Lets consider following
1. Disk Capacity - 10GB -- {{File#getTotalSpace}}
2. DFS reserved - 1GB   -- {{dfs.datanode.du.reserved}}
3. DFS Usage     -  2GB  -- {{dfsUsed}}
4. Usage by other files - 3GB -- Actual {{non-dfsusage}}.

Assuming, System reserved ( which is ~5%) as 0 for this example.
total disk used is 5GB = 2GB by dfs files + 3GB by non-dfs files.
{{File.getUsableSpace()}} returns 5GB.

Lets substitute this values in derived expressions.
*Before HDFS-5215*
nonDfsUsage=File#getTotalSpace - dfs.datanode.du.reserved - dfsUsed - File#getUsableSpace

nonDfsUsage = 10G - 1G - 2G - 5G
nonDfsUsage = 2G.
{color:red}{{nonDfsUsage=2GB}} is not same as actual non-dfs usage, 3GB.{color}

*After HDFS-5215* and as per patch.
nonDfsUsage=(File#getTotalSpace - dfs.datanode.du.reserved) - dfsUsed - (File#getUsableSpace
- dfs.datanode.du.reserved)
nonDfSUsage= (10G - 1G ) - 2G - (5G - 1G)
nonDfSUsage = 9G - 2G - 4G
nonDfsUsage = 3G
{color:green}{{nonDfsUsage = 3G}} is same as expected actual usage. i.e. 3G{color}

By seeing the above example, 
I believe, post HDFS-5215, calculation turns out to be the correct one. 
{{dfs.datanode,du.reserved}} is expected to be the freespace, which should not be used by
DFS. So its reasonable to subtract this in getAvailable(), but not while finding the non-dfs

Do you agree [~cnauroth]?

> DFS reserved space is erroneously counted towards non-DFS used.
> ---------------------------------------------------------------
>                 Key: HDFS-9038
>                 URL: https://issues.apache.org/jira/browse/HDFS-9038
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 2.7.1
>            Reporter: Chris Nauroth
>            Assignee: Brahma Reddy Battula
>         Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch, HDFS-9038-004.patch, HDFS-9038-005.patch,
HDFS-9038-006.patch, HDFS-9038.patch
> HDFS-5215 changed the DataNode volume available space calculation to consider the reserved
space held by the {{dfs.datanode.du.reserved}} configuration property.  As a side effect,
reserved space is now counted towards non-DFS used.  I don't believe it was intentional to
change the definition of non-DFS used.  This issue proposes restoring the prior behavior:
do not count reserved space towards non-DFS used.

This message was sent by Atlassian JIRA

View raw message