hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "maobaolong (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9038) DFS reserved space is erroneously counted towards non-DFS used.
Date Thu, 04 May 2017 08:27:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15996340#comment-15996340
] 

maobaolong commented on HDFS-9038:
----------------------------------

```java
public long getNonDfsUsed() throws IOException {
    long actualNonDfsUsed = getActualNonDfsUsed();
    if (actualNonDfsUsed < reserved) {
      return 0L;
    }
    return actualNonDfsUsed - reserved;
  }
```

The code block above is the function to caculate nonDfsUsed, but in fact it will let the result
to be 0L out of expect. Such as this following situation:

du.reserved  = 50G
Disk Capacity = 2048G
Disk Available = 2000G
Dfs used = 30G

usage.getUsed() = dirFile.getTotalSpace() - dirFile.getFreeSpace()
                            = 2048G - 2000G
                            = 48G
getActualNonDfsUsed  =  usage.getUsed() - getDfsUsed()
                                      =  48G - 30G
                                      = 18G
18G < 50G, so the function `getNonDfsUsed` actualNonDfsUsed < reserved, and the NonDfsUsed
will return 0, is that logic make sense?

> DFS reserved space is erroneously counted towards non-DFS used.
> ---------------------------------------------------------------
>
>                 Key: HDFS-9038
>                 URL: https://issues.apache.org/jira/browse/HDFS-9038
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 2.7.1
>            Reporter: Chris Nauroth
>            Assignee: Brahma Reddy Battula
>             Fix For: 2.8.0, 2.7.4, 3.0.0-alpha2
>
>         Attachments: GetFree.java, HDFS-9038-002.patch, HDFS-9038-003.patch, HDFS-9038-004.patch,
HDFS-9038-005.patch, HDFS-9038-006.patch, HDFS-9038-007.patch, HDFS-9038-008.patch, HDFS-9038-009.patch,
HDFS-9038-010.patch, HDFS-9038-011.patch, HDFS-9038-branch-2.7.patch, HDFS-9038.patch
>
>
> HDFS-5215 changed the DataNode volume available space calculation to consider the reserved
space held by the {{dfs.datanode.du.reserved}} configuration property.  As a side effect,
reserved space is now counted towards non-DFS used.  I don't believe it was intentional to
change the definition of non-DFS used.  This issue proposes restoring the prior behavior:
do not count reserved space towards non-DFS used.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message