hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9038) Reserved space is erroneously counted towards non-DFS used.
Date Tue, 01 Dec 2015 23:13:11 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15034818#comment-15034818
] 

Arpit Agarwal commented on HDFS-9038:
-------------------------------------

Thanks for the updated patch [~brahmareddy].

bq. 3. I think we should cap the returned value to 0 as a matter of defensive coding against
negative values. 
Hi [~cnauroth], this is a good idea.

bq. 2. The latest getNonDfsUsed does not include reservedForReplicas. I think it should, since
the reservedForReplicas amount is effectively in use by HDFS.
Brahma changed this at my suggestion. Exposing reserved bytes through nonDfsUsed was inadvertent
side effect of HDFS-6898. I see Brahma filed HDFS-9481 to expose reserved bytes as a separate
counter which should present a clearer picture to administrators. What do you think?

bq. I think we should stick with File#getUsableSpace here
I looked at the JDK code to find the difference between {{File#getUsableSpace}} and {{File#getFreeSpace}}.
http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/9b8c96f96a0f/src/solaris/native/java/io/UnixFileSystem_md.c#l412

So it comes down to the difference between {{f_bavail}} and {{f_bfree}}.

{code}
           struct statvfs {
               unsigned long  f_bsize;    /* filesystem block size */
               unsigned long  f_frsize;   /* fragment size */
               fsblkcnt_t     f_blocks;   /* size of fs in f_frsize units */
               fsblkcnt_t     f_bfree;    /* # free blocks */
               fsblkcnt_t     f_bavail;   /* # free blocks for unprivileged users */
{code}
{{f_bavail}} does not include some system-reserved partition space. The consequence of using
{{f_bavail}} is that non-DFS used will include this space which is 5% of the volume size on
ext2fs and successors, because {{File#getTotalSpace}} counts everything.
http://www.microhowto.info/howto/reduce_the_space_reserved_for_root_on_an_ext2_ext3_or_ext4_filesystem.html

This may be confusing to administrators.

> Reserved space is erroneously counted towards non-DFS used.
> -----------------------------------------------------------
>
>                 Key: HDFS-9038
>                 URL: https://issues.apache.org/jira/browse/HDFS-9038
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode
>    Affects Versions: 2.7.1
>            Reporter: Chris Nauroth
>            Assignee: Brahma Reddy Battula
>         Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch, HDFS-9038-004.patch, HDFS-9038-005.patch,
HDFS-9038.patch
>
>
> HDFS-5215 changed the DataNode volume available space calculation to consider the reserved
space held by the {{dfs.datanode.du.reserved}} configuration property.  As a side effect,
reserved space is now counted towards non-DFS used.  I don't believe it was intentional to
change the definition of non-DFS used.  This issue proposes restoring the prior behavior:
do not count reserved space towards non-DFS used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message