hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-8041) Consider remaining space during block blockplacement if dfs space is highly utilized
Date Wed, 01 Apr 2015 23:31:54 GMT
Kihwal Lee created HDFS-8041:
--------------------------------

             Summary: Consider remaining space during block blockplacement if dfs space is
highly utilized
                 Key: HDFS-8041
                 URL: https://issues.apache.org/jira/browse/HDFS-8041
             Project: Hadoop HDFS
          Issue Type: Improvement
            Reporter: Kihwal Lee


This feature is helpful in avoiding smaller nodes (i.e. heterogeneous environment) getting
constantly being full when the overall space utilization is over a certain threshold.  When
the utilization is low, balancer can keep up, but once the average per-node byte goes over
the capacity of the smaller nodes, they get full so quickly even after perfect balance.

This jira proposes an improvement that can be optionally enabled in order to slow down the
rate of space usage growth of smaller nodes if the overall storage utilization is over a configured
threshold.  It will not replace balancer, rather will help balancer keep up. Also, the primary
replica placement will not be affected. Only the replicas typically placed in a remote rack
will be subject to this check.

The appropriate threshold is cluster configuration specific. There is no generally good value
to set, thus it is disabled by default. We have seen cases where the threshold of 85% - 90%
would help. Figuring when {{totalSpaceUsed / numNodes}} becomes close to the capacity of a
smaller node is helpful in determining the threshold.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message