hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8041) Consider remaining space during block blockplacement if dfs space is highly utilized
Date Thu, 09 Apr 2015 00:11:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14486383#comment-14486383

Hadoop QA commented on HDFS-8041:

{color:green}+1 overall{color}.  Here are the results of testing the latest attachment 
  against trunk revision 285b31e.

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 1 new or modified
test files.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

    {color:green}+1 eclipse:eclipse{color}.  The patch built with eclipse:eclipse.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version
2.0.3) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:green}+1 core tests{color}.  The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10214//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10214//console

This message is automatically generated.

> Consider remaining space during block blockplacement if dfs space is highly utilized
> ------------------------------------------------------------------------------------
>                 Key: HDFS-8041
>                 URL: https://issues.apache.org/jira/browse/HDFS-8041
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Kihwal Lee
>            Assignee: Kihwal Lee
>         Attachments: HDFS-8041.v1.patch, HDFS-8041.v2.patch, HDFS-8041.v3.patch, HDFS-8041.v4.patch
> This feature is helpful in avoiding smaller nodes (i.e. heterogeneous environment) getting
constantly being full when the overall space utilization is over a certain threshold.  When
the utilization is low, balancer can keep up, but once the average per-node byte goes over
the capacity of the smaller nodes, they get full so quickly even after perfect balance.
> This jira proposes an improvement that can be optionally enabled in order to slow down
the rate of space usage growth of smaller nodes if the overall storage utilization is over
a configured threshold.  It will not replace balancer, rather will help balancer keep up.
Also, the primary replica placement will not be affected. Only the replicas typically placed
in a remote rack will be subject to this check.
> The appropriate threshold is cluster configuration specific. There is no generally good
value to set, thus it is disabled by default. We have seen cases where the threshold of 85%
- 90% would help. Figuring when {{totalSpaceUsed / numNodes}} becomes close to the capacity
of a smaller node is helpful in determining the threshold.

This message was sent by Atlassian JIRA

View raw message