hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-8818) Allow Balancer to run faster
Date Wed, 03 May 2017 19:18:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995477#comment-15995477
] 

Kihwal Lee edited comment on HDFS-8818 at 5/3/17 7:17 PM:
----------------------------------------------------------

I set dfs.datanode.balance.max.concurrent.moves to 15, which is what we used to run with 2.7.
71 thread pools with 15 threads in each created. The 72th one is small and then started to
fail.   So I still see "skipping..." message in the log.  The throughput is still visibly
lower, after a minute of initial spike.  The iteration lasts a bit longer, but the average
throughput is still similarly low.  Other config values are all set to default.

I believe I have presented enough evidence.


was (Author: kihwal):
I set the dfs.datanode.balance.max.concurrent.moves to 15, which is what we used to use with
2.7. 71 thread pools with 15 threads in each created. The 72th one is small and then started
to fail.   So I still see "skipping..." message in the log.  The throughput is still visibly
lower, after a minute of initial spike.  The iteration lasts a bit longer, but the average
throughput is still similarly low.  Other config values are all set to default.

I believe I have presented enough evidence.

> Allow Balancer to run faster
> ----------------------------
>
>                 Key: HDFS-8818
>                 URL: https://issues.apache.org/jira/browse/HDFS-8818
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: balancer & mover
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>             Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1
>
>         Attachments: bal1.png, bal2.png, h8818_20150723.patch, h8818_20150727.patch,
HDFS-8818-branch-2.7.00.patch
>
>
> The original design of Balancer is intentionally to make it run slowly so that the balancing
activities won't affect the normal cluster activities and the running jobs.
> There are new use case that cluster admin may choose to balance the cluster when the
cluster load is low, or in a maintain window.  So that we should have an option to allow Balancer
to run faster.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message