hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daryn Sharp (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8818) Allow Balancer to run faster
Date Wed, 03 May 2017 19:04:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995450#comment-15995450
] 

Daryn Sharp commented on HDFS-8818:
-----------------------------------

bq. The new design is more flexible than the previous one since we can control the number
of thread per datanode pair.

That sounds good on paper but flexibility does not negate the fact it's proven not to scale.
 I'm sure the redesign works great on a couple dozen node cluster.  As illustrated by Kihwal,
it limps along on a 280 node cluster running slower than before and is virtually unusable
on multi-thousand node clusters even with HDFS-11377.

This has to be fixed in a manner that restores previous performance or be reverted.  A jira
touting "run faster" can't make the balancer slower and unfit for production...

> Allow Balancer to run faster
> ----------------------------
>
>                 Key: HDFS-8818
>                 URL: https://issues.apache.org/jira/browse/HDFS-8818
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: balancer & mover
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>             Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1
>
>         Attachments: bal1.png, bal2.png, h8818_20150723.patch, h8818_20150727.patch,
HDFS-8818-branch-2.7.00.patch
>
>
> The original design of Balancer is intentionally to make it run slowly so that the balancing
activities won't affect the normal cluster activities and the running jobs.
> There are new use case that cluster admin may choose to balance the cluster when the
cluster load is low, or in a maintain window.  So that we should have an option to allow Balancer
to run faster.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message