hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo Nicholas Sze (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8818) Allow Balancer to run faster
Date Tue, 02 May 2017 21:52:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15993855#comment-15993855
] 

Tsz Wo Nicholas Sze commented on HDFS-8818:
-------------------------------------------

Hi [~kihwal], let's have a more detailed discussion before reverting.  

The patch here (a thread pool per datanode pair) is indeed an improvement for the previous
design (a global thread pool) since it limits the number of threads assigned to a particular
datanode pair.  Previously, if the first datanode pair has a lot of pending moves, all the
threads will be used to execute the moves for that pair so that it will be very slow since
it cannot utilize the entire network.

We also has tested the new code a lot and see significant performance improvement.

Have you tested it with HDFS-11377?

> Allow Balancer to run faster
> ----------------------------
>
>                 Key: HDFS-8818
>                 URL: https://issues.apache.org/jira/browse/HDFS-8818
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: balancer & mover
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Kihwal Lee
>             Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1
>
>         Attachments: h8818_20150723.patch, h8818_20150727.patch, HDFS-8818-branch-2.7.00.patch
>
>
> The original design of Balancer is intentionally to make it run slowly so that the balancing
activities won't affect the normal cluster activities and the running jobs.
> There are new use case that cluster admin may choose to balance the cluster when the
cluster load is low, or in a maintain window.  So that we should have an option to allow Balancer
to run faster.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message