hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8818) Allow Balancer to run faster
Date Thu, 04 May 2017 13:43:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15996738#comment-15996738
] 

Kihwal Lee commented on HDFS-8818:
----------------------------------

bq.  Do you have HDFS-8824 in your runs?  I suspect the first run has it but the second one
does not.
It is the up-to-date branch-2.8, so all runs had it. About HDFS-8824, you do realize that
over time older nodes will end up with only small blocks, if it is set permanently?  It will
look good for quick balancing, but may not be good in long term. We run with the min block
size set to 1.

bq. The performance did have been improved around 100x.
Can you reveal more details on the nature of the testing?  It is unrealistic to expect 100x
in our typical use case with the base line being 2.7. 
What was your config when you tested on the 500 node cluster?  What was the nature of imbalance?
Did the default values work?  If not, how did you get there? Do you expect regular users to
easily get there?  At what point did you hit HDFS-11377?



> Allow Balancer to run faster
> ----------------------------
>
>                 Key: HDFS-8818
>                 URL: https://issues.apache.org/jira/browse/HDFS-8818
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: balancer & mover
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>             Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1
>
>         Attachments: bal1.png, bal2.png, h8818_20150723.patch, h8818_20150727.patch,
HDFS-8818-branch-2.7.00.patch
>
>
> The original design of Balancer is intentionally to make it run slowly so that the balancing
activities won't affect the normal cluster activities and the running jobs.
> There are new use case that cluster admin may choose to balance the cluster when the
cluster load is low, or in a maintain window.  So that we should have an option to allow Balancer
to run faster.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message