hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo Nicholas Sze (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8818) Allow Balancer to run faster
Date Thu, 04 May 2017 21:03:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15997429#comment-15997429

Tsz Wo Nicholas Sze commented on HDFS-8818:

> Can you reveal more details on the nature of the testing? It is unrealistic to expect
100x in our typical use case with the base line being 2.7.

In our tests, we ran balancer over a 500-node cluster.  We were only able to get ~5GB per
minute before.  Then, we were able to get ~500GB per minute after a serious of balancer improvement
including this.  This JIRA is the most critical since, without this, balancer schedules most
the moves in the first few datanode pairs and the remaining datanodes are mostly idle.

Below are the confs:

- Datanode
dfs.datanode.balance.max.concurrent.moves: 4 x #disks
dfs.datanode.balance.bandwidthPerSec: 10737418240 (=10GB)

- Balancer
dfs.datanode.balance.max.concurrent.moves: 4 x #disks
dfs.balancer.moverThreads: 20,000
dfs.balancer.max-size-to-move: 107374182400 (=100GB)
dfs.balancer.getBlocks.min-block-size: 104857600 (=100MB)

> Allow Balancer to run faster
> ----------------------------
>                 Key: HDFS-8818
>                 URL: https://issues.apache.org/jira/browse/HDFS-8818
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: balancer & mover
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>             Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1
>         Attachments: bal1.png, bal2.png, h8818_20150723.patch, h8818_20150727.patch,
> The original design of Balancer is intentionally to make it run slowly so that the balancing
activities won't affect the normal cluster activities and the running jobs.
> There are new use case that cluster admin may choose to balance the cluster when the
cluster load is low, or in a maintain window.  So that we should have an option to allow Balancer
to run faster.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message