hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kihwal Lee (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8818) Allow Balancer to run faster
Date Wed, 03 May 2017 13:24:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994840#comment-15994840

Kihwal Lee commented on HDFS-8818:

We see this problem in EVERY single cluster we tried this on.

The first big three bumps (20 min each) is 2.8 with this change reverted.  The following small
ones are the result of non-reverted 2.8 with the hang fix (HDFS-11377).  It is still no where
near what it used to be.

This is the stock 2.8 balancer without HDFS-11377.

I don't doubt that it worked great for the case you designed and tested for.  However, please
do realize that it will be a regression for many other users.  I am not denying the shortcomings
of the existing design. But the new design has clear issues, which cannot be worked around
with config tweaks.  Even if it works better for some cases, it is a regression.

> Allow Balancer to run faster
> ----------------------------
>                 Key: HDFS-8818
>                 URL: https://issues.apache.org/jira/browse/HDFS-8818
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: balancer & mover
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>             Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1
>         Attachments: bal1.png, bal2.png, h8818_20150723.patch, h8818_20150727.patch,
> The original design of Balancer is intentionally to make it run slowly so that the balancing
activities won't affect the normal cluster activities and the running jobs.
> There are new use case that cluster admin may choose to balance the cluster when the
cluster load is low, or in a maintain window.  So that we should have an option to allow Balancer
to run faster.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message