Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Tue, 23 Jun 2015 04:00:09 +0000 (UTC)
From: "Ming Ma (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12475465.1285808088000.143776.1435032009257@Atlassian.JIRA>
In-Reply-To: <JIRA.12475465.1285808088000@Atlassian.JIRA>
References: <JIRA.12475465.1285808088000@Atlassian.JIRA>
 <JIRA.12475465.1285808088742@arcas>
Subject: [jira] [Commented] (HDFS-1431) Balancer should work with the logic
 of BlockPlacementPolicy
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-1431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14597133#comment-14597133 ] 

Ming Ma commented on HDFS-1431:
-------------------------------

When I met with [~andrew.wang], [~ctrezzo], [~atm], [~cmccabe] the other day, we had brief discussion about balancer. To make balancer use BlockPlacementPolicy, alternatively we can run balancer inside namenode. Namenode already has the necessary information. It needs to provide balancer throttling with some refactoring. But overall it seems it shouldn't create much overhead on namenode. It will be great to heard from others about this approach on potential issues such as scale and performance.

> Balancer should work with the logic of BlockPlacementPolicy
> -----------------------------------------------------------
>
>                 Key: HDFS-1431
>                 URL: https://issues.apache.org/jira/browse/HDFS-1431
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: balancer & mover
>    Affects Versions: 0.22.0
>            Reporter: Scott Chen
>            Assignee: Scott Chen
>         Attachments: HDFS-1431.txt
>
>
> Currently Balancer does not obtain information from BlockPlacementPolicy so it can transfer the blocks without checking with BlockPlacementPolicy.
> This causes the policy break after balancing the cluster.
> There are some new policies proposed in HDFS-1094 and MAPREDUCE-1831 in which the block placement follows some pattern.
> The pattern can be broken by Balancer.
> I propose that we add the following method in BlockPlacementPolicy:
> {code}
>   abstract public boolean canBeMoved(String fileName, Block block,
>     DatanodeInfo source, DatanodeInfo destination);
> {code}
> And make Balancer use it in
> {code}
>   private boolean isGoodBlockCandidate(Source source,
>       BalancerDatanode target, BalancerBlock block)
> {code}
> What do you think?


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)