Return-Path: <hdfs-issues-return-165389-archive-asf-public=cust-asf.ponee.io@hadoop.apache.org>
X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io
Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io
Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183])
	by cust-asf2.ponee.io (Postfix) with ESMTP id B1620200B64
	for <archive-asf-public-internal@cust-asf2.ponee.io>; Tue,  2 Aug 2016 15:09:22 +0200 (CEST)
Received: by cust-asf.ponee.io (Postfix)
	id B00C8160A76; Tue,  2 Aug 2016 13:09:22 +0000 (UTC)
Delivered-To: archive-asf-public@cust-asf.ponee.io
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by cust-asf.ponee.io (Postfix) with SMTP id E9BFC160AB0
	for <archive-asf-public@cust-asf.ponee.io>; Tue,  2 Aug 2016 15:09:21 +0200 (CEST)
Received: (qmail 93913 invoked by uid 500); 2 Aug 2016 13:09:20 -0000
Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:hdfs-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:hdfs-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:hdfs-issues@hadoop.apache.org>
List-Id: <hdfs-issues.hadoop.apache.org>
Delivered-To: mailing list hdfs-issues@hadoop.apache.org
Received: (qmail 93599 invoked by uid 99); 2 Aug 2016 13:09:20 -0000
Received: from arcas.apache.org (HELO arcas) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Aug 2016 13:09:20 +0000
Received: from arcas.apache.org (localhost [127.0.0.1])
	by arcas (Postfix) with ESMTP id 9A3562C0059
	for <hdfs-issues@hadoop.apache.org>; Tue,  2 Aug 2016 13:09:20 +0000 (UTC)
Date: Tue, 2 Aug 2016 13:09:20 +0000 (UTC)
From: "Yiqun Lin (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12994277.1470142765000.198818.1470143360628@Atlassian.JIRA>
In-Reply-To: <JIRA.12994277.1470142765000@Atlassian.JIRA>
References: <JIRA.12994277.1470142765000@Atlassian.JIRA> <JIRA.12994277.1470142765503@arcas>
Subject: [jira] [Updated] (HDFS-10716) The target node should be removed in
 balancer when the scheduled size of bytes that firstly reduced to less than
 0
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394
archived-at: Tue, 02 Aug 2016 13:09:22 -0000


     [ https://issues.apache.org/jira/browse/HDFS-10716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yiqun Lin updated HDFS-10716:
-----------------------------
    Attachment: HDFS-10716.001.patch

> The target node should be removed in balancer when the scheduled size of bytes that firstly reduced to less than 0
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-10716
>                 URL: https://issues.apache.org/jira/browse/HDFS-10716
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: balancer & mover
>            Reporter: Yiqun Lin
>            Assignee: Yiqun Lin
>         Attachments: HDFS-10716.001.patch, failing.log
>
>
> In HDFS-10602, we found a failing case that the balancer moves data always between 2 DNs. And it made the balancer can't be finished. I debug the code for this, I found there seems a bug in choosing pending blocks in {{Dispatcher.Source.chooseNextMove}}.
> The codes:
> {code}
>     private PendingMove chooseNextMove() {
>       for (Iterator<Task> i = tasks.iterator(); i.hasNext();) {
>         final Task task = i.next();
>         final DDatanode target = task.target.getDDatanode();
>         final PendingMove pendingBlock = new PendingMove(this, task.target);
>         if (target.addPendingBlock(pendingBlock)) {
>           // target is not busy, so do a tentative block allocation
>           if (pendingBlock.chooseBlockAndProxy()) {
>             long blockSize = pendingBlock.reportedBlock.getNumBytes(this);
>             incScheduledSize(-blockSize);
>             task.size -= blockSize;
>             // If the size of bytes that need to be moved was first reduced to less than 0
>             // it should also be removed.
>             if (task.size == 0) {
>               i.remove();
>             }
>             return pendingBlock;
>             //...
> {code}
> The value of task.size was assigned in {{Balancer#matchSourceWithTargetToMove}}
> {code}
>     long size = Math.min(source.availableSizeToMove(), target.availableSizeToMove());
>     final Task task = new Task(target, size);
> {code}
> This value was depended on the source and target node, and this value will not always can be reduced to 0 in choosing pending blocks. And then, it will still move the data to the target node even if the size of bytes that needed to move has been already reduced less than 0. And finally it will make the data imbalance again in cluster.
> We can opitimize for this as this title mentioned, I think this can speed the balancer.
> Can see the failling logs for this.(Concentrating on the change record for the scheduled size of target node. That's my added info for debug, like this).
> {code}
> 2016-08-01 16:51:57,492 [pool-51-thread-1] INFO  balancer.Dispatcher (Dispatcher.java:chooseNextMove(799)) - TargetNode: 58794, bytes scheduled to move, after: -67, before: 33
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org