Return-Path: X-Original-To: apmail-hadoop-common-commits-archive@www.apache.org Delivered-To: apmail-hadoop-common-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0E03C18C32 for ; Tue, 3 Nov 2015 21:09:32 +0000 (UTC) Received: (qmail 15563 invoked by uid 500); 3 Nov 2015 21:09:19 -0000 Delivered-To: apmail-hadoop-common-commits-archive@hadoop.apache.org Received: (qmail 14959 invoked by uid 500); 3 Nov 2015 21:09:18 -0000 Mailing-List: contact common-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-commits@hadoop.apache.org Received: (qmail 14317 invoked by uid 99); 3 Nov 2015 21:09:18 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 03 Nov 2015 21:09:18 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 16181E055E; Tue, 3 Nov 2015 21:09:18 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: jing9@apache.org To: common-commits@hadoop.apache.org Date: Tue, 03 Nov 2015 21:09:36 -0000 Message-Id: <152adc77e6794481b2c3fa1452d11f8b@git.apache.org> In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [20/50] hadoop git commit: HDFS-4937. ReplicationMonitor can infinite-loop in BlockPlacementPolicyDefault#chooseRandom(). Contributed by Kihwal Lee. HDFS-4937. ReplicationMonitor can infinite-loop in BlockPlacementPolicyDefault#chooseRandom(). Contributed by Kihwal Lee. Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/43539b5f Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/43539b5f Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/43539b5f Branch: refs/heads/HDFS-8966 Commit: 43539b5ff4ac0874a8a454dc93a2a782b0e0ea8f Parents: ce31b22 Author: Kihwal Lee Authored: Fri Oct 30 09:27:21 2015 -0500 Committer: Kihwal Lee Committed: Fri Oct 30 09:29:13 2015 -0500 ---------------------------------------------------------------------- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 3 +++ .../blockmanagement/BlockPlacementPolicyDefault.java | 12 ++++++++++++ 2 files changed, 15 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hadoop/blob/43539b5f/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt index 38b9e55..f6a22a1 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt +++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt @@ -2201,6 +2201,9 @@ Release 2.8.0 - UNRELEASED HDFS-9332. Fix Precondition failures from NameNodeEditLogRoller while saving namespace. (wang) + HDFS-4937. ReplicationMonitor can infinite-loop in + BlockPlacementPolicyDefault#chooseRandom() (kihwal) + Release 2.7.2 - UNRELEASED INCOMPATIBLE CHANGES http://git-wip-us.apache.org/repos/asf/hadoop/blob/43539b5f/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java ---------------------------------------------------------------------- diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java index d9b8d60..f610574 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java +++ b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyDefault.java @@ -659,6 +659,7 @@ public class BlockPlacementPolicyDefault extends BlockPlacementPolicy { int numOfAvailableNodes = clusterMap.countNumOfAvailableNodes( scope, excludedNodes); + int refreshCounter = numOfAvailableNodes; StringBuilder builder = null; if (LOG.isDebugEnabled()) { builder = debugLoggingBuilder.get(); @@ -708,6 +709,17 @@ public class BlockPlacementPolicyDefault extends BlockPlacementPolicy { // If no candidate storage was found on this DN then set badTarget. badTarget = (storage == null); } + // Refresh the node count. If the live node count became smaller, + // but it is not reflected in this loop, it may loop forever in case + // the replicas/rack cannot be satisfied. + if (--refreshCounter == 0) { + refreshCounter = clusterMap.countNumOfAvailableNodes(scope, + excludedNodes); + // It has already gone through enough number of nodes. + if (refreshCounter <= excludedNodes.size()) { + break; + } + } } if (numOfReplicas>0) {