Return-Path: X-Original-To: apmail-hadoop-hdfs-commits-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CE3E11147E for ; Thu, 3 Jul 2014 20:03:32 +0000 (UTC) Received: (qmail 63722 invoked by uid 500); 3 Jul 2014 20:03:31 -0000 Delivered-To: apmail-hadoop-hdfs-commits-archive@hadoop.apache.org Received: (qmail 63669 invoked by uid 500); 3 Jul 2014 20:03:31 -0000 Mailing-List: contact hdfs-commits-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-dev@hadoop.apache.org Delivered-To: mailing list hdfs-commits@hadoop.apache.org Received: (qmail 63657 invoked by uid 99); 3 Jul 2014 20:03:31 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jul 2014 20:03:31 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=5.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 03 Jul 2014 20:03:29 +0000 Received: from eris.apache.org (localhost [127.0.0.1]) by eris.apache.org (Postfix) with ESMTP id D52212388AAA; Thu, 3 Jul 2014 20:03:09 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1607734 - in /hadoop/common/branches/branch-2.5/hadoop-hdfs-project: ./ hadoop-hdfs/ hadoop-hdfs/src/main/java/ hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/ hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/ hado... Date: Thu, 03 Jul 2014 20:03:09 -0000 To: hdfs-commits@hadoop.apache.org From: wang@apache.org X-Mailer: svnmailer-1.0.9 Message-Id: <20140703200309.D52212388AAA@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: wang Date: Thu Jul 3 20:03:08 2014 New Revision: 1607734 URL: http://svn.apache.org/r1607734 Log: HDFS-6493. Change dfs.namenode.startup.delay.block.deletion to second instead of millisecond. Contributed by Juan Yu. Modified: hadoop/common/branches/branch-2.5/hadoop-hdfs-project/ (props changed) hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/ (props changed) hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/ (props changed) hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingInvalidateBlock.java Propchange: hadoop/common/branches/branch-2.5/hadoop-hdfs-project/ ------------------------------------------------------------------------------ Merged /hadoop/common/trunk/hadoop-hdfs-project:r1607732 Propchange: hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/ ------------------------------------------------------------------------------ Merged /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs:r1607732 Modified: hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt?rev=1607734&r1=1607733&r2=1607734&view=diff ============================================================================== --- hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt (original) +++ hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Thu Jul 3 20:03:08 2014 @@ -242,6 +242,9 @@ Release 2.5.0 - UNRELEASED HDFS-6620. Snapshot docs should specify about preserve options with cp command (Stephen Chu via umamahesh) + HDFS-6493. Change dfs.namenode.startup.delay.block.deletion to second + instead of millisecond. (Juan Yu via wang) + OPTIMIZATIONS HDFS-6214. Webhdfs has poor throughput for files >2GB (daryn) Propchange: hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/ ------------------------------------------------------------------------------ Merged /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java:r1607732 Modified: hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java?rev=1607734&r1=1607733&r2=1607734&view=diff ============================================================================== --- hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java (original) +++ hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java Thu Jul 3 20:03:08 2014 @@ -252,8 +252,8 @@ public class DFSConfigKeys extends Commo public static final long DFS_NAMENODE_PATH_BASED_CACHE_REFRESH_INTERVAL_MS_DEFAULT = 30000L; /** Pending period of block deletion since NameNode startup */ - public static final String DFS_NAMENODE_STARTUP_DELAY_BLOCK_DELETION_MS_KEY = "dfs.namenode.startup.delay.block.deletion.ms"; - public static final long DFS_NAMENODE_STARTUP_DELAY_BLOCK_DELETION_MS_DEFAULT = 0L; + public static final String DFS_NAMENODE_STARTUP_DELAY_BLOCK_DELETION_SEC_KEY = "dfs.namenode.startup.delay.block.deletion.sec"; + public static final long DFS_NAMENODE_STARTUP_DELAY_BLOCK_DELETION_SEC_DEFAULT = 0L; // Whether to enable datanode's stale state detection and usage for reads public static final String DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_READ_KEY = "dfs.namenode.avoid.read.stale.datanode"; Modified: hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java?rev=1607734&r1=1607733&r2=1607734&view=diff ============================================================================== --- hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java (original) +++ hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java Thu Jul 3 20:03:08 2014 @@ -263,8 +263,8 @@ public class BlockManager { heartbeatManager = datanodeManager.getHeartbeatManager(); final long pendingPeriod = conf.getLong( - DFSConfigKeys.DFS_NAMENODE_STARTUP_DELAY_BLOCK_DELETION_MS_KEY, - DFSConfigKeys.DFS_NAMENODE_STARTUP_DELAY_BLOCK_DELETION_MS_DEFAULT); + DFSConfigKeys.DFS_NAMENODE_STARTUP_DELAY_BLOCK_DELETION_SEC_KEY, + DFSConfigKeys.DFS_NAMENODE_STARTUP_DELAY_BLOCK_DELETION_SEC_DEFAULT) * 1000L; invalidateBlocks = new InvalidateBlocks( datanodeManager.blockInvalidateLimit, pendingPeriod); Modified: hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java?rev=1607734&r1=1607733&r2=1607734&view=diff ============================================================================== --- hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java (original) +++ hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java Thu Jul 3 20:03:08 2014 @@ -34,6 +34,7 @@ import org.apache.hadoop.hdfs.protocol.D import org.apache.hadoop.hdfs.server.namenode.NameNode; import org.apache.hadoop.hdfs.util.LightWeightHashSet; import org.apache.hadoop.util.Time; +import org.apache.hadoop.hdfs.DFSUtil; import com.google.common.annotations.VisibleForTesting; @@ -67,8 +68,8 @@ class InvalidateBlocks { } private void printBlockDeletionTime(final Log log) { - log.info(DFSConfigKeys.DFS_NAMENODE_STARTUP_DELAY_BLOCK_DELETION_MS_KEY - + " is set to " + pendingPeriodInMs + " ms."); + log.info(DFSConfigKeys.DFS_NAMENODE_STARTUP_DELAY_BLOCK_DELETION_SEC_KEY + + " is set to " + DFSUtil.durationToString(pendingPeriodInMs)); SimpleDateFormat sdf = new SimpleDateFormat("yyyy MMM dd HH:mm:ss"); Calendar calendar = new GregorianCalendar(); calendar.add(Calendar.SECOND, (int) (this.pendingPeriodInMs / 1000)); Modified: hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml?rev=1607734&r1=1607733&r2=1607734&view=diff ============================================================================== --- hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml (original) +++ hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml Thu Jul 3 20:03:08 2014 @@ -1996,4 +1996,16 @@ + + dfs.namenode.startup.delay.block.deletion.sec + 0 + The delay in seconds at which we will pause the blocks deletion + after Namenode startup. By default it's disabled. + In the case a directory has large number of directories and files are + deleted, suggested delay is one hour to give the administrator enough time + to notice large number of pending deletion blocks and take corrective + action. + + + Modified: hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingInvalidateBlock.java URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingInvalidateBlock.java?rev=1607734&r1=1607733&r2=1607734&view=diff ============================================================================== --- hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingInvalidateBlock.java (original) +++ hadoop/common/branches/branch-2.5/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingInvalidateBlock.java Thu Jul 3 20:03:08 2014 @@ -55,7 +55,7 @@ public class TestPendingInvalidateBlock conf = new Configuration(); conf.setLong(DFSConfigKeys.DFS_BLOCK_SIZE_KEY, BLOCKSIZE); // block deletion pending period - conf.setLong(DFSConfigKeys.DFS_NAMENODE_STARTUP_DELAY_BLOCK_DELETION_MS_KEY, 1000 * 5); + conf.setLong(DFSConfigKeys.DFS_NAMENODE_STARTUP_DELAY_BLOCK_DELETION_SEC_KEY, 5L); // set the block report interval to 2s conf.setLong(DFSConfigKeys.DFS_BLOCKREPORT_INTERVAL_MSEC_KEY, 2000); conf.setLong(DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_KEY, 1);