hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6186) Pause deletion of blocks when the namenode starts up
Date Fri, 16 May 2014 10:48:57 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13999000#comment-13999000
] 

Hudson commented on HDFS-6186:
------------------------------

SUCCESS: Integrated in Hadoop-trunk-Commit #5605 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5605/])
HDFS-6186. Pause deletion of blocks when the namenode starts up. Contributed by Jing Zhao.
(jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1594314)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/InvalidateBlocks.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingInvalidateBlock.java


> Pause deletion of blocks when the namenode starts up
> ----------------------------------------------------
>
>                 Key: HDFS-6186
>                 URL: https://issues.apache.org/jira/browse/HDFS-6186
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>            Reporter: Suresh Srinivas
>            Assignee: Jing Zhao
>             Fix For: 2.5.0
>
>         Attachments: HDFS-6186.000.patch, HDFS-6186.002.patch, HDFS-6186.003.patch
>
>
> HDFS namenode can delete blocks very quickly, given the deletion happens as a parallel
operation spread across many datanodes. One of the frequent anxieties I see is that a lot
of data can be deleted very quickly, when a cluster is brought up, especially when one of
the storage directories has failed and namenode metadata was copied from another storage.
Copying wrong metadata would results in some of the newer files (if old metadata was copied)
being deleted along with their blocks. 
> HDFS-5986 now captures the number of pending deletion block on namenode webUI and JMX.
I propose pausing deletion of blocks for a configured period of time (default 1 hour?) after
namenode comes out of safemode. This will give enough time for the administrator to notice
large number of pending deletion blocks and take corrective action.
> Thoughts?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message