hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yi Liu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2975) Rename with overwrite flag true can make NameNode to stuck in safemode on NN (crash + restart).
Date Mon, 01 Sep 2014 08:49:21 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14117198#comment-14117198
] 

Yi Liu commented on HDFS-2975:
------------------------------

Thanks Uma for review and comment. Your consideration is reasonable.
After think more, we need to track blocks incrementally if HA enabled and in safe-mode. 
* While loading the editlog, for delete (or rename with overwrite) Op, we need to track blocks
incrementally and can call {{removeBlocksAndUpdateSafemodeTotal}}.
* While HDFS client calls delete (or rename with overwrite) Op, NN is not in safe-mode, and
we don't need to track blocks incrementally.  We can do incremental deletion of blocks directly.

So current delete is correct. And for rename with overwrite, we have called {{removeBlocksAndUpdateSafemodeTotal}}
which loading editlog. 
So the behavior in the patch is exact what we expect. 

> Rename with overwrite flag true can make NameNode to stuck in safemode on NN (crash +
restart).
> -----------------------------------------------------------------------------------------------
>
>                 Key: HDFS-2975
>                 URL: https://issues.apache.org/jira/browse/HDFS-2975
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>            Reporter: Uma Maheswara Rao G
>            Assignee: Yi Liu
>         Attachments: HDFS-2975.001.patch
>
>
> When we rename the file with overwrite flag as true, it will delete the destination file
blocks. After deleting the blocks, whenever it releases the fsNameSystem lock, NN can give
the invalidation work to corresponding DNs to delete the blocks.
> Parallaly it will sync the rename related edits to editlog file. At this step before
NN sync the edits if NN crashes, NN can stuck into safemode on restart. This is because block
already deleted from the DN as part of invalidations. But dst file still exist as rename edits
not persisted in log file and no DN will report that blocks now.
> This is similar to HDFS-2815
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message