hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Srinivas (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-1143) Implement Background deletion
Date Tue, 10 Jul 2012 16:20:35 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12993260#comment-12993260
] 

Suresh Srinivas edited comment on HDFS-1143 at 7/10/12 4:19 PM:
----------------------------------------------------------------

Blocks collected from the detached inode are deleted without FSNamesystem lock. Currently
the block map in BlockManager and DatanodeDescriptor BlockInfo link list is only protected
by FSNamesystem lock. What is the synchronization for them when blocks are deleted. That is
the reason why HDFS-173 decided to delete blocks holding FSNamesystem lock for a short duration
at a time.
                
      was (Author: sureshms):
    Blocks collected from the detached inode are deleted without FSNamesystem block. Currently
the block map in BlockManager and DatanodeDescriptor BlockInfo link list is only protected
by FSNamesystem lock. What is the synchronization for them when delete blocks. That is the
reason why HDFS-173 decided to delete blocks holding FSNamesystem lock for a short duration
at a time.
                  
> Implement Background deletion
> -----------------------------
>
>                 Key: HDFS-1143
>                 URL: https://issues.apache.org/jira/browse/HDFS-1143
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.22.0
>            Reporter: Dmytro Molkov
>            Assignee: Scott Chen
>         Attachments: HDFS-1143-v2.txt, HDFS-1143.txt
>
>
> Right now if you try to delete massive number of files from the namenode it will freeze
(sometimes for minutes). Most of the time is spent going through the blocks map and invalidating
all the blocks.
> This can probably be improved by having a background GC process. The deletion will basically
just remove the inode being deleted and then give the subtree that was just deleted to the
background thread running cleanup.
> This way the namenode becomes available for the clients soon after deletion, and all
the heavy operations are done in the background.
> Thoughts?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message