hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Srinivas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-173) Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes
Date Thu, 20 Aug 2009 17:44:14 GMT

    [ https://issues.apache.org/jira/browse/HDFS-173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745524#action_12745524
] 

Suresh Srinivas commented on HDFS-173:
--------------------------------------

Currently following steps are taken for recursive deletion of directories:
# remove target inode from the parents. This recursively traverses the subtree to compute
quota used by the subtree to update the parent node quota.
# travers the subtree of target node again and delete inodes and collect blocks
# remove lease for the target node and the subtree
# delete blocks from the block map, corrupt map and schedule block for deletion at datanode
by adding it to {{BlockManager.recentInvalidateSets}}

While adding to {{BlockManager.recentInvalidateSets}}, in step 4 above, an info level log
is printed for every replica. In my testing, deletion time for 1 million files with a block
each, came down from 15 seconds to 3 seconds on commenting this log out.

In the previous patch there was concern that during incremental deletion, the file inodes
from the target node is still referenced in the lease and by the blocks.

Here is the proposed change:
Step 1 - FSNamesystem lock is held for longer time here
# remove target inode from the parents. At this time, two tree traversals are from the previous
implementation will be combined to one. Computing quota used by the subtree, deletion of inodes
and collecting blocks will be done in a single step. In this step, while collecting blocks,
the inode referenced by the block will be set to null.


Step 2: lock held for shorted duration for incremental block deletion
# Over multiple iteration, collected blocks will be deleted from block map, corrupt map and
added {{BlockManager.recentInvalidateSets}}
# Logging of addition to invalidate sets will be done outside the lock. Instead of printing
a log for every block replica, a single log will be printed for all the replicas of a block.


> Recursively deleting a directory with millions of files makes NameNode unresponsive for
other commands until the deletion completes
> -----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-173
>                 URL: https://issues.apache.org/jira/browse/HDFS-173
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>         Attachments: HDFS-173.patch
>
>
> Delete a directory with millions of files. This could take several minutes (observed
12 mins for 9 million files). While the operation is in progress FSNamesystem lock is held
and the requests from clients are not handled until deletion completes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message