hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Raghu Angadi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-173) Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes
Date Fri, 21 Aug 2009 01:30:14 GMT

    [ https://issues.apache.org/jira/browse/HDFS-173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745764#action_12745764
] 

Raghu Angadi commented on HDFS-173:
-----------------------------------

The above looks better.. simpler to show correctness.

The key observation is that logging of each block deletion is the main culprit (80 - 90% of
the time spent). The above method moves block deletion to out side the main deletion. 

With the patch, when some one deletes directory tree with 10M it might still lock NN on the
order of a minute.

Another approach could be to emulate a clients recursive deletion : This would not involve
any semantic changes to NN internals.
It would be something like :

{code}
delete(dir) {
    lastDeleted = "";
    while ( lastDeleted != dir) {
          subTree = findSubTree(dir, max_children = 1000); //under lock
          delete(subTree) // under lock.
            lastDeleted = subTree;
   }
} {code}

This essentially breaks one deletion in to many deletions for large trees. It also implies
there would be many deletion entries in edit log, which is not an issue.

findSubTree() would not iterate through entire tree each time. It stops depth-first traversal
at an inner node if number  of files (or blocks) under it is larger than given limit.



> Recursively deleting a directory with millions of files makes NameNode unresponsive for
other commands until the deletion completes
> -----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-173
>                 URL: https://issues.apache.org/jira/browse/HDFS-173
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>         Attachments: HDFS-173.patch
>
>
> Delete a directory with millions of files. This could take several minutes (observed
12 mins for 9 million files). While the operation is in progress FSNamesystem lock is held
and the requests from clients are not handled until deletion completes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message