hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Suresh Srinivas (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-173) Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes
Date Fri, 14 Aug 2009 00:58:14 GMT

    [ https://issues.apache.org/jira/browse/HDFS-173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743042#action_12743042
] 

Suresh Srinivas commented on HDFS-173:
--------------------------------------

Here is the approach I am planning to take:
Currently all the files and subdirectories is recursively deleted holding FSNameSystem lock.
Deletion of a directory with large number of files could take significant time. This results
in other operations having to wait until the completion of delete to perform any namenode
operation. 

The proposed change is to perform deletion incrementally. First the target directory to be
deleted will be removed from the directory tree, to prevent further changes in that directory.
Then the files and directories under it are deleted 1000 at a time, holding the FSNameSystem
lock. Between two deletes, relinquish FSNameSystem lock, to allow namenode operations for
other clients. There is no sleep between deletes, as it could add up to long deletion time,
when deleting a large directory. Client that issues delete would block until the deletion
is complete.

> Recursively deleting a directory with millions of files makes NameNode unresponsive for
other commands until the deletion completes
> -----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-173
>                 URL: https://issues.apache.org/jira/browse/HDFS-173
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>
> Delete a directory with millions of files. This could take several minutes (observed
12 mins for 9 million files). While the operation is in progress FSNamesystem lock is held
and the requests from clients are not handled until deletion completes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message