Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Message-ID: <1585171663.1250211494818.JavaMail.jira@brutus>
Date: Thu, 13 Aug 2009 17:58:14 -0700 (PDT)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Subject: [jira] Commented: (HDFS-173) Recursively deleting a directory with
 millions of files makes NameNode unresponsive for other commands until the
 deletion completes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743042#action_12743042 ] 

Suresh Srinivas commented on HDFS-173:
--------------------------------------

Here is the approach I am planning to take:
Currently all the files and subdirectories is recursively deleted holding FSNameSystem lock. Deletion of a directory with large number of files could take significant time. This results in other operations having to wait until the completion of delete to perform any namenode operation. 

The proposed change is to perform deletion incrementally. First the target directory to be deleted will be removed from the directory tree, to prevent further changes in that directory. Then the files and directories under it are deleted 1000 at a time, holding the FSNameSystem lock. Between two deletes, relinquish FSNameSystem lock, to allow namenode operations for other clients. There is no sleep between deletes, as it could add up to long deletion time, when deleting a large directory. Client that issues delete would block until the deletion is complete.

> Recursively deleting a directory with millions of files makes NameNode unresponsive for other commands until the deletion completes
> -----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-173
>                 URL: https://issues.apache.org/jira/browse/HDFS-173
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>
> Delete a directory with millions of files. This could take several minutes (observed 12 mins for 9 million files). While the operation is in progress FSNamesystem lock is held and the requests from clients are not handled until deletion completes.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.