hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6761) Improve Trash Emptier
Date Wed, 12 May 2010 21:00:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12866732#action_12866732
] 

dhruba borthakur commented on HADOOP-6761:
------------------------------------------

Ok, sounds good.

The current description says "The current default policy is to delete files from /trash that
are more than 6 hours old. In the future, this policy will be configurable through a well
defined interface." This is old information anyways and can be updated. I agree.

> Improve Trash Emptier
> ---------------------
>
>                 Key: HADOOP-6761
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6761
>             Project: Hadoop Common
>          Issue Type: Improvement
>            Reporter: Dmytro Molkov
>            Assignee: Dmytro Molkov
>         Attachments: HADOOP-6761.patch
>
>
> There are two inefficiencies in the Trash functionality right now that have caused some
problems for us.
> First if you configured your trash interval to be one day (24 hours) that means that
you store 2 days worth of data eventually. The Current and the previous timestamp that will
not be deleted until the end of the interval.
> And another problem is accumulating a lot of data in Trash before the Emptier wakes up.
If there are a couple of million files trashed and the Emptier does deletion on HDFS the NameNode
will freeze until everything is removed. (this particular problem hopefully will be addressed
with HDFS-1143).
> My proposal is to have two configuration intervals. One for deleting the trashed data
and another for checkpointing. This way for example for intervals of one day and one hour
we will only store 25 hours of data instead of 48 right now and the deletions will be happening
in smaller chunks every hour of the day instead of a huge deletion at the end of the day now.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message