hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Casey Brotherton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8118) Delay in checkpointing Trash can leave trash for 2 intervals before deleting
Date Fri, 10 Apr 2015 14:54:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14489732#comment-14489732

Casey Brotherton commented on HDFS-8118:

Hello Harsh, 

Thanks for the comment.

The sleeping time is based on fs.trash.checkpoint.interval.

There is code to change the checkpoint interval to not be larger than the deletion interval
in the constructor.
It could be changed to ensure that the checkpoint is no more than half the deletion interval.

One thing to consider is that currently the maximum time that you can recover a deleted file
from trash is a fs.trash.checkpoint.interval + fs.trash.interval.

( With both trash defaults set to 24 hours, deleting a file one minute after midnight, that
file gets placed into a checkpoint 24 hours later, 
and the checkpoint is deleted 24 hours later. )


> Delay in checkpointing Trash can leave trash for 2 intervals before deleting
> ----------------------------------------------------------------------------
>                 Key: HDFS-8118
>                 URL: https://issues.apache.org/jira/browse/HDFS-8118
>             Project: Hadoop HDFS
>          Issue Type: Bug
>            Reporter: Casey Brotherton
>            Priority: Trivial
> When the fs.trash.checkpoint.interval and the fs.trash.interval are set non-zero and
the same, it is possible for trash to be left for two intervals.
> The TrashPolicyDefault will use a floor and ceiling function to ensure that the Trash
will be checkpointed every "interval" of minutes.
> Each user's trash is checkpointed individually.  The time resolution of the checkpoint
timestamp is to the second.
> If the seconds switch while one user is checkpointing, then the next user's timestamp
will be later.
> This will cause the next user's checkpoint to not be deleted at the next interval.
> I have recreated this in a lab cluster 
> I also have a suggestion for a patch that I can upload later tonight after testing it

This message was sent by Atlassian JIRA

View raw message