hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15575) Add a way for an FS instance to say "really, no trash interval at all"
Date Mon, 02 Jul 2018 11:47:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16529752#comment-16529752

Steve Loughran commented on HADOOP-15575:

You could do all of this with a custom trash policy; add an S3A aware one which asked the
FS bucket conf, delegating to the default trash policy for all other stores.
You'd want to override {{Trash.getEmptier()}} to return a new Emptier...this would be instantiated
in the NN of any (transient) HDFS cluster, so still need to do the normal cleanup for HDFS.
As it never gets pointed at the object store (presumably), it's not doing any work there..and
even if it was, assuming the ~/.Trash was never filled up, it'd not have anything to delete.

> Add a way for an FS instance to say "really, no trash interval at all"
> ----------------------------------------------------------------------
>                 Key: HADOOP-15575
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15575
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs, fs/s3
>    Affects Versions: 3.1.0
>            Reporter: Steve Loughran
>            Priority: Major
> Object stores like S3 often offer [object versioning|https://docs.aws.amazon.com/AmazonS3/latest/dev/ObjectVersioning.html]
as a way to recover from accidental deletions. There is no need to use the Fs Shell trash
> Proposed: a way for an FS instance to tell the shell to not use trash, even if the client
has configured it. The current {{getServerDefaults().getTrashInterval()}} returns a value
from the server, but it *only* overrides the ""fs.trash.interval" if it !=0. So if a server
says "don't use trash" it gets ignored.
> If a special value (-1) is returned (or we use a new field?), FS instances can return
to the shell saying "no need for trash". Then you can turn it on for a bucket by bucket basis;
any store with the feature enabled can do the same thing.
> Alternative option: A special S3A-aware trash policy which does a getFileSystem.getConf.getBoolean("fs.s3a.trash.skip")
& skips trash if set. This will all you to disable it on a bucket-by-bucket basis. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message