hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15880) WASB doesn't honor fs.trash.interval and this fails to auto purge trash folder
Date Thu, 25 Oct 2018 16:30:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16663973#comment-16663973

Steve Loughran commented on HADOOP-15880:

no, they don't.

this is something which could be covered in documentation.

For S3a: better to use Versioning than trash, as trash uses rename, and of course rename takes

I don't see wasb supporting that; you'll have to do it by hand

> WASB doesn't honor fs.trash.interval and this fails to auto purge trash folder
> ------------------------------------------------------------------------------
>                 Key: HADOOP-15880
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15880
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: documentation, fs/azure
>    Affects Versions: 2.7.3
>         Environment: Any HDInsigth cluster pointing to WASB. 
>            Reporter: Sunil Kumar Chakrapani
>            Priority: Major
>              Labels: WASB
> when "fs.trash.interval" is set to a value,  trash for the local hdfs got cleared where
as the trash folder on WASB doesn't get deleted and the files get piled up on WASB store..
> WASB doesn't pick up  fs.trash.interval value and this fails to auto purge trash folder
on WASB store.
> *Issue : WASB doesn't honor fs.trash.interval and this fails to auto purge trash folder*
> *Steps to reproduce Scenario:*
> *Delete any file stored on HDFS*
> hdfs dfs -D "fs.default.name=hdfs://mycluster/" -rm /hivestore.txt
> 18/10/23 06:18:05 INFO fs.TrashPolicyDefault: Moved: 'hdfs://mycluster/hivestore.txt'
to trash at: hdfs://mycluster/user/sshuser/.Trash/Current/hivestore.txt
> *When deleted the file is moved to trash folder* 
> hdfs dfs -rm wasb:///hivestore.txt
> 18/10/23 06:19:13 INFO fs.TrashPolicyDefault: Moved: 'wasb://kcspark-2018-10-18t17-07-40-524z@kcdnsproxy.blob.core.windows.net/hivestore.txt'
to trash at: wasb://kcspark-2018-10-18t17-07-40-524z@kcdnsproxy.blob.core.windows.net/user/sshuser/.Trash/Current/hivestore.txt
> *Reduced the fs.trash.interval from 360 to 1 and restarted all related services.*
> *Trash for the local hdfs gets cleared honoring the "fs.trash.interval" value.*
> hdfs dfs -D "fs.default.name=hdfs://mycluster/" -ls hdfs://mycluster/user/sshuser/.Trash/Current/
> ls: File hdfs://mycluster/user/sshuser/.Trash/Current does not exist.
> *Where as the trash for WASB doesn't get cleared.*
> hdfs dfs -ls wasb://kcspark-2018-10-18t17-07-40-524z@kcdnsproxy.blob.core.windows.net/user/sshuser/.Trash/Current/
> Found 1 items
> -rw-r--r-- 1 sshuser supergroup 1084 2018-10-23 06:19 wasb://kcspark-2018-10-18t17-07-40-524z@kcdnsproxy.blob.core.windows.net/user/sshuser/.Trash/Current/hivestore.txt

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message