hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sunil Kumar Chakrapani (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HADOOP-15880) WASB doesn't honor fs.trash.interval and this fails to auto purge trash folder
Date Thu, 25 Oct 2018 00:40:00 GMT
Sunil Kumar Chakrapani created HADOOP-15880:
-----------------------------------------------

             Summary: WASB doesn't honor fs.trash.interval and this fails to auto purge trash
folder
                 Key: HADOOP-15880
                 URL: https://issues.apache.org/jira/browse/HADOOP-15880
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs/azure
    Affects Versions: 2.7.3
         Environment: Any HDInsigth cluster pointing to WASB. 
            Reporter: Sunil Kumar Chakrapani


when "fs.trash.interval" is set to a value,  trash for the local hdfs got cleared where as
the trash folder on WASB doesn't get deleted and the files get piled up on WASB store..

WASB doesn't pick up  fs.trash.interval value and this fails to auto purge trash folder
on WASB store.

 

*Issue : WASB doesn't honor fs.trash.interval and this fails to auto purge trash folder*

*Steps to reproduce Scenario:*

*Delete any file stored on HDFS*

hdfs dfs -D "fs.default.name=hdfs://mycluster/" -rm /hivestore.txt
18/10/23 06:18:05 INFO fs.TrashPolicyDefault: Moved: 'hdfs://mycluster/hivestore.txt' to trash
at: hdfs://mycluster/user/sshuser/.Trash/Current/hivestore.txt

*When deleted the file is moved to trash folder* 
hdfs dfs -rm wasb:///hivestore.txt
18/10/23 06:19:13 INFO fs.TrashPolicyDefault: Moved: 'wasb://kcspark-2018-10-18t17-07-40-524z@kcdnsproxy.blob.core.windows.net/hivestore.txt'
to trash at: wasb://kcspark-2018-10-18t17-07-40-524z@kcdnsproxy.blob.core.windows.net/user/sshuser/.Trash/Current/hivestore.txt

*Reduced the fs.trash.interval from 360 to 1 and restarted all related services.*

*Trash for the local hdfs gets cleared honoring the "fs.trash.interval" value.*

hdfs dfs -D "fs.default.name=hdfs://mycluster/" -ls hdfs://mycluster/user/sshuser/.Trash/Current/
ls: File hdfs://mycluster/user/sshuser/.Trash/Current does not exist.

*Where as the trash for WASB doesn't get cleared.*

hdfs dfs -ls wasb://kcspark-2018-10-18t17-07-40-524z@kcdnsproxy.blob.core.windows.net/user/sshuser/.Trash/Current/
Found 1 items
-rw-r--r-- 1 sshuser supergroup 1084 2018-10-23 06:19 wasb://kcspark-2018-10-18t17-07-40-524z@kcdnsproxy.blob.core.windows.net/user/sshuser/.Trash/Current/hivestore.txt

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org


Mime
View raw message