falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Samarth Gupta (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (FALCON-321) Feed evictor deleting more stuff than it should
Date Wed, 26 Feb 2014 04:48:19 GMT

    [ https://issues.apache.org/jira/browse/FALCON-321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13912511#comment-13912511
] 

Samarth Gupta edited comment on FALCON-321 at 2/26/14 4:46 AM:
---------------------------------------------------------------

[~raghavgautam]  [~svenkat] this is intentional and a feature currently.  We dont want to
leave too many empty folder on HDFS.  This was not happening earlier and have been enabled
now. 
In fact at inmobi we added in regression to ensure that. 

Feature is open for debate ..... :) 


was (Author: samarthg):
this is intentional and a feature currently.  We dont want to leave too many empty folder
on HDFS.  This was not happening earlier and have been enabled now. 

Feature is open for debate ..... :) 

> Feed evictor deleting more stuff than it should
> -----------------------------------------------
>
>                 Key: FALCON-321
>                 URL: https://issues.apache.org/jira/browse/FALCON-321
>             Project: Falcon
>          Issue Type: Bug
>            Reporter: Raghav Kumar Gautam
>            Priority: Blocker
>              Labels: system-tests
>
> In FeedEvictor.java we have:
> {code:java}
> private void deleteParentIfEmpty(FileSystem fs, Path parent, Path feedBasePath) throws
IOException {
>         if (feedBasePath.equals(parent)) {
>             LOG.info("Not deleting feed base path:" + parent);
>         } else {
>             if (fs.getContentSummary(parent).getFileCount() == 0) {
>                 LOG.info("Parent path: " + parent + " is empty, deleting path");
>                 if (fs.delete(parent, true)) {
>                     LOG.info("Deleted empty dir: " + parent);
>                 } else {
>                     throw new IOException("Unable to delete parent path:" + parent);
>                 }
>                 deleteParentIfEmpty(fs, parent.getParent(), feedBasePath);
>             }
>         }
>     }
> {code}
> In the fs.getContentSummary(parent).getFileCount() call if the parent has no files but
has directories then we delete the parent directory. Which is incorrect.
> Here is log from falcon-regression's RetentionTest.testRetention(parameters: hours, 24,
true, daily) :
> {noformat}
> 2014-02-24 15:09:45,034 INFO [main] org.apache.falcon.retention.FeedEvictor: Applying
retention on DATA=hdfs://raghav5-falcon-5.cs1cloud.internal:8020/retention/testFolders/${YEAR}/${MONTH}/${DAY}/${HOUR}#META=hdfs://raghav5-falcon-5.cs1cloud.internal:8020/projects/ivory/clicksMetaData#STATS=hdfs://raghav5-falcon-5.cs1cloud.internal:8020/projects/ivory/clicksStats#TMP=/tmp
type: instance, Limit: hours(24), timezone: UTC, frequency: hours, storageFILESYSTEM
> 2014-02-24 15:09:45,051 INFO [main] org.apache.falcon.retention.FeedEvictor: Normalized
path : /retention/testFolders/${YEAR}/${MONTH}/${DAY}/${HOUR}
> 2014-02-24 15:09:45,123 INFO [main] org.apache.falcon.retention.FeedEvictor: Searching
for /retention/testFolders/*/*/*/*
> 2014-02-24 15:09:45,486 INFO [main] org.apache.falcon.retention.FeedEvictor: Deleted
instance :/retention/testFolders/2014/01/21/00
> 2014-02-24 15:09:45,500 INFO [main] org.apache.falcon.retention.FeedEvictor: Parent path:
/retention/testFolders/2014/01/21 is empty, deleting path
> 2014-02-24 15:09:45,509 INFO [main] org.apache.falcon.retention.FeedEvictor: Deleted
empty dir: /retention/testFolders/2014/01/21
> 2014-02-24 15:09:45,511 INFO [main] org.apache.falcon.retention.FeedEvictor: Parent path:
/retention/testFolders/2014/01 is empty, deleting path
> 2014-02-24 15:09:45,517 INFO [main] org.apache.falcon.retention.FeedEvictor: Deleted
empty dir: /retention/testFolders/2014/01
> 2014-02-24 15:09:45,518 INFO [main] org.apache.falcon.retention.FeedEvictor: Parent path:
/retention/testFolders/2014 is empty, deleting path
> 2014-02-24 15:09:45,525 INFO [main] org.apache.falcon.retention.FeedEvictor: Deleted
empty dir: /retention/testFolders/2014
> 2014-02-24 15:09:45,526 INFO [main] org.apache.falcon.retention.FeedEvictor: Not deleting
feed base path:/retention/testFolders
> {noformat}
> Stacktrace:
> {noformat}
> Failing Oozie Launcher, Main class [org.apache.falcon.retention.FeedEvictor], main()
threw exception, Unable to delete instance: /retention/testFolders/2014/01/21/03
> java.io.IOException: Unable to delete instance: /retention/testFolders/2014/01/21/03
> 	at org.apache.falcon.retention.FeedEvictor.deleteInstance(FeedEvictor.java:321)
> 	at org.apache.falcon.retention.FeedEvictor.fileSystemEvictor(FeedEvictor.java:174)
> 	at org.apache.falcon.retention.FeedEvictor.evictFS(FeedEvictor.java:149)
> 	at org.apache.falcon.retention.FeedEvictor.evict(FeedEvictor.java:139)
> 	at org.apache.falcon.retention.FeedEvictor.run(FeedEvictor.java:121)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> 	at org.apache.falcon.retention.FeedEvictor.main(FeedEvictor.java:93)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message