falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karishma Gulati (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (FALCON-321) Feed evictor deleting more stuff than it should
Date Thu, 27 Feb 2014 10:00:31 GMT

     [ https://issues.apache.org/jira/browse/FALCON-321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Karishma Gulati updated FALCON-321:
-----------------------------------

    Attachment: oozie-logs-test-2.txt
                oozie-logs-test-1.txt
                results.txt

PFA a summary of the results, as well as the oozie logs of 2 separate retention tests.

Used the same feed definition given in the chain above.

Worked fine for us.

Please update if this doesn't answer the problem, and there's some other test case that needs
to be run. 

> Feed evictor deleting more stuff than it should
> -----------------------------------------------
>
>                 Key: FALCON-321
>                 URL: https://issues.apache.org/jira/browse/FALCON-321
>             Project: Falcon
>          Issue Type: Bug
>            Reporter: Raghav Kumar Gautam
>            Priority: Blocker
>              Labels: system-tests
>         Attachments: oozie-logs-test-1.txt, oozie-logs-test-2.txt, results.txt
>
>
> In FeedEvictor.java we have:
> {code:java}
> private void deleteParentIfEmpty(FileSystem fs, Path parent, Path feedBasePath) throws
IOException {
>         if (feedBasePath.equals(parent)) {
>             LOG.info("Not deleting feed base path:" + parent);
>         } else {
>             if (fs.getContentSummary(parent).getFileCount() == 0) {
>                 LOG.info("Parent path: " + parent + " is empty, deleting path");
>                 if (fs.delete(parent, true)) {
>                     LOG.info("Deleted empty dir: " + parent);
>                 } else {
>                     throw new IOException("Unable to delete parent path:" + parent);
>                 }
>                 deleteParentIfEmpty(fs, parent.getParent(), feedBasePath);
>             }
>         }
>     }
> {code}
> In the fs.getContentSummary(parent).getFileCount() call if the parent has no files but
has directories then we delete the parent directory. Which is incorrect.
> Here is log from falcon-regression's RetentionTest.testRetention(parameters: hours, 24,
true, daily) :
> {noformat}
> 2014-02-24 15:09:45,034 INFO [main] org.apache.falcon.retention.FeedEvictor: Applying
retention on DATA=hdfs://raghav5-falcon-5.cs1cloud.internal:8020/retention/testFolders/${YEAR}/${MONTH}/${DAY}/${HOUR}#META=hdfs://raghav5-falcon-5.cs1cloud.internal:8020/projects/ivory/clicksMetaData#STATS=hdfs://raghav5-falcon-5.cs1cloud.internal:8020/projects/ivory/clicksStats#TMP=/tmp
type: instance, Limit: hours(24), timezone: UTC, frequency: hours, storageFILESYSTEM
> 2014-02-24 15:09:45,051 INFO [main] org.apache.falcon.retention.FeedEvictor: Normalized
path : /retention/testFolders/${YEAR}/${MONTH}/${DAY}/${HOUR}
> 2014-02-24 15:09:45,123 INFO [main] org.apache.falcon.retention.FeedEvictor: Searching
for /retention/testFolders/*/*/*/*
> 2014-02-24 15:09:45,486 INFO [main] org.apache.falcon.retention.FeedEvictor: Deleted
instance :/retention/testFolders/2014/01/21/00
> 2014-02-24 15:09:45,500 INFO [main] org.apache.falcon.retention.FeedEvictor: Parent path:
/retention/testFolders/2014/01/21 is empty, deleting path
> 2014-02-24 15:09:45,509 INFO [main] org.apache.falcon.retention.FeedEvictor: Deleted
empty dir: /retention/testFolders/2014/01/21
> 2014-02-24 15:09:45,511 INFO [main] org.apache.falcon.retention.FeedEvictor: Parent path:
/retention/testFolders/2014/01 is empty, deleting path
> 2014-02-24 15:09:45,517 INFO [main] org.apache.falcon.retention.FeedEvictor: Deleted
empty dir: /retention/testFolders/2014/01
> 2014-02-24 15:09:45,518 INFO [main] org.apache.falcon.retention.FeedEvictor: Parent path:
/retention/testFolders/2014 is empty, deleting path
> 2014-02-24 15:09:45,525 INFO [main] org.apache.falcon.retention.FeedEvictor: Deleted
empty dir: /retention/testFolders/2014
> 2014-02-24 15:09:45,526 INFO [main] org.apache.falcon.retention.FeedEvictor: Not deleting
feed base path:/retention/testFolders
> {noformat}
> Stacktrace:
> {noformat}
> Failing Oozie Launcher, Main class [org.apache.falcon.retention.FeedEvictor], main()
threw exception, Unable to delete instance: /retention/testFolders/2014/01/21/03
> java.io.IOException: Unable to delete instance: /retention/testFolders/2014/01/21/03
> 	at org.apache.falcon.retention.FeedEvictor.deleteInstance(FeedEvictor.java:321)
> 	at org.apache.falcon.retention.FeedEvictor.fileSystemEvictor(FeedEvictor.java:174)
> 	at org.apache.falcon.retention.FeedEvictor.evictFS(FeedEvictor.java:149)
> 	at org.apache.falcon.retention.FeedEvictor.evict(FeedEvictor.java:139)
> 	at org.apache.falcon.retention.FeedEvictor.run(FeedEvictor.java:121)
> 	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> 	at org.apache.falcon.retention.FeedEvictor.main(FeedEvictor.java:93)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message