falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venkatesh Seetharam (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-284) Hcatalog based feed retention doesn't work when partition filter spans across multiple partition keys
Date Thu, 27 Mar 2014 00:00:18 GMT

    [ https://issues.apache.org/jira/browse/FALCON-284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13948683#comment-13948683
] 

Venkatesh Seetharam commented on FALCON-284:
--------------------------------------------

Patch looks good with minor nits:

* org.apache.falcon.lifecycle.TableStorageFeedEvictorIT#validateInstancePaths - unnecessary
return
* org.apache.falcon.lifecycle.TableStorageFeedEvictorIT#addMultiColDatedPartitions - Use StringBuilder
instead of StringBuffer
* org.apache.falcon.retention.FeedEvictor#createFilter - remove unused variable storage
* org.apache.falcon.retention.FeedEvictor#dropPartitions - redundant null init in List<CatalogPartition>
catalogPartitions = null; can be a ternary as well
* org.apache.falcon.retention.FeedEvictor#dropPartitionInstances - redundant check of isTableExternal
post OR  - if (!isTableExternal || (isTableExternal && deleted))  can be if (!isTableExternal
|| deleted)

Also there are 14 checkstyle violations:

{code}
[INFO] Starting audit...
/Users/seetharam/Hortonworks/dev/apache-workspace/falcon/incubator-falcon/webapp/src/test/java/org/apache/falcon/lifecycle/TableStorageFeedEvictorIT.java:79:
Line is longer than 120 characters.
/Users/seetharam/Hortonworks/dev/apache-workspace/falcon/incubator-falcon/webapp/src/test/java/org/apache/falcon/lifecycle/TableStorageFeedEvictorIT.java:243:
method def throws at indentation level 12 not at correct indentation, 8
/Users/seetharam/Hortonworks/dev/apache-workspace/falcon/incubator-falcon/webapp/src/test/java/org/apache/falcon/lifecycle/TableStorageFeedEvictorIT.java:289:
if child at indentation level 14 not at correct indentation, 16
/Users/seetharam/Hortonworks/dev/apache-workspace/falcon/incubator-falcon/webapp/src/test/java/org/apache/falcon/lifecycle/TableStorageFeedEvictorIT.java:289:
method call child at indentation level 14 not at correct indentation, 16
/Users/seetharam/Hortonworks/dev/apache-workspace/falcon/incubator-falcon/webapp/src/test/java/org/apache/falcon/lifecycle/TableStorageFeedEvictorIT.java:292:
finally child at indentation level 10 not at correct indentation, 12
/Users/seetharam/Hortonworks/dev/apache-workspace/falcon/incubator-falcon/webapp/src/test/java/org/apache/falcon/lifecycle/TableStorageFeedEvictorIT.java:292:
method call child at indentation level 10 not at correct indentation, 12
/Users/seetharam/Hortonworks/dev/apache-workspace/falcon/incubator-falcon/webapp/src/test/java/org/apache/falcon/lifecycle/TableStorageFeedEvictorIT.java:293:
finally child at indentation level 10 not at correct indentation, 12
/Users/seetharam/Hortonworks/dev/apache-workspace/falcon/incubator-falcon/webapp/src/test/java/org/apache/falcon/lifecycle/TableStorageFeedEvictorIT.java:293:
method call child at indentation level 10 not at correct indentation, 12
/Users/seetharam/Hortonworks/dev/apache-workspace/falcon/incubator-falcon/webapp/src/test/java/org/apache/falcon/lifecycle/TableStorageFeedEvictorIT.java:424:
method def throws at indentation level 12 not at correct indentation, 8
/Users/seetharam/Hortonworks/dev/apache-workspace/falcon/incubator-falcon/webapp/src/test/java/org/apache/falcon/lifecycle/TableStorageFeedEvictorIT.java:448:
Line is longer than 120 characters.
/Users/seetharam/Hortonworks/dev/apache-workspace/falcon/incubator-falcon/webapp/src/test/java/org/apache/falcon/lifecycle/TableStorageFeedEvictorIT.java:451:
Line is longer than 120 characters.
/Users/seetharam/Hortonworks/dev/apache-workspace/falcon/incubator-falcon/webapp/src/test/java/org/apache/falcon/lifecycle/TableStorageFeedEvictorIT.java:492:
method call child at indentation level 16 not at correct indentation, 20
/Users/seetharam/Hortonworks/dev/apache-workspace/falcon/incubator-falcon/webapp/src/test/java/org/apache/falcon/lifecycle/TableStorageFeedEvictorIT.java:493:
method call child at indentation level 16 not at correct indentation, 20
/Users/seetharam/Hortonworks/dev/apache-workspace/falcon/incubator-falcon/webapp/src/test/java/org/apache/falcon/lifecycle/TableStorageFeedEvictorIT.java:530:73:
'+' should be on a new line.
Audit done.
{code}

Please revise the patch and run the following before uploading the patch. 
{code}mvn clean verify -Phadoop-2,test-patch{code}

> Hcatalog based feed retention doesn't work when partition filter spans across multiple
partition keys
> -----------------------------------------------------------------------------------------------------
>
>                 Key: FALCON-284
>                 URL: https://issues.apache.org/jira/browse/FALCON-284
>             Project: Falcon
>          Issue Type: Bug
>    Affects Versions: 0.5
>            Reporter: Satish Mittal
>            Assignee: Satish Mittal
>         Attachments: FALCON-284.patch.1.txt, FALCON-284.patch.txt
>
>
> When an HCatalog based feed is scheduled in falcon, retention only looks at the first
partition key that satisfies either of date pattern: yyyy | MM | dd | HH | mm. As a result,
it calculates a partition filter that contains only one of these patterns. However if HCatalog
table is defined in such a way that date spans across multiple partition keys (year/month/day/hour/minute),
then feed retention doesn't delete any partitions that are granular than first level (year).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message