falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peeyush Bishnoi" <bpeey...@yahoo.co.in>
Subject Re: Review Request 38450: FALCON-965 Open up Feed lifecycle for user extension
Date Fri, 18 Sep 2015 10:35:02 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38450/#review99507
-----------------------------------------------------------



client/src/main/resources/feed-0.1.xsd (line 120)
<https://reviews.apache.org/r/38450/#comment156404>

    Can you add details about lifecycle in xsd doc section.



client/src/main/resources/feed-0.1.xsd (line 150)
<https://reviews.apache.org/r/38450/#comment156398>

    As we are defining retention through lifecycle, do we require older retention specifically.
Is this required for backward compatibility, please confirm.



common/src/main/java/org/apache/falcon/entity/FeedHelper.java (line 401)
<https://reviews.apache.org/r/38450/#comment156403>

    "limit" is the key attribute for retention. It does not seem good to keep limit in key-value
property section. It should be element in retention-stage.



common/src/main/java/org/apache/falcon/entity/FeedHelper.java (line 428)
<https://reviews.apache.org/r/38450/#comment156411>

    Returning DefaultPolicyName does not make sense. If user want to use retention stage,
they must specify the required policy.



common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java (line 126)
<https://reviews.apache.org/r/38450/#comment156406>

    What is the requirement to define two type of retention in entity global and cluster.
From user experience perspective, how this will help the user. Can we think to make it as
one like currently.



common/src/main/java/org/apache/falcon/lifecycle/retention/NominalTimeBasedDelete.java (line
60)
<https://reviews.apache.org/r/38450/#comment156412>

    "limit" is the key attribute for retention. It does not seem good to keep limit in key-value
property section. It should be element in retention-stage.



lifecycle/pom.xml (line 117)
<https://reviews.apache.org/r/38450/#comment156407>

    Can we use latest Oozie coordinator xsd version.



lifecycle/pom.xml (line 136)
<https://reviews.apache.org/r/38450/#comment156408>

    Can we use latest Oozie workflow xsd version.



lifecycle/pom.xml (line 154)
<https://reviews.apache.org/r/38450/#comment156409>

    can we use latest hive action xsd version.



lifecycle/pom.xml (line 173)
<https://reviews.apache.org/r/38450/#comment156410>

    Can we use latest bundle xsd version.



lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/retention/NominalTimeBasedWorkflowBuilder.java
(line 119)
<https://reviews.apache.org/r/38450/#comment156414>

    Can we move this function to class HiveUtil.



lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/utils/OozieBuilderUtils.java
(line 294)
<https://reviews.apache.org/r/38450/#comment156427>

    For each inner iteration, you are setting local List variable to null and adding jar files.
With this all jar files will not be added from multiple actions. Can you check this.



lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/utils/OozieBuilderUtils.java
(line 301)
<https://reviews.apache.org/r/38450/#comment156425>

    What about support for other action like FS,SSH,Sqoop .



lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/utils/OozieBuilderUtils.java
(line 359)
<https://reviews.apache.org/r/38450/#comment156415>

    Can we move this function to class HiveUtil.



lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/utils/OozieBuilderUtils.java
(line 393)
<https://reviews.apache.org/r/38450/#comment156416>

    Can we move this function to class HiveUtil.



lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/utils/OozieBuilderUtils.java
(line 434)
<https://reviews.apache.org/r/38450/#comment156417>

    Can we move this function to class HiveUtil.



src/conf/startup.properties (line 55)
<https://reviews.apache.org/r/38450/#comment156420>

    Can we make the lifecycle policy to be defined like: *.falcon.feed.lifecycle.policies=org.apache.falcon.lifecycle.retention.NominalTimeBasedDelete#NominalTimeBasedDelete


- Peeyush Bishnoi


On Sept. 17, 2015, 7:52 a.m., Ajay Yadava wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38450/
> -----------------------------------------------------------
> 
> (Updated Sept. 17, 2015, 7:52 a.m.)
> 
> 
> Review request for Falcon.
> 
> 
> Bugs: FALCON-965
>     https://issues.apache.org/jira/browse/FALCON-965
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> For details on feed lifecycle feature and motivation behind it, please refer FALCON-965.
This is the base framework for lifecycle with a feature parity of retention stage as a reference
implementation.
> 
> 
> Diffs
> -----
> 
>   client/src/main/resources/feed-0.1.xsd 4ff8baa 
>   common/src/main/java/org/apache/falcon/entity/FeedHelper.java 572923b 
>   common/src/main/java/org/apache/falcon/entity/parser/CrossEntityValidations.java 2696552

>   common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java f22a343

>   common/src/main/java/org/apache/falcon/lifecycle/AbstractPolicyBuilderFactory.java
PRE-CREATION 
>   common/src/main/java/org/apache/falcon/lifecycle/FeedLifecycleStage.java PRE-CREATION

>   common/src/main/java/org/apache/falcon/lifecycle/LifecyclePolicy.java PRE-CREATION

>   common/src/main/java/org/apache/falcon/lifecycle/PolicyBuilder.java PRE-CREATION 
>   common/src/main/java/org/apache/falcon/lifecycle/retention/NominalTimeBasedDelete.java
PRE-CREATION 
>   common/src/main/java/org/apache/falcon/service/LifecyclePolicyStore.java PRE-CREATION

>   common/src/main/java/org/apache/falcon/workflow/WorkflowEngineFactory.java 756c6b8

>   common/src/main/resources/startup.properties 39a412d 
>   common/src/test/java/org/apache/falcon/entity/AbstractTestBase.java 6179855 
>   common/src/test/java/org/apache/falcon/entity/FeedHelperTest.java c70cfcc 
>   common/src/test/java/org/apache/falcon/entity/parser/FeedEntityParserTest.java 754fb06

>   common/src/test/resources/config/feed/feed-0.3.xml PRE-CREATION 
>   common/src/test/resources/config/feed/feed-0.4.xml PRE-CREATION 
>   lifecycle/pom.xml PRE-CREATION 
>   lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/OoziePolicyBuilderFactory.java
PRE-CREATION 
>   lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/retention/NominalTimeBasedCoordinatorBuilder.java
PRE-CREATION 
>   lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/retention/NominalTimeBasedDeleteBuilder.java
PRE-CREATION 
>   lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/retention/NominalTimeBasedWorkflowBuilder.java
PRE-CREATION 
>   lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/utils/OozieBuilderUtils.java
PRE-CREATION 
>   lifecycle/src/main/resources/action/feed/eviction-action.xml PRE-CREATION 
>   lifecycle/src/main/resources/binding/jaxb-binding.xjb PRE-CREATION 
>   oozie/pom.xml 157edf9 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedBundleBuilder.java b819dee 
>   oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java
7d0174a 
>   oozie/src/test/resources/feed/fs-retention-feed.xml PRE-CREATION 
>   oozie/src/test/resources/feed/fs-retention-lifecycle-feed.xml PRE-CREATION 
>   pom.xml 646de69 
>   src/conf/startup.properties 305ac36 
> 
> Diff: https://reviews.apache.org/r/38450/diff/
> 
> 
> Testing
> -------
> 
> Unit tests have been added for all the methods.
> I have compared the new artifacts(workflow, bundle, config-default and coordinator xmls)
with the old ones.
> 
> 
> Thanks,
> 
> Ajay Yadava
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message