falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pragya Mittal (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (FALCON-1675) Lifecycle stage is not honoured when only cluster level lifecycle is implemented.
Date Fri, 18 Dec 2015 19:41:46 GMT

     [ https://issues.apache.org/jira/browse/FALCON-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Pragya Mittal updated FALCON-1675:
----------------------------------
    Description: 
In case of feed lifecycle :
If we define lifecycle at cluster level and global level lifecycle is not defined, instead
of honouring lifecycle at cluster level it falls back to old retention level.

Feed definition is :
{noformat}
<feed name="RetentionLifecycleTest-inPath-eeb77a18" description="clicks log" xmlns="uri:falcon:feed:0.1">
    <frequency>hours(1)</frequency>
    <timezone>UTC</timezone>
    <late-arrival cut-off="hours(1)"/>
    <clusters>
        <cluster name="RetentionLifecycleTest-corp-02f1746f" type="source">
            <validity start="2015-12-18T18:48Z" end="2015-12-18T20:48Z"/>
            <retention limit="hours(10)" action="delete"/>
            <lifecycle>
                <retention-stage>
                    <frequency>hours(3)</frequency>
                    <queue>default</queue>
                    <priority>LOW</priority>
                    <properties>
                        <property name="retention.policy.agebaseddelete.limit" value="hours(4)"/>
                    </properties>
                </retention-stage>
            </lifecycle>
        </cluster>
    </clusters>
    <locations>
        <location type="data" path="/tmp/falcon-regression/RetentionLifecycleTest/testFolders/${YEAR}/${MONTH}/${DAY}/${HOUR}"/>
        <location type="stats" path="/projects/falcon/clicksStats"/>
        <location type="meta" path="/projects/falcon/clicksMetaData"/>
    </locations>
    <ACL owner="pragya.mittal" group="dataqa" permission="*"/>
    <schema location="/schema/clicks" provider="protobuf"/>
    <properties/>
</feed>
{noformat}

Job has 
retention frequency = 360min (6 hours)
priority = NORMAL
Intsnace retains last 10 instances instead of 4 (as is defined in cluster level lifecycle)

  was:
In case of feed lifecycle :
If we define lifecycle at cluster level and global level lifecycle is not defined, instead
of honouring lifecycle at cluster level it falls back to old retention level.


> Lifecycle stage is not honoured when only cluster level lifecycle is implemented.
> ---------------------------------------------------------------------------------
>
>                 Key: FALCON-1675
>                 URL: https://issues.apache.org/jira/browse/FALCON-1675
>             Project: Falcon
>          Issue Type: Bug
>          Components: feed
>    Affects Versions: 0.9
>         Environment: QA
>            Reporter: Pragya Mittal
>
> In case of feed lifecycle :
> If we define lifecycle at cluster level and global level lifecycle is not defined, instead
of honouring lifecycle at cluster level it falls back to old retention level.
> Feed definition is :
> {noformat}
> <feed name="RetentionLifecycleTest-inPath-eeb77a18" description="clicks log" xmlns="uri:falcon:feed:0.1">
>     <frequency>hours(1)</frequency>
>     <timezone>UTC</timezone>
>     <late-arrival cut-off="hours(1)"/>
>     <clusters>
>         <cluster name="RetentionLifecycleTest-corp-02f1746f" type="source">
>             <validity start="2015-12-18T18:48Z" end="2015-12-18T20:48Z"/>
>             <retention limit="hours(10)" action="delete"/>
>             <lifecycle>
>                 <retention-stage>
>                     <frequency>hours(3)</frequency>
>                     <queue>default</queue>
>                     <priority>LOW</priority>
>                     <properties>
>                         <property name="retention.policy.agebaseddelete.limit" value="hours(4)"/>
>                     </properties>
>                 </retention-stage>
>             </lifecycle>
>         </cluster>
>     </clusters>
>     <locations>
>         <location type="data" path="/tmp/falcon-regression/RetentionLifecycleTest/testFolders/${YEAR}/${MONTH}/${DAY}/${HOUR}"/>
>         <location type="stats" path="/projects/falcon/clicksStats"/>
>         <location type="meta" path="/projects/falcon/clicksMetaData"/>
>     </locations>
>     <ACL owner="pragya.mittal" group="dataqa" permission="*"/>
>     <schema location="/schema/clicks" provider="protobuf"/>
>     <properties/>
> </feed>
> {noformat}
> Job has 
> retention frequency = 360min (6 hours)
> priority = NORMAL
> Intsnace retains last 10 instances instead of 4 (as is defined in cluster level lifecycle)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message