falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pragya Mittal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1526) Update happens twice in case of feed containing lifecycle
Date Tue, 13 Oct 2015 14:31:05 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955030#comment-14955030
] 

Pragya Mittal commented on FALCON-1526:
---------------------------------------

According to recent observations , this bug is not specific to lifecycle. Any feed/process
containing multiple clusters (lifecycle may or may not be defined) when updated will lead
to bundle duplication on all the clusters defined in that feed/process.

Example feed definition :
{noformat}
<?xml version="1.0" encoding="UTF-8"?><feed xmlns="uri:falcon:feed:0.1" name="PrismFeedUpdateTest-agregated-logs16-bba5ed74"
description="clicks log">
    <frequency>hours(1)</frequency>
    <timezone>UTC</timezone>
    <late-arrival cut-off="hours(6)"/>
    <clusters>
        <cluster name="PrismFeedUpdateTest-corp-c228e1ca" type="source">
            <validity start="2015-10-13T13:24Z" end="2099-01-01T00:00Z"/>
            <retention limit="hours(10)" action="delete"/>
        </cluster>
        <cluster name="PrismFeedUpdateTest-corp-6d3bec13" type="target">
            <validity start="2015-10-13T13:24Z" end="2099-01-01T00:00Z"/>
            <retention limit="hours(10)" action="delete"/>
        </cluster>
    </clusters>
    <locations>
        <location type="data" path="/tmp/falcon-regression/PrismFeedUpdateTest/output/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
        <location type="stats" path="/projects/falcon/clicksStats"/>
        <location type="meta" path="/projects/falcon/clicksMetaData"/>
    </locations>
    <ACL owner="pragya" group="dataqa" permission="*"/>
    <schema location="/schema/clicks" provider="protobuf"/>
    <properties>
        <property name="field1" value="value1"/>
        <property name="field2" value="value2"/>
        <property name="queueName" value="myQueue"/>
    </properties>
</feed>
{noformat}


Updated query :
{noformat}
2015-10-13 19:43:36,473 INFO  - [TestNGInvoker-updateFeedQueueDependentMultipleProcessOneProcessZeroInput():]
~ Request Url: http://192.168.138.236:16000/api/entities/update/feed/PrismFeedUpdateTest-agregated-logs16-bba5ed74?user.name=pragya
(BaseRequest:174)
{noformat}

Update response :
{noformat}
The web service response is:
<?xml version="1.0" encoding="UTF-8"?><result>
  <status>SUCCEEDED</status>
  <message>prism/Updated successfully
falcon/update/ua1/Updated successfully(feed) PrismFeedUpdateTest-agregated-logs16-bba5ed74/Effective
Time: 2015-10-13T14:24Z. Old bundle id: 0000009-151013153106797-oozie-oozi-B. Old coordinator
id: 0000010-151013153106797-oozie-oozi-C,0000011-151013153106797-oozie-oozi-C. New coordinator
id: 0000024-151013153106797-oozie-oozi-C,0000025-151013153106797-oozie-oozi-C
(feed) PrismFeedUpdateTest-agregated-logs16-bba5ed74/Effective Time: 2015-10-13T14:24Z. Old
bundle id: 0000005-151013065531196-oozie-oozi-B. Old coordinator id: 0000006-151013065531196-oozie-oozi-C.
New bundle id: 0000019-151013065531196-oozie-oozi-B

ua2/Updated successfully(feed) PrismFeedUpdateTest-agregated-logs16-bba5ed74/Effective Time:
2015-10-13T14:24Z. Old bundle id: 0000019-151013065531196-oozie-oozi-B. Old coordinator id:
0000020-151013065531196-oozie-oozi-C. New coordinator id: 0000024-151013065531196-oozie-oozi-C
(feed) PrismFeedUpdateTest-agregated-logs16-bba5ed74/Effective Time: 2015-10-13T14:24Z. Old
bundle id: 0000023-151013153106797-oozie-oozi-B. Old coordinator id: 0000025-151013153106797-oozie-oozi-C,0000024-151013153106797-oozie-oozi-C.
New bundle id: 0000029-151013153106797-oozie-oozi-B


</message>
{noformat}

> Update happens twice in case of feed containing lifecycle
> ---------------------------------------------------------
>
>                 Key: FALCON-1526
>                 URL: https://issues.apache.org/jira/browse/FALCON-1526
>             Project: Falcon
>          Issue Type: Bug
>          Components: prism
>    Affects Versions: 0.8
>            Reporter: Pragya Mittal
>            Assignee: Ajay Yadava
>
> If a feed containing lifecycle tag is updated then two bundles are created corresponding
to retention(source, target) and replication(target).
> Feed definition is :
> {code}
> <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
> <feed name="retention-feed1" description="Input File" xmlns="uri:falcon:feed:0.1">
>     <partitions>
>         <partition name="colo"/>
>         <partition name="eventTime"/>
>         <partition name="impressionHour"/>
>         <partition name="pricingModel"/>
>     </partitions>
>     <frequency>minutes(5)</frequency>
>     <timezone>UTC</timezone>
>     <late-arrival cut-off="minutes(2)"/>
>     <clusters>
>         <cluster name="A969761bb-af20fc5f" type="source">
>             <validity start="2015-10-09T14:00Z" end="2015-10-09T18:00Z"/>
>             <retention limit="days(1000000)" action="delete"/>
>         </cluster>
>         <cluster name="A969761bb-69e50f1d" type="target">
>             <validity start="2015-10-09T14:00Z" end="2015-10-09T15:00Z"/>
>             <retention limit="days(1000000)" action="delete"/>
>             <locations>
>                 <location type="data" path="/tmp/falcon-regression/LifecycleRetentionAndReplicationTest/target/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
>             </locations>
>         </cluster>
>     </clusters>
>     <locations>
>         <location type="data" path="/tmp/falcon-regression/LifecycleRetentionAndReplicationTest/source/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}"/>
>         <location type="stats" path="/data/regression/fetlrc/billing/stats"/>
>         <location type="meta" path="/data/regression/fetlrc/billing/metadata"/>
>     </locations>
>     <ACL owner="pragya" group="dataqa" permission="*"/>
>     <schema location="/databus/streams_local/click_rr/schema/" provider="protobuf"/>
>     <properties>
>         <property name="field1" value="value1"/>
>         <property name="field2" value="value2"/>
>     </properties>
>     <lifecycle>
>         <retention-stage>
>             <frequency>minutes(10)</frequency>
>             <queue>default</queue>
>             <priority>NORMAL</priority>
>             <properties>
>                 <property name="retention.policy.agebaseddelete.limit" value="minutes(15)"/>
>             </properties>
>         </retention-stage>
>     </lifecycle>
> </feed>
> {code}
> Update response log is :
> {noformat}
> dataqa@8RPCG32:/mnt/users/pragya/lifecycle$ fef -name retention-feed1 -update -file feed.xml
> falcon/update/ua1/Updated successfully(feed) retention-feed1/Effective Time: 2015-10-09T14:25Z.
Old bundle id: 0000053-151009121634425-oozie-oozi-B. Old coordinator id: 0000054-151009121634425-oozie-oozi-C.
New bundle id: 0000064-151009121634425-oozie-oozi-B
> (feed) retention-feed1/Effective Time: 2015-10-09T14:25Z. Old bundle id: 0000039-151009142953874-oozie-oozi-B.
Old coordinator id: 0000041-151009142953874-oozie-oozi-C,0000040-151009142953874-oozie-oozi-C.
New bundle id: 0000053-151009142953874-oozie-oozi-B
> ua2/Updated successfully(feed) retention-feed1/Effective Time: 2015-10-09T14:25Z. Old
bundle id: 0000053-151009142953874-oozie-oozi-B. Old coordinator id: 0000054-151009142953874-oozie-oozi-C,0000055-151009142953874-oozie-oozi-C.
New bundle id: 0000059-151009142953874-oozie-oozi-B
> (feed) retention-feed1/Effective Time: 2015-10-09T14:25Z. Old bundle id: 0000064-151009121634425-oozie-oozi-B.
Old coordinator id: 0000065-151009121634425-oozie-oozi-C. New coordinator id: 0000069-151009121634425-oozie-oozi-C
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message