falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Balu Vellanki (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FALCON-1580) Feed Retention corner case : does not clean up feed instances
Date Mon, 02 Nov 2015 23:50:27 GMT
Balu Vellanki created FALCON-1580:
-------------------------------------

             Summary: Feed Retention corner case : does not clean up feed instances 
                 Key: FALCON-1580
                 URL: https://issues.apache.org/jira/browse/FALCON-1580
             Project: Falcon
          Issue Type: Bug
          Components: feed
    Affects Versions: 0.8
            Reporter: Balu Vellanki


If a Feed entity is defined as follows,
{code}
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<feed name="rawEmailFeed" description="Raw customer email feed" xmlns="uri:falcon:feed:0.1">
    <tags>externalSystem=USWestEmailServers</tags>
    <groups>churnAnalysisDataPipeline</groups>
    <frequency>hours(1)</frequency>
    <timezone>UTC</timezone>
    <late-arrival cut-off="hours(1)"/>
    <clusters>
        <cluster name="primaryCluster" type="source">
            <validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
            <retention limit="days(30)" action="delete"/>
        </cluster>
    </clusters>
    <locations>
        <location type="data" path="/user/ambari-qa/falcon/demo/primary/input/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
        <location type="stats" path="/"/>
        <location type="meta" path="/"/>
    </locations>
    <ACL owner="ambari-qa" group="users" permission="0x755"/>
    <schema location="/none" provider="/none"/>
</feed>
{code}

The feed is valid for one day, hence the feed bundle is valid for a day. The retention coordinator
completes after a day with status SUCCEEDED. During this one day, there are no feed instances
older than 30 days. So no feed instances are deleted.

After 30 days, the user expects feed instances to be deleted. But there is no retention coordinator
running to delete the feed instances. The solution is to set the "endtime" for retention coordinator
 to "feed validityEndTime + retentionAgeLimit".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message