falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Balu Vellanki" <bvella...@hortonworks.com>
Subject Re: Review Request 40892: FALCON-1644 Retention : Some feed instances are never deleted by retention jobs.
Date Mon, 14 Dec 2015 19:18:07 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40892/
-----------------------------------------------------------

(Updated Dec. 14, 2015, 7:18 p.m.)


Review request for Falcon and Venkat Ranganathan.


Changes
-------

Thanks for pointing it out Ajay. I updated the patch and tested it end2end.


Bugs: Falcon-1644
    https://issues.apache.org/jira/browse/Falcon-1644


Repository: falcon-git


Description
-------

Here is a sample feed validity.

        <cluster name="primaryCluster" type="source">
            <validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
            <retention limit="hours(10)" action="delete"/>
        </cluster>

In the above example, the validity time is "the time interval when the feed is valid on this
cluster". After the validity time ends, falcon is not expected to perform any operations on
the feed. The retention job for this feed will be run from validity start time up to validity
end time, and will delete any feed instances older than 10 hours. Some instances of Feed will
never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z
will never be deleted.
Ideally, the retention coordinator job should run from "validity start time" up to "validity
end time + retention age limit" to ensure all instances are handled.


Diffs (updated)
-----

  common/src/main/java/org/apache/falcon/entity/FeedHelper.java 29daff3 
  common/src/main/resources/runtime.properties f499dd9 
  docs/src/site/twiki/FalconDocumentation.twiki f384a42 
  lifecycle/src/main/java/org/apache/falcon/lifecycle/engine/oozie/retention/AgeBasedCoordinatorBuilder.java
47ab2fc 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 69ca2c3

  oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java d034b1a

  src/conf/runtime.properties 4bc1bc0 
  webapp/src/test/resources/runtime.properties 1da0ca7 

Diff: https://reviews.apache.org/r/40892/diff/


Testing
-------

End2End testing done to ensure retention coord endtime is set to "validity end time + retention
age limit"


Thanks,

Balu Vellanki


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message