falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venkat Ranganathan" <n....@live.com>
Subject Re: Review Request 40892: FALCON-1644 Retention : Some feed instances are never deleted by retention jobs.
Date Thu, 03 Dec 2015 05:27:44 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40892/#review108780
-----------------------------------------------------------

Ship it!


Thanks for fixing it Balu


oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java (line
64)
<https://reviews.apache.org/r/40892/#comment168250>

    I think this is a good change but it is a different behavior.   Initially thought it would
be good to  to provide a runtime property to say whether we want to use this behavior (which
we can make as default or the old behavior.
    
    But I think this is more of a bug fix and should be OK without any runtime property to
switch to old behavior.  May be others have different opinion.


- Venkat Ranganathan


On Dec. 2, 2015, 8:39 p.m., Balu Vellanki wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40892/
> -----------------------------------------------------------
> 
> (Updated Dec. 2, 2015, 8:39 p.m.)
> 
> 
> Review request for Falcon and Venkat Ranganathan.
> 
> 
> Bugs: Falcon-1644
>     https://issues.apache.org/jira/browse/Falcon-1644
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> Here is a sample feed validity.
> 
>         <cluster name="primaryCluster" type="source">
>             <validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
>             <retention limit="hours(10)" action="delete"/>
>         </cluster>
> 
> In the above example, the validity time is "the time interval when the feed is valid
on this cluster". After the validity time ends, falcon is not expected to perform any operations
on the feed. The retention job for this feed will be run from validity start time up to validity
end time, and will delete any feed instances older than 10 hours. Some instances of Feed will
never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z
will never be deleted.
> Ideally, the retention coordinator job should run from "validity start time" up to "validity
end time + retention age limit" to ensure all instances are handled.
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/falcon/entity/FeedHelper.java d601c5d 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java
69ca2c3 
>   oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java
d034b1a 
> 
> Diff: https://reviews.apache.org/r/40892/diff/
> 
> 
> Testing
> -------
> 
> End2End testing done to ensure retention coord endtime is set to "validity end time +
retention age limit"
> 
> 
> Thanks,
> 
> Balu Vellanki
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message