Mailing-List: contact dev-help@falcon.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@falcon.apache.org
Content-Type: multipart/alternative;
 boundary="===============8229077206003583952=="
MIME-Version: 1.0
Subject: Re: Review Request 40892: FALCON-1644 Retention : Some feed instances
 are never deleted by retention jobs.
From: "Ajay Yadava" <ajaynsit@gmail.com>
To: "Venkat Ranganathan" <n.r.v@live.com>
Cc: "Balu Vellanki" <bvellanki@hortonworks.com>,
 "Falcon" <dev@falcon.incubator.apache.org>,
 "Ajay Yadava" <ajaynsit@gmail.com>
Date: Thu, 10 Dec 2015 11:36:21 -0000
Message-ID: <20151210113621.1619.1504@reviews.apache.org>
Auto-Submitted: auto-generated
Sender: "Ajay Yadava" <noreply@reviews.apache.org>
References: <20151208192017.1618.25928@reviews.apache.org>
In-Reply-To: <20151208192017.1618.25928@reviews.apache.org>
Reply-To: "Ajay Yadava" <ajaynsit@gmail.com>

--===============8229077206003583952==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40892/#review109743
-----------------------------------------------------------


common/src/main/java/org/apache/falcon/entity/FeedHelper.java (line 1015)
<https://reviews.apache.org/r/40892/#comment169367>

    I just noticed that you are using frequency for lifecycle and limit for old retention. I think limit is correct for both. For lifecycle this is available in the property:  "retention.policy.agebaseddelete.limit"


oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java (line 68)
<https://reviews.apache.org/r/40892/#comment169368>

    We need same change for lifecycle retention stage also.


- Ajay Yadava


On Dec. 8, 2015, 7:20 p.m., Balu Vellanki wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40892/
> -----------------------------------------------------------
> 
> (Updated Dec. 8, 2015, 7:20 p.m.)
> 
> 
> Review request for Falcon and Venkat Ranganathan.
> 
> 
> Bugs: Falcon-1644
>     https://issues.apache.org/jira/browse/Falcon-1644
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> Here is a sample feed validity.
> 
>         <cluster name="primaryCluster" type="source">
>             <validity start="2015-10-30T01:00Z" end="2015-10-30T10:00Z"/>
>             <retention limit="hours(10)" action="delete"/>
>         </cluster>
> 
> In the above example, the validity time is "the time interval when the feed is valid on this cluster". After the validity time ends, falcon is not expected to perform any operations on the feed. The retention job for this feed will be run from validity start time up to validity end time, and will delete any feed instances older than 10 hours. Some instances of Feed will never be deleted. In the above example, feed instances at between 2015-10-30T00:00Z and 2015-10-30T10:00Z will never be deleted.
> Ideally, the retention coordinator job should run from "validity start time" up to "validity end time + retention age limit" to ensure all instances are handled.
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/falcon/entity/FeedHelper.java d601c5d 
>   common/src/main/resources/runtime.properties f499dd9 
>   docs/src/site/twiki/FalconDocumentation.twiki f384a42 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java 69ca2c3 
>   oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java d034b1a 
>   src/conf/runtime.properties 4bc1bc0 
>   webapp/src/test/resources/runtime.properties 1da0ca7 
> 
> Diff: https://reviews.apache.org/r/40892/diff/
> 
> 
> Testing
> -------
> 
> End2End testing done to ensure retention coord endtime is set to "validity end time + retention age limit"
> 
> 
> Thanks,
> 
> Balu Vellanki
> 
>


--===============8229077206003583952==--