Mailing-List: contact dev-help@falcon.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@falcon.apache.org
Date: Tue, 10 Nov 2015 01:18:11 +0000 (UTC)
From: "Balu Vellanki (JIRA)" <jira@apache.org>
To: dev@falcon.incubator.apache.org
Message-ID: <JIRA.12852432.1438819690000.22890.1447118291048@Atlassian.JIRA>
In-Reply-To: <JIRA.12852432.1438819690000@Atlassian.JIRA>
References: <JIRA.12852432.1438819690000@Atlassian.JIRA>
 <JIRA.12852432.1438819690097@arcas>
Subject: [jira] [Commented] (FALCON-1372) Retention does not work in corner
 cases
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/FALCON-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14997781#comment-14997781 ] 

Balu Vellanki commented on FALCON-1372:
---------------------------------------

The bug is caused due to the way java.util.Calendar handles DAY_OF_MONTH. In FeedHelper, the getDate(...) method sets DAY_OF_MONTH to 0 for date patterns like ${YEAR}/${MONTH}.  This causes the month to be setback by 1. So "2015/11" will be treated as Oct 31st, 2015. Hence this instance will be evicted. 

 For a date like "2015/11" , Calender should be set to  cal.set(2015, 10, 1, 0, 0, 0);  The MONTH should be "value - 1" and The DAY_OF_MONTH begins with a 1 instead of 0. 

> Retention does not work in corner cases 
> ----------------------------------------
>
>                 Key: FALCON-1372
>                 URL: https://issues.apache.org/jira/browse/FALCON-1372
>             Project: Falcon
>          Issue Type: Bug
>    Affects Versions: 0.6.1
>            Reporter: Balu Vellanki
>            Assignee: Balu Vellanki
>
> Originally reported by [~rostafiychuk].
> RetentionTest#testRetention (24, DAYS, true, MONTHLY, true)
> Uploaded feed:
> {code:xml}
> <?xml version="1.0" encoding="UTF-8"?><feed xmlns="uri:falcon:feed:0.1" name="A852708b0-3057547e" description="clicks log">
>     <frequency>hours(1)</frequency>
>     <timezone>UTC</timezone>
>     <late-arrival cut-off="hours(6)"/>
>     <clusters>
>         <cluster name="A852708b0-cd30589f" type="source">
>             <validity start="2012-01-30T00:00Z" end="2099-03-31T23:59Z"/>
>             <retention limit="days(24)" action="delete"/>
>         </cluster>
>     </clusters>
>     <locations>
>         <location type="data" path="/tmp/falcon-regression/RetentionTest/testFolders/${YEAR}/${MONTH}"/>
>         <location type="stats" path="/projects/falcon/clicksStats"/>
>         <location type="meta" path="/projects/falcon/clicksMetaData"/>
>     </locations>
>     <ACL owner="hrt_qa" group="users" permission="*"/>
>     <schema location="/schema/clicks" provider="protobuf"/>
>     <properties/>
> </feed>
> {code}
> Created dirs:
> {code}
> dataDates = [2012/06, 2012/07, 2012/08, 2012/09, 2012/10, 2012/11, 2012/12, 2013/01, 2013/02, 2013/03, 2013/04, 2013/05, 2013/06, 2013/07, 2013/08, 2013/09, 2013/10, 2013/11, 2013/12, 2014/01, 2014/02, 2014/03, 2014/04, 2014/05, 2014/06, 2014/07, 2014/08, 2014/09, 2014/10, 2014/11, 2014/12, 2015/01, 2015/02, 2015/03, 2015/04, 2015/05, 2015/06, 2015/07, 2015/08, 2015/09, 2015/10, 2015/11, 2015/12, 2016/01, 2016/02, 2016/03, 2016/04, 2016/05, 2016/06, 2016/07, 2016/08, 2016/09, 2016/10, 2016/11, 2016/12, 2017/01, 2017/02, 2017/03, 2017/04, 2017/05, 2017/06, 2017/07, 2017/08, 2017/09, 2017/10, 2017/11, 2017/12, 2018/01, 2018/02, 2018/03, 2018/04, 2018/05, 2018/06, 2018/07] {code}
> expected for deletion:
> {code} [2012/06, 2012/07, 2012/08, 2012/09, 2012/10, 2012/11, 2012/12, 2013/01, 2013/02, 2013/03, 2013/04, 2013/05, 2013/06, 2013/07, 2013/08, 2013/09, 2013/10, 2013/11, 2013/12, 2014/01, 2014/02, 2014/03, 2014/04, 2014/05, 2014/06, 2014/07, 2014/08, 2014/09, 2014/10, 2014/11, 2014/12, 2015/01, 2015/02, 2015/03, 2015/04, 2015/05]
> {code}
> deleted:
> {code:javascript}[2012/06, 2012/07, 2012/08, 2012/09, 2012/10, 2012/11, 2012/12, 2013/01, 2013/02, 2013/03, 2013/04, 2013/05, 2013/06, 2013/07, 2013/08, 2013/09, 2013/10, 2013/11, 2013/12, 2014/01, 2014/02, 2014/03, 2014/04, 2014/05, 2014/06, 2014/07, 2014/08, 2014/09, 2014/10, 2014/11, 2014/12, 2015/01, 2015/02, 2015/03, 2015/04, 2015/05, 2015/06]
> {code}
> 24 days ago was 31th of May, but *2015/06* was deleted too.
> Another case:
> RetentionTest#testRetention (10080, HOURS, false, MONTHLY, true)
> {code:xml}
> <retention limit="hours(10080)" action="delete"/>
> {code}
> 10080 hours ago was 30th of April, but dir *"2014/05"* was deleted.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)