falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Balu Vellanki (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1524) Improve Lifecycle Retention validation checks
Date Mon, 12 Oct 2015 20:07:05 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14953651#comment-14953651

Balu Vellanki commented on FALCON-1524:

[~ajayyadava] :
Can retentionFrequency.getFrequencyAsInt() be a fraction? If yes, in method validateRetentionFrequencyForOozie(...)
, you need to check for case where retentionFrequency.getTimeUnit() is  days. If  retentionFrequency.getFrequencyAsInt()
cannot be a fraction, you only have to check  for case Frequency.TimeUnit.minutes. Hours will
always be > 1. 

Falcon user should be able to set retention frequency limit for Oozie Builders in runtime.properties
(default being 1hour).

> Improve Lifecycle Retention validation checks
> ---------------------------------------------
>                 Key: FALCON-1524
>                 URL: https://issues.apache.org/jira/browse/FALCON-1524
>             Project: Falcon
>          Issue Type: Improvement
>            Reporter: Ajay Yadava
>            Assignee: Ajay Yadava
>             Fix For: 0.8
>         Attachments: FALCON-1524.patch
> 2 enhancements:
> 1. Check that retention should not be more frequent than the data availability.
> 2. Oozie doesn't behave appropriately for retention for high frequencies like minutes(3)
etc. Retention follows LAST_ONLY execution order and for high frequencies and default configuration
retention might not execute. To make it work you will need to override 
> oozie.service.CoordMaterializeTriggerService.materialization.window
> and
> oozie.service.CoordMaterializeTriggerService.lookup.interval
> which might affect the Oozie system performance.
> Hence we should set a limit like hours(1) for retention frequency and retention more
frequent than that should throw validation error.
> Originally reported by [~pragya.mittal]

This message was sent by Atlassian JIRA

View raw message