falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ajay Yadava (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1524) Improve Lifecycle Retention validation checks
Date Tue, 13 Oct 2015 06:28:05 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14954481#comment-14954481
] 

Ajay Yadava commented on FALCON-1524:
-------------------------------------

Thanks for reviewing [~bvellanki].

Frequency can not be fraction. This is ensured by checks at xsd level. Here is the relevant
section from feed xsd. 
{code}
<xs:simpleType name="frequency-type">
        <xs:annotation>
            <xs:appinfo>
                <jaxb:javaType name="org.apache.falcon.entity.v0.Frequency"
                               parseMethod="org.apache.falcon.entity.v0.Frequency.fromString"
                               printMethod="org.apache.falcon.entity.v0.Frequency.toString"/>
            </xs:appinfo>
        </xs:annotation>
        <xs:restriction base="xs:string">
            <xs:pattern value="(minutes|hours|days|months)\([1-9]\d*\)"/>
        </xs:restriction>
    </xs:simpleType>
{code}
You are spot on in saying that I don't need to check for hours (I thought I deleted that but
somehow the patch doesn't contain it :) Thanks for catching it. 

The reason I have not put the limit as a property as it is specific to Oozie Scheduler only
e.g. for native scheduler this won't(shouldn't be as minutely frequencies are a strong requirement
for native scheduler) be the case.

> Improve Lifecycle Retention validation checks
> ---------------------------------------------
>
>                 Key: FALCON-1524
>                 URL: https://issues.apache.org/jira/browse/FALCON-1524
>             Project: Falcon
>          Issue Type: Improvement
>            Reporter: Ajay Yadava
>            Assignee: Ajay Yadava
>             Fix For: 0.8
>
>         Attachments: FALCON-1524.patch
>
>
> 2 enhancements:
> 1. Check that retention should not be more frequent than the data availability.
> 2. Oozie doesn't behave appropriately for retention for high frequencies like minutes(3)
etc. Retention follows LAST_ONLY execution order and for high frequencies and default configuration
retention might not execute. To make it work you will need to override 
> oozie.service.CoordMaterializeTriggerService.materialization.window
> and
> oozie.service.CoordMaterializeTriggerService.lookup.interval
> which might affect the Oozie system performance.
> Hence we should set a limit like hours(1) for retention frequency and retention more
frequent than that should throw validation error.
> Originally reported by [~pragya.mittal]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message