falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ajay Yadava" <ajayn...@gmail.com>
Subject Re: Review Request 39242: Improve Lifecycle Retention Validation checks
Date Tue, 13 Oct 2015 07:28:53 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39242/
-----------------------------------------------------------

(Updated Oct. 13, 2015, 7:28 a.m.)


Review request for Falcon.


Changes
-------

Addressed review comment from Balu Vellanki.


Bugs: FALCON-1524
    https://issues.apache.org/jira/browse/FALCON-1524


Repository: falcon-git


Description
-------

2 enhancements:
1. Check that retention should not be more frequent than the data availability.
2. Oozie doesn't behave appropriately for retention for high frequencies like minutes(3) etc.
Retention follows LAST_ONLY execution order and for high frequencies and default configuration
retention might not execute. To make it work you will need to override 
oozie.service.CoordMaterializeTriggerService.materialization.window
and
oozie.service.CoordMaterializeTriggerService.lookup.interval
which might affect the Oozie system performance.
Hence we should set a limit like hours(1) for retention frequency and retention more frequent
than that should throw validation error.
Originally reported by Pragya Mittal


Diffs (updated)
-----

  common/src/main/java/org/apache/falcon/entity/parser/FeedEntityParser.java 6be2495 
  common/src/main/java/org/apache/falcon/lifecycle/retention/AgeBasedDelete.java a4ae780 
  common/src/main/java/org/apache/falcon/util/DateUtil.java f89ef64 
  common/src/test/java/org/apache/falcon/entity/parser/FeedEntityParserTest.java 1c43800 

Diff: https://reviews.apache.org/r/39242/diff/


Testing
-------

Added Unit tests for all the scenarios.


Thanks,

Ajay Yadava


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message