falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Srikanth Sundarrajan" <srik...@hotmail.com>
Subject Re: Review Request 34161: Falcon Native Scheduler - Refactor existing workflow/coord/bundle builder
Date Tue, 07 Jul 2015 04:41:04 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34161/#review90639
-----------------------------------------------------------

Ship it!


Ship It!

- Srikanth Sundarrajan


On July 3, 2015, 11:38 a.m., Pallavi Rao wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34161/
> -----------------------------------------------------------
> 
> (Updated July 3, 2015, 11:38 a.m.)
> 
> 
> Review request for Falcon.
> 
> 
> Bugs: FALCON-1170
>     https://issues.apache.org/jira/browse/FALCON-1170
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> The current set of builders assume that a bundle is going to be submitted to Oozie, and
hence, populate configuration properties at the coord level. When a native scheduler is built
out, we will submitting workflows to Oozie (not coords/bundles) as the scheduling of the workflows
will be done within Falcon.
> 
> Re-factored the builder code as follows:
> 1. Generate a config-default.xml along with workflow.xml whenever a workflow is built.
The default config file will have most of the configurations set in it. Note that the config
values can be overridden during run/re-run using job.properties.
> 2. Move all workflow related configuration to config-default.xml and keep only coord
related variables (input, output, timestamp, startTime, endTime, frequency) in the coordinator.xml
> 3. Make bundle.xml really lean. Only app path, entity name and path will be set there.
> 4. Mostly code (methods) moved around to ensure the config is populated at the right
level.
> 
> 
> Diffs
> -----
> 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieBundleBuilder.java 03063f4 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java e5d75fb 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java f00290e 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java
49f9e07 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FSReplicationWorkflowBuilder.java
1d97204 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java
de6f373 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationWorkflowBuilder.java
aa936ad 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java
c896d5a 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java
51e081f 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/HCatReplicationWorkflowBuilder.java
72bbca4 
>   oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java
60f9fe1 
>   oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java
8b18ecc 
>   oozie/src/main/java/org/apache/falcon/util/OozieUtils.java 0ae229c 
>   oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java
b223447 
>   oozie/src/test/java/org/apache/falcon/oozie/process/AbstractTestBase.java 6488682 
>   oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java
4e5c3f0 
> 
> Diff: https://reviews.apache.org/r/34161/diff/
> 
> 
> Testing
> -------
> 
> UTs modified. ITs run and manual testing done using falcon examples.
> 
> 
> Thanks,
> 
> Pallavi Rao
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message