falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pallavi Rao" <pallavi....@inmobi.com>
Subject Re: Review Request 34161: Falcon Native Scheduler - Refactor existing workflow/coord/bundle builder
Date Tue, 30 Jun 2015 04:40:52 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34161/
-----------------------------------------------------------

(Updated June 30, 2015, 4:40 a.m.)


Review request for Falcon.


Changes
-------

Rebased patch


Bugs: FALCON-1170
    https://issues.apache.org/jira/browse/FALCON-1170


Repository: falcon-git


Description
-------

The current set of builders assume that a bundle is going to be submitted to Oozie, and hence,
populate configuration properties at the coord level. When a native scheduler is built out,
we will submitting workflows to Oozie (not coords/bundles) as the scheduling of the workflows
will be done within Falcon.

Re-factored the builder code as follows:
1. Generate a config-default.xml along with workflow.xml whenever a workflow is built. The
default config file will have most of the configurations set in it. Note that the config values
can be overridden during run/re-run using job.properties.
2. Move all workflow related configuration to config-default.xml and keep only coord related
variables (input, output, timestamp, startTime, endTime, frequency) in the coordinator.xml
3. Make bundle.xml really lean. Only app path, entity name and path will be set there.
4. Mostly code (methods) moved around to ensure the config is populated at the right level.


Diffs (updated)
-----

  oozie/src/main/java/org/apache/falcon/oozie/OozieBundleBuilder.java 03063f4 
  oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java e5d75fb 
  oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java f00290e 
  oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java 49f9e07

  oozie/src/main/java/org/apache/falcon/oozie/feed/FSReplicationWorkflowBuilder.java 1d97204

  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java
de6f373 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationWorkflowBuilder.java aa936ad

  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java c896d5a

  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java 51e081f

  oozie/src/main/java/org/apache/falcon/oozie/feed/HCatReplicationWorkflowBuilder.java 72bbca4

  oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java
60f9fe1 
  oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java
8b18ecc 
  oozie/src/main/java/org/apache/falcon/util/OozieUtils.java 0ae229c 
  oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java b223447

  oozie/src/test/java/org/apache/falcon/oozie/process/AbstractTestBase.java 6488682 
  oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java
4e5c3f0 

Diff: https://reviews.apache.org/r/34161/diff/


Testing
-------

UTs modified. ITs run and manual testing done using falcon examples.


Thanks,

Pallavi Rao


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message