falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shwetha...@inmobi.com
Subject Re: Review Request 23844: FALCON-481 Simplify process parent workflow
Date Tue, 29 Jul 2014 08:41:36 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23844/
-----------------------------------------------------------

(Updated July 29, 2014, 8:41 a.m.)


Review request for Falcon.


Changes
-------

Refactored to build workflow on the fly and only action templates are checked in(no workflows).

Removed caching of templates (Every consumer of loaded action template modified the action.
So, everyone needs their own copy of the loaded action template)


Bugs: FALCON-481
    https://issues.apache.org/jira/browse/FALCON-481


Repository: falcon-git


Description
-------

Currently, process parent workflow contains 2 decision nodes which take up oozie processing
and oozie db. The refactoring creates customised workflows for fs/hcat, recordsize/without
recordsize, pig/oozie/hive workflow engine processes. It also defines pre-processing and post-processing
actions only once so that the definitions are consistent


Diffs (updated)
-----

  common/src/main/java/org/apache/falcon/entity/EntityUtil.java 6f50829 
  oozie/src/main/java/org/apache/falcon/oozie/OozieBundleBuilder.java 62d95fa 
  oozie/src/main/java/org/apache/falcon/oozie/OozieCoordinatorBuilder.java 5f483f0 
  oozie/src/main/java/org/apache/falcon/oozie/OozieEntityBuilder.java 1238b82 
  oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java 2ef162b

  oozie/src/main/java/org/apache/falcon/oozie/feed/FSReplicationWorkflowBuilder.java PRE-CREATION

  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationCoordinatorBuilder.java
0b582ef 
  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedReplicationWorkflowBuilder.java 2537725

  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionCoordinatorBuilder.java ac38532

  oozie/src/main/java/org/apache/falcon/oozie/feed/FeedRetentionWorkflowBuilder.java eee4fe9

  oozie/src/main/java/org/apache/falcon/oozie/feed/HCatReplicationWorkflowBuilder.java PRE-CREATION

  oozie/src/main/java/org/apache/falcon/oozie/process/HiveProcessWorkflowBuilder.java 79a1883

  oozie/src/main/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilder.java 977d8c1

  oozie/src/main/java/org/apache/falcon/oozie/process/PigProcessWorkflowBuilder.java 29f601d

  oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionCoordinatorBuilder.java
e46ae6e 
  oozie/src/main/java/org/apache/falcon/oozie/process/ProcessExecutionWorkflowBuilder.java
2e3a5c1 
  oozie/src/main/java/org/apache/falcon/util/OozieUtils.java 9e1c82d 
  oozie/src/main/resources/workflow/falcon-table-export.hql 37fd1b7 
  oozie/src/main/resources/workflow/falcon-table-import.hql 653d580 
  oozie/src/main/resources/workflow/feed/eviction-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/feed/falcon-table-export.hql PRE-CREATION 
  oozie/src/main/resources/workflow/feed/falcon-table-import.hql PRE-CREATION 
  oozie/src/main/resources/workflow/feed/replication-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/feed/table-cleanup.xml PRE-CREATION 
  oozie/src/main/resources/workflow/feed/table-export.xml PRE-CREATION 
  oozie/src/main/resources/workflow/feed/table-import.xml PRE-CREATION 
  oozie/src/main/resources/workflow/post-process.xml PRE-CREATION 
  oozie/src/main/resources/workflow/pre-process.xml PRE-CREATION 
  oozie/src/main/resources/workflow/process-parent-workflow.xml 4a2495c 
  oozie/src/main/resources/workflow/process/hive-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/process/oozie-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/process/pig-action.xml PRE-CREATION 
  oozie/src/main/resources/workflow/replication-workflow.xml 0748acf 
  oozie/src/main/resources/workflow/retention-workflow.xml 5138865 
  oozie/src/test/java/org/apache/falcon/oozie/feed/OozieFeedWorkflowBuilderTest.java c99c36c

  oozie/src/test/java/org/apache/falcon/oozie/process/AbstractTestBase.java cc0c419 
  oozie/src/test/java/org/apache/falcon/oozie/process/OozieProcessWorkflowBuilderTest.java
3655af9 
  src/main/examples/entity/hcat/hcat-out-feed.xml 26d1f18 

Diff: https://reviews.apache.org/r/23844/diff/


Testing
-------

UTs + end to end tests for fs usecases

Pending: hcat testing


Thanks,

shwethags


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message