falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From shwetha...@inmobi.com
Subject Re: Review Request 16511: Process update for wf changes
Date Tue, 31 Dec 2013 10:06:51 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16511/
-----------------------------------------------------------

(Updated Dec. 31, 2013, 10:06 a.m.)


Review request for Falcon.


Bugs: FALCON-206
    https://issues.apache.org/jira/browse/FALCON-206


Repository: falcon-git


Description
-------

At process schedule, process mapper creates a checksum file that contains the list of user
wf/lib files and their checksum. The process mapper also copies the user wf and lib to staging
directory(where parent wf and coord xml are stored). The parent workflow references the user
workflow copied to staging dir. 
Process update updates in oozie only if entity is updated or user wf/lib is updated.
With this change, when user wf/lib is updated, the currently running instances will not fail.
This will also make adding new input to process easier (which includes updating process and
workflow.xml)
Limitations: user oozie workflow should use only relative paths for libs/pig scripts/hive
scripts


Diffs
-----

  common/src/main/java/org/apache/falcon/entity/ClusterHelper.java e332aba 
  common/src/main/java/org/apache/falcon/entity/EntityUtil.java ba80cac 
  common/src/main/java/org/apache/falcon/update/UpdateHelper.java fc69933 
  common/src/main/resources/log4j.xml 734d17c 
  common/src/main/resources/startup.properties 5473f5d 
  common/src/test/java/org/apache/falcon/entity/parser/FeedUpdateTest.java f39f300 
  common/src/test/java/org/apache/falcon/entity/parser/ProcessEntityParserTest.java e656772

  common/src/test/java/org/apache/falcon/update/UpdateHelperTest.java 42bcee0 
  common/src/test/resources/config/process/process-0.1.xml bb5cd35 
  common/src/test/resources/config/process/process-0.2.xml c4cd83e 
  common/src/test/resources/config/process/process-table.xml 1d6a8f0 
  feed/src/main/java/org/apache/falcon/workflow/OozieFeedWorkflowBuilder.java 7b9095f 
  oozie/src/main/java/org/apache/falcon/service/SharedLibraryHostingService.java c47ec01 
  oozie/src/main/java/org/apache/falcon/workflow/OozieWorkflowBuilder.java 1978c53 
  oozie/src/main/java/org/apache/falcon/workflow/engine/OozieHouseKeepingService.java 7e2f8a4

  oozie/src/main/java/org/apache/falcon/workflow/engine/OozieWorkflowEngine.java 1ed1ff7 
  prism/src/main/java/org/apache/falcon/resource/AbstractEntityManager.java 7ec6cd1 
  prism/src/main/resources/log4j.xml ac1d9e4 
  process/src/main/java/org/apache/falcon/converter/OozieProcessMapper.java 8749f07 
  process/src/main/java/org/apache/falcon/workflow/OozieProcessWorkflowBuilder.java 1329733

  process/src/test/java/org/apache/falcon/converter/OozieProcessMapperTest.java 7d5f4d1 
  replication/src/main/java/org/apache/falcon/replication/CustomReplicator.java ba01c80 
  replication/src/main/java/org/apache/falcon/replication/FeedReplicator.java 17227bf 
  replication/src/main/java/org/apache/falcon/replication/FilteredCopyListing.java 9c97a44

  src/bin/service-start.sh 430bb1a 
  src/conf/log4j.xml 0b28ddd 
  src/conf/startup.properties 69613f6 
  webapp/src/main/resources/log4j.xml 6790576 
  webapp/src/test/java/org/apache/falcon/logging/LogMoverIT.java 24c2959 
  webapp/src/test/java/org/apache/falcon/resource/EntityManagerJerseyIT.java cb2fcbb 
  webapp/src/test/java/org/apache/falcon/resource/TestContext.java d224f90 
  webapp/src/test/java/org/apache/falcon/util/OozieTestUtils.java 690fb6b 

Diff: https://reviews.apache.org/r/16511/diff/


Testing
-------

UT, tested with embedded deployment


Thanks,

shwethags


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message