falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venkat Ranganathan" <n....@live.com>
Subject Re: Review Request 35833: FALCON-1188: Falcon support for Hive Replication
Date Tue, 11 Aug 2015 03:08:18 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35833/
-----------------------------------------------------------

(Updated Aug. 10, 2015, 8:08 p.m.)


Review request for Falcon and Srikanth Sundarrajan.


Changes
-------

Patch with the recipe test case fixed (Thanks Sowmya)


Bugs: FALCON-1188
    https://issues.apache.org/jira/browse/FALCON-1188


Repository: falcon-git


Description
-------

Falcon support for HiveDR

As referenced in the document attached to FALCON-1188, this patch introduces Falcon support
for HiveDR

It is implemented as recipe and added to Falcon addons package and can be included into the
base product using the maven profile hivedr.

This supports HiveDR in both secure and unsecure clusters.

This is a large code base and we have repackage it to be in the addons so that we can quickly
integrate with falcon master branch and continue further refinements/development in the master
branch instead of creating a separate branch introducing merge issues

This work is the result of contribution of Balu Vellanki, Peeyush Bishnoi, Sowmya Ramesh and
Venkat Ranganathan with  help from Hive folks


Diffs (updated)
-----

  Installation-steps.txt bb92a85 
  addons/hivedr/README PRE-CREATION 
  addons/hivedr/pom.xml PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/DefaultPartitioner.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/EventSourcer.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRArgs.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDROptions.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRTool.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/LastReplicatedEvents.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/MetaStoreEventSourcer.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/Partitioner.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/ReplicationEventMetadata.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/exception/HiveReplicationException.java
PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyCommitter.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyMapper.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyReducer.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/util/DBReplicationStatus.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/util/DRStatusStore.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/util/DelimiterUtils.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventSourcerUtils.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventUtils.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/util/FileUtils.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRStatusStore.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRUtils.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveMetastoreUtils.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/util/ReplicationStatus.java PRE-CREATION

  addons/hivedr/src/main/resources/log4j.xml PRE-CREATION 
  addons/hivedr/src/test/java/org/apache/falcon/hive/DBReplicationStatusTest.java PRE-CREATION

  addons/hivedr/src/test/java/org/apache/falcon/hive/DRTest.java PRE-CREATION 
  addons/hivedr/src/test/java/org/apache/falcon/hive/HiveDRStatusStoreTest.java PRE-CREATION

  addons/hivedr/src/test/java/org/apache/falcon/hive/ReplicationStatusTest.java PRE-CREATION

  addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-template.xml 824e6f5

  addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml 145d489

  addons/recipes/hdfs-replication/src/main/resources/hdfs-replication.properties 19b8459 
  addons/recipes/hive-disaster-recovery/README.txt PRE-CREATION 
  addons/recipes/hive-disaster-recovery/pom.xml PRE-CREATION 
  addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure-template.xml
PRE-CREATION 
  addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure-workflow.xml
PRE-CREATION 
  addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure.properties
PRE-CREATION 
  addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-template.xml
PRE-CREATION 
  addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-workflow.xml
PRE-CREATION 
  addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery.properties
PRE-CREATION 
  client/pom.xml afa91c9 
  client/src/main/java/org/apache/falcon/cli/FalconCLI.java 148f789 
  client/src/main/java/org/apache/falcon/client/FalconClient.java 9649e10 
  client/src/main/java/org/apache/falcon/recipe/HdfsReplicationRecipeTool.java PRE-CREATION

  client/src/main/java/org/apache/falcon/recipe/HdfsReplicationRecipeToolOptions.java PRE-CREATION

  client/src/main/java/org/apache/falcon/recipe/HiveReplicationRecipeTool.java PRE-CREATION

  client/src/main/java/org/apache/falcon/recipe/HiveReplicationRecipeToolOptions.java PRE-CREATION

  client/src/main/java/org/apache/falcon/recipe/Recipe.java PRE-CREATION 
  client/src/main/java/org/apache/falcon/recipe/RecipeFactory.java PRE-CREATION 
  client/src/main/java/org/apache/falcon/recipe/RecipeTool.java 069db9f 
  client/src/main/java/org/apache/falcon/recipe/RecipeToolArgs.java baa4846 
  client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java a1c29cd 
  client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java PRE-CREATION

  docs/src/site/twiki/InstallationSteps.twiki 3dd034b 
  oozie/src/main/java/org/apache/falcon/service/SharedLibraryHostingService.java e3de6a4 
  pom.xml 34a5471 
  replication/pom.xml 8c4d6b4 
  src/main/assemblies/distributed-package.xml 1e48c94 
  src/main/assemblies/standalone-package.xml 682d52f 
  test-tools/hadoop-webapp/pom.xml cadc7f5 
  webapp/pom.xml 063d42c 
  webapp/src/test/java/org/apache/falcon/cli/FalconCLIIT.java 90acb59 
  webapp/src/test/resources/hdfs-replication-template.xml PRE-CREATION 
  webapp/src/test/resources/hdfs-replication.properties PRE-CREATION 
  webapp/src/test/resources/process.properties 1eb282c 

Diff: https://reviews.apache.org/r/35833/diff/


Testing
-------

HiveDR has been validated both in secure and unsecure cluster configurations.  Checkstyle
and RAT checks pass.   There is an associated UI enhancement in Falcon UI that is provided
independently.
Please see the README on component version requirements.


Thanks,

Venkat Ranganathan


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message