falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venkat Ranganathan" <n....@live.com>
Subject Review Request 35833: FALCON-1188: Falcon support for Hive Replication
Date Wed, 24 Jun 2015 18:35:49 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35833/
-----------------------------------------------------------

Review request for Falcon and Srikanth Sundarrajan.


Bugs: FALCON-1188
    https://issues.apache.org/jira/browse/FALCON-1188


Repository: falcon-git


Description
-------

Falcon support for HiveDR

As referenced in the document attached to FALCON-1188, this patch introduces Falcon support
for HiveDR

It is implemented as recipe and added to Falcon addons package and can be included into the
base product using the maven profile hivedr.

This supports HiveDR in both secure and unsecure clusters.

This is a large code base and we have repackage it to be in the addons so that we can quickly
integrate with falcon master branch and continue further refinements/development in the master
branch instead of creating a separate branch introducing merge issues

This work is the result of contribution of Balu Vellanki, Peeyush Bishnoi, Sowmya Ramesh and
Venkat Ranganathan with  help from Hive folks


Diffs
-----

  addons/hivedr/README PRE-CREATION 
  addons/hivedr/pom.xml PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/DefaultPartitioner.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/EventSourcer.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRArgs.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDROptions.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRTool.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/LastEvents.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/MetaStoreEventSourcer.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/Partitioner.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/ReplicationEventMetadata.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/exception/HiveReplicationException.java
PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyCommitter.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyMapper.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyReducer.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/util/DBReplicationStatus.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/util/DRStatusStore.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/util/DelimiterUtils.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventSourcerUtils.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventUtils.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/util/FileUtils.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRStatusStore.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRUtils.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveMetastoreUtils.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/util/ReplicationStatus.java PRE-CREATION

  addons/hivedr/src/main/resources/log4j.xml PRE-CREATION 
  addons/hivedr/src/test/java/org/apache/falcon/hive/DBReplicationStatusTest.java PRE-CREATION

  addons/hivedr/src/test/java/org/apache/falcon/hive/DRTest.java PRE-CREATION 
  addons/hivedr/src/test/java/org/apache/falcon/hive/HiveDRStatusStoreTest.java PRE-CREATION

  addons/hivedr/src/test/java/org/apache/falcon/hive/ReplicationStatusTest.java PRE-CREATION

  addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-template.xml 824e6f5

  addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml 145d489

  addons/recipes/hdfs-replication/src/main/resources/hdfs-replication.properties 19b8459 
  addons/recipes/hive-disaster-recovery/README.txt PRE-CREATION 
  addons/recipes/hive-disaster-recovery/pom.xml PRE-CREATION 
  addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure-template.xml
PRE-CREATION 
  addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure-workflow.xml
PRE-CREATION 
  addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-secure.properties
PRE-CREATION 
  addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-template.xml
PRE-CREATION 
  addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery-workflow.xml
PRE-CREATION 
  addons/recipes/hive-disaster-recovery/src/main/resources/hive-disaster-recovery.properties
PRE-CREATION 
  client/pom.xml afa91c9 
  client/src/main/java/org/apache/falcon/cli/FalconCLI.java f169917 
  client/src/main/java/org/apache/falcon/client/FalconClient.java 20c32e4 
  client/src/main/java/org/apache/falcon/recipe/HdfsReplicationRecipeTool.java PRE-CREATION

  client/src/main/java/org/apache/falcon/recipe/HdfsReplicationRecipeToolOptions.java PRE-CREATION

  client/src/main/java/org/apache/falcon/recipe/HiveReplicationRecipeTool.java PRE-CREATION

  client/src/main/java/org/apache/falcon/recipe/HiveReplicationRecipeToolOptions.java PRE-CREATION

  client/src/main/java/org/apache/falcon/recipe/Recipe.java PRE-CREATION 
  client/src/main/java/org/apache/falcon/recipe/RecipeFactory.java PRE-CREATION 
  client/src/main/java/org/apache/falcon/recipe/RecipeTool.java 069db9f 
  client/src/main/java/org/apache/falcon/recipe/RecipeToolArgs.java baa4846 
  client/src/main/java/org/apache/falcon/recipe/RecipeToolOptions.java a1c29cd 
  client/src/main/java/org/apache/falcon/recipe/util/RecipeProcessBuilderUtils.java PRE-CREATION

  common/pom.xml 36de1f5 
  docs/src/site/twiki/InstallationSteps.twiki 1dd242a 
  oozie/src/main/java/org/apache/falcon/service/SharedLibraryHostingService.java e3de6a4 
  pom.xml 0689899 
  replication/pom.xml 8c4d6b4 
  src/main/assemblies/distributed-package.xml 1e48c94 
  src/main/assemblies/standalone-package.xml 682d52f 
  test-tools/hadoop-webapp/pom.xml cadc7f5 
  webapp/pom.xml 063d42c 
  webapp/src/test/java/org/apache/falcon/cli/FalconCLIIT.java 90acb59 
  webapp/src/test/resources/process.properties 1eb282c 

Diff: https://reviews.apache.org/r/35833/diff/


Testing
-------

HiveDR has been validated both in secure and unsecure cluster configurations.  Checkstyle
and RAT checks pass.   There is an associated UI enhancement in Falcon UI that is provided
independently.
Please see the README on component version requirements.


Thanks,

Venkat Ranganathan


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message