falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venkat Ranganathan" <n....@live.com>
Subject Re: Review Request 35833: FALCON-1188: Falcon support for Hive Replication
Date Thu, 09 Jul 2015 03:55:42 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35833/
-----------------------------------------------------------

(Updated July 8, 2015, 8:55 p.m.)


Review request for Falcon and Srikanth Sundarrajan.


Changes
-------

More review fixes - cleanup locations properly handled, exceptions properly thrown etc


Bugs: FALCON-1188
    https://issues.apache.org/jira/browse/FALCON-1188


Repository: falcon-git


Description
-------

Falcon support for HiveDR

As referenced in the document attached to FALCON-1188, this patch introduces Falcon support
for HiveDR

It is implemented as recipe and added to Falcon addons package and can be included into the
base product using the maven profile hivedr.

This supports HiveDR in both secure and unsecure clusters.

This is a large code base and we have repackage it to be in the addons so that we can quickly
integrate with falcon master branch and continue further refinements/development in the master
branch instead of creating a separate branch introducing merge issues

This work is the result of contribution of Balu Vellanki, Peeyush Bishnoi, Sowmya Ramesh and
Venkat Ranganathan with  help from Hive folks


Diffs (updated)
-----

  addons/hivedr/README PRE-CREATION 
  addons/hivedr/pom.xml PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/DefaultPartitioner.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/EventSourcer.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRArgs.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDROptions.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/HiveDRTool.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/LastReplicatedEvents.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/MetaStoreEventSourcer.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/Partitioner.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/ReplicationEventMetadata.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/exception/HiveReplicationException.java
PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyCommitter.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyMapper.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/mapreduce/CopyReducer.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/util/DBReplicationStatus.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/util/DRStatusStore.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/util/DelimiterUtils.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventSourcerUtils.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/util/EventUtils.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/util/FileUtils.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRStatusStore.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveDRUtils.java PRE-CREATION 
  addons/hivedr/src/main/java/org/apache/falcon/hive/util/HiveMetastoreUtils.java PRE-CREATION

  addons/hivedr/src/main/java/org/apache/falcon/hive/util/ReplicationStatus.java PRE-CREATION

  addons/hivedr/src/main/resources/log4j.xml PRE-CREATION 
  addons/hivedr/src/test/java/org/apache/falcon/hive/DBReplicationStatusTest.java PRE-CREATION

  addons/hivedr/src/test/java/org/apache/falcon/hive/DRTest.java PRE-CREATION 
  addons/hivedr/src/test/java/org/apache/falcon/hive/HiveDRStatusStoreTest.java PRE-CREATION

  addons/hivedr/src/test/java/org/apache/falcon/hive/ReplicationStatusTest.java PRE-CREATION


Diff: https://reviews.apache.org/r/35833/diff/


Testing
-------

HiveDR has been validated both in secure and unsecure cluster configurations.  Checkstyle
and RAT checks pass.   There is an associated UI enhancement in Falcon UI that is provided
independently.
Please see the README on component version requirements.


Thanks,

Venkat Ranganathan


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message