falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sowmya Ramesh" <sram...@hortonworks.com>
Subject Re: Review Request 35468: FeedReplicator improvement to include more DistCP options
Date Tue, 16 Jun 2015 22:18:08 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35468/
-----------------------------------------------------------

(Updated June 16, 2015, 10:18 p.m.)


Review request for Falcon, Pallavi Rao, Srikanth Sundarrajan, and Venkat Ranganathan.


Changes
-------

Applied feedback!


Bugs: FALCON-668
    https://issues.apache.org/jira/browse/FALCON-668


Repository: falcon-git


Description
-------

FeedReplicator improvement to include more DistCP options listed below. User is expected to
pass these as custom properties in the feed defined for replicaiton and those will be propagated
to DistCp tool.

* overwrite
* ignore errors
* skip checksum
* remove deleted files
* preserve block size 
* preserve replication count
* preserve permissions


Diffs (updated)
-----

  common/src/main/java/org/apache/falcon/util/ReplicationDistCpOption.java PRE-CREATION 
  docs/src/site/twiki/EntitySpecification.twiki 1ed2cb5 
  oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java 49f9e07

  oozie/src/main/java/org/apache/falcon/oozie/feed/FSReplicationWorkflowBuilder.java 1d97204

  oozie/src/main/java/org/apache/falcon/oozie/feed/HCatReplicationWorkflowBuilder.java 72bbca4

  replication/src/main/java/org/apache/falcon/replication/FeedReplicator.java b2175b2 
  replication/src/test/java/org/apache/falcon/replication/FeedReplicatorTest.java 539d00d


Diff: https://reviews.apache.org/r/35468/diff/


Testing
-------

Verified custom property added got propagated to FeedReplicator Java action and the replication
succeeded.

Action configuration:

<main-class>org.apache.falcon.replication.FeedReplicator</main-class>
  <arg>-Dfalcon.include.path=hftp://sandbox.hortonworks.com:50070/user/ambari-qa/dr/test/primaryCluster/feed/input/2015/06/08</arg>
  <arg>-Dmapred.job.queue.name=default</arg>
  <arg>-Dmapred.job.priority=NORMAL</arg>
  <arg>-maxMaps</arg>
  <arg>33</arg>
  <arg>-mapBandwidth</arg>
  <arg>2</arg>
  <arg>-sourcePaths</arg>
  <arg>hftp://sandbox.hortonworks.com:50070/user/ambari-qa/dr/test/primaryCluster/feed/input/2015/06/08</arg>
  <arg>-targetPath</arg>
  <arg>hdfs://sandbox.hortonworks.com:8020/user/ambari-qa/dr/test/backupCluster/feed/output/2015/06/08</arg>
  <arg>-falconFeedStorageType</arg>
  <arg>FILESYSTEM</arg>
  <arg>-availabilityFlag</arg>
  <arg>NA</arg>
  <arg>-overwrite</arg>
  <arg>true</arg>
  <arg>-ignoreErrors</arg>
  <arg>false</arg>
  <arg>-preservePermissions</arg>
  <arg>false</arg>


Thanks,

Sowmya Ramesh


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message