falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pallavi Rao" <pallavi....@inmobi.com>
Subject Re: Review Request 35468: FeedReplicator improvement to include more DistCP options
Date Thu, 18 Jun 2015 09:09:15 GMT

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35468/#review88351
-----------------------------------------------------------

Ship it!


Ship It!

- Pallavi Rao


On June 16, 2015, 10:18 p.m., Sowmya Ramesh wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35468/
> -----------------------------------------------------------
> 
> (Updated June 16, 2015, 10:18 p.m.)
> 
> 
> Review request for Falcon, Pallavi Rao, Srikanth Sundarrajan, and Venkat Ranganathan.
> 
> 
> Bugs: FALCON-668
>     https://issues.apache.org/jira/browse/FALCON-668
> 
> 
> Repository: falcon-git
> 
> 
> Description
> -------
> 
> FeedReplicator improvement to include more DistCP options listed below. User is expected
to pass these as custom properties in the feed defined for replicaiton and those will be propagated
to DistCp tool.
> 
> * overwrite
> * ignore errors
> * skip checksum
> * remove deleted files
> * preserve block size 
> * preserve replication count
> * preserve permissions
> 
> 
> Diffs
> -----
> 
>   common/src/main/java/org/apache/falcon/util/ReplicationDistCpOption.java PRE-CREATION

>   docs/src/site/twiki/EntitySpecification.twiki 1ed2cb5 
>   oozie/src/main/java/org/apache/falcon/oozie/OozieOrchestrationWorkflowBuilder.java
49f9e07 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/FSReplicationWorkflowBuilder.java
1d97204 
>   oozie/src/main/java/org/apache/falcon/oozie/feed/HCatReplicationWorkflowBuilder.java
72bbca4 
>   replication/src/main/java/org/apache/falcon/replication/FeedReplicator.java b2175b2

>   replication/src/test/java/org/apache/falcon/replication/FeedReplicatorTest.java 539d00d

> 
> Diff: https://reviews.apache.org/r/35468/diff/
> 
> 
> Testing
> -------
> 
> Verified custom property added got propagated to FeedReplicator Java action and the replication
succeeded.
> 
> Action configuration:
> 
> <main-class>org.apache.falcon.replication.FeedReplicator</main-class>
>   <arg>-Dfalcon.include.path=hftp://sandbox.hortonworks.com:50070/user/ambari-qa/dr/test/primaryCluster/feed/input/2015/06/08</arg>
>   <arg>-Dmapred.job.queue.name=default</arg>
>   <arg>-Dmapred.job.priority=NORMAL</arg>
>   <arg>-maxMaps</arg>
>   <arg>33</arg>
>   <arg>-mapBandwidth</arg>
>   <arg>2</arg>
>   <arg>-sourcePaths</arg>
>   <arg>hftp://sandbox.hortonworks.com:50070/user/ambari-qa/dr/test/primaryCluster/feed/input/2015/06/08</arg>
>   <arg>-targetPath</arg>
>   <arg>hdfs://sandbox.hortonworks.com:8020/user/ambari-qa/dr/test/backupCluster/feed/output/2015/06/08</arg>
>   <arg>-falconFeedStorageType</arg>
>   <arg>FILESYSTEM</arg>
>   <arg>-availabilityFlag</arg>
>   <arg>NA</arg>
>   <arg>-overwrite</arg>
>   <arg>true</arg>
>   <arg>-ignoreErrors</arg>
>   <arg>false</arg>
>   <arg>-preservePermissions</arg>
>   <arg>false</arg>
> 
> 
> Thanks,
> 
> Sowmya Ramesh
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message