falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sowmya Ramesh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FALCON-1894) HDFS Data replication cannot be initiated independent of Oozie server location
Date Thu, 02 Jun 2016 23:56:59 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313342#comment-15313342

Sowmya Ramesh commented on FALCON-1894:

[~abush]: Issue will be seen only if HDFS DR is initiated from Falcon UI as it doesn't use
the logic in FalconClient.
Recipes submitted through Falcon CLI will not hit this issue.

Below code in HdfsReplicationRecipeTool takes care of generating the absolute path 

// Construct fully qualified hdfs src path
        String srcPaths = recipeProperties.getProperty(HdfsReplicationRecipeToolOptions
        StringBuilder absoluteSrcPaths = new StringBuilder();
        String srcFsPath = recipeProperties.getProperty(
        if (StringUtils.isNotEmpty(srcFsPath)) {
            srcFsPath = StringUtils.removeEnd(srcFsPath, File.separator);
        if (StringUtils.isNotEmpty(srcPaths)) {
            String[] paths = srcPaths.split(COMMA_SEPARATOR);

            for (String path : paths) {
                StringBuilder srcpath = new StringBuilder(srcFsPath);

Fix you have recommended will not work if user wants to replicate multiple comma separated
Better workaround will be for the suer to provide the fully qualified source path to replicate
instead of modifying the  workflow template as below

drSourceDir=hftp://c6402.ambari.apache.org:50070/user/ambari-qa/test1/, hftp://c6402.ambari.apache.org:50070/user/ambari-qa/test2/

With https://issues.apache.org/jira/browse/FALCON-1107 recipe is renamed as extensions and
processing is moved to server side. Logic to construct the fully qualified path is on the
server side and hence will work for both UI and cmd line. 

I am closing this as fixed as it will not be sen after 0.10 release.


> HDFS Data replication cannot be initiated independent of Oozie server location
> ------------------------------------------------------------------------------
>                 Key: FALCON-1894
>                 URL: https://issues.apache.org/jira/browse/FALCON-1894
>             Project: Falcon
>          Issue Type: Bug
>          Components: general
>    Affects Versions: trunk
>            Reporter: Alex Bush
>            Priority: Minor
>             Fix For: trunk
> The HDFS mirroring scripts allow replication between two clusters.
> Currently, even though the UI allows the replication in any direction between clusters
independent of which cluster the Falcon and Oozie servers belong to this is not observed and
the source cluster is always the cluster with Oozie/Falcon server.
> Steps to reproduce:
> 1) Define both clusters in Falcon server on cluster 2
> 2) Set up HDFS mirroring in Falcon server on cluster 2 from Cluster 1 to Cluster 2 and
set to run on Oozie server of Cluster 2
> Result:
> Falcon will replicate data from Cluster 2 to Cluster 2
> Cause:
> In hdfs-replication-workflow.xml, the source dir should be defined like the target dir
by including clusterfs:
> https://github.com/apache/falcon/blob/master/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml#L63
> <arg>${drSourceDir}</arg>
> should be
> <arg>${drSourceClusterFS}${drSourceDir}</arg>
> like
> https://github.com/apache/falcon/blob/master/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml#L65

This message was sent by Atlassian JIRA

View raw message