falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sowmya Ramesh (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (FALCON-1894) HDFS Data replication cannot be initiated independent of Oozie server location
Date Mon, 06 Jun 2016 17:58:21 GMT

    [ https://issues.apache.org/jira/browse/FALCON-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313342#comment-15313342
] 

Sowmya Ramesh edited comment on FALCON-1894 at 6/6/16 5:57 PM:
---------------------------------------------------------------

[~abush]: Issue will be seen only if HDFS DR is initiated from Falcon UI as it doesn't use
the logic in FalconClient.
Recipes submitted through Falcon CLI will not hit this issue.

Below code in HdfsReplicationRecipeTool takes care of generating the absolute path 

{noformat}
// Construct fully qualified hdfs src path
        String srcPaths = recipeProperties.getProperty(HdfsReplicationRecipeToolOptions
                .REPLICATION_SOURCE_DIR.getName());
        StringBuilder absoluteSrcPaths = new StringBuilder();
        String srcFsPath = recipeProperties.getProperty(
                HdfsReplicationRecipeToolOptions.REPLICATION_SOURCE_CLUSTER_FS_WRITE_ENDPOINT.getName());
        if (StringUtils.isNotEmpty(srcFsPath)) {
            srcFsPath = StringUtils.removeEnd(srcFsPath, File.separator);
        }
        if (StringUtils.isNotEmpty(srcPaths)) {
            String[] paths = srcPaths.split(COMMA_SEPARATOR);

            for (String path : paths) {
                StringBuilder srcpath = new StringBuilder(srcFsPath);
                srcpath.append(path.trim());
                srcpath.append(COMMA_SEPARATOR);
                absoluteSrcPaths.append(srcpath);
            }
        }
{noformat}

Fix you have recommended will not work if user wants to replicate multiple comma separated
directories.
Better workaround will be for the suer to provide the fully qualified source path to replicate
instead of modifying the  workflow template as below

{noformat}
drSourceDir=hftp://c6402.ambari.apache.org:50070/user/ambari-qa/test1/, hftp://c6402.ambari.apache.org:50070/user/ambari-qa/test2/
{noformat}

Full qualified paths should be specified only for drSourceDir and not for drTargetDir.

With https://issues.apache.org/jira/browse/FALCON-1107 recipe is renamed as extensions and
processing is moved to server side. Logic to construct the fully qualified path is on the
server side and hence will work for both UI and cmd line. 

I am closing this as fixed as it will not be sen after 0.10 release.

Thanks!






was (Author: sowmyaramesh):
[~abush]: Issue will be seen only if HDFS DR is initiated from Falcon UI as it doesn't use
the logic in FalconClient.
Recipes submitted through Falcon CLI will not hit this issue.

Below code in HdfsReplicationRecipeTool takes care of generating the absolute path 

{noformat}
// Construct fully qualified hdfs src path
        String srcPaths = recipeProperties.getProperty(HdfsReplicationRecipeToolOptions
                .REPLICATION_SOURCE_DIR.getName());
        StringBuilder absoluteSrcPaths = new StringBuilder();
        String srcFsPath = recipeProperties.getProperty(
                HdfsReplicationRecipeToolOptions.REPLICATION_SOURCE_CLUSTER_FS_WRITE_ENDPOINT.getName());
        if (StringUtils.isNotEmpty(srcFsPath)) {
            srcFsPath = StringUtils.removeEnd(srcFsPath, File.separator);
        }
        if (StringUtils.isNotEmpty(srcPaths)) {
            String[] paths = srcPaths.split(COMMA_SEPARATOR);

            for (String path : paths) {
                StringBuilder srcpath = new StringBuilder(srcFsPath);
                srcpath.append(path.trim());
                srcpath.append(COMMA_SEPARATOR);
                absoluteSrcPaths.append(srcpath);
            }
        }
{noformat}

Fix you have recommended will not work if user wants to replicate multiple comma separated
directories.
Better workaround will be for the suer to provide the fully qualified source path to replicate
instead of modifying the  workflow template as below

{noformat}
drSourceDir=hftp://c6402.ambari.apache.org:50070/user/ambari-qa/test1/, hftp://c6402.ambari.apache.org:50070/user/ambari-qa/test2/
{noformat}

With https://issues.apache.org/jira/browse/FALCON-1107 recipe is renamed as extensions and
processing is moved to server side. Logic to construct the fully qualified path is on the
server side and hence will work for both UI and cmd line. 

I am closing this as fixed as it will not be sen after 0.10 release.

Thanks!





> HDFS Data replication cannot be initiated independent of Oozie server location
> ------------------------------------------------------------------------------
>
>                 Key: FALCON-1894
>                 URL: https://issues.apache.org/jira/browse/FALCON-1894
>             Project: Falcon
>          Issue Type: Bug
>          Components: general
>    Affects Versions: trunk
>            Reporter: Alex Bush
>            Assignee: Sowmya Ramesh
>            Priority: Minor
>             Fix For: trunk, 0.10
>
>
> The HDFS mirroring scripts allow replication between two clusters.
> Currently, even though the UI allows the replication in any direction between clusters
independent of which cluster the Falcon and Oozie servers belong to this is not observed and
the source cluster is always the cluster with Oozie/Falcon server.
> Steps to reproduce:
> 1) Define both clusters in Falcon server on cluster 2
> 2) Set up HDFS mirroring in Falcon server on cluster 2 from Cluster 1 to Cluster 2 and
set to run on Oozie server of Cluster 2
> Result:
> Falcon will replicate data from Cluster 2 to Cluster 2
> Cause:
> In hdfs-replication-workflow.xml, the source dir should be defined like the target dir
by including clusterfs:
> https://github.com/apache/falcon/blob/master/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml#L63
> <arg>${drSourceDir}</arg>
> should be
> <arg>${drSourceClusterFS}${drSourceDir}</arg>
>  
> like
> https://github.com/apache/falcon/blob/master/addons/recipes/hdfs-replication/src/main/resources/hdfs-replication-workflow.xml#L65



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message