hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tsz Wo Nicholas Sze (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6376) Distcp data between two HA clusters requires another configuration
Date Tue, 17 Nov 2015 01:51:11 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15007804#comment-15007804
] 

Tsz Wo Nicholas Sze commented on HDFS-6376:
-------------------------------------------

> ... Users can use a new configuration property "dfs.internal.nameservices" to explicitly
specify the name services belonging to the local cluster, while continue using the configuration
property "dfs.nameservices" to specify all the name services in the local and remote clusters.


Is this an incompatible change?  Balancer does not work with such setting since it will use
all the NN specified in dfs.nameservices.  As a result, it will try to balance both the local
and the remote clusters.

> Distcp data between two HA clusters requires another configuration
> ------------------------------------------------------------------
>
>                 Key: HDFS-6376
>                 URL: https://issues.apache.org/jira/browse/HDFS-6376
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: datanode, federation, hdfs-client
>    Affects Versions: 2.2.0, 2.3.0, 2.4.0
>         Environment: Hadoop 2.3.0
>            Reporter: Dave Marion
>            Assignee: Dave Marion
>             Fix For: 2.6.0
>
>         Attachments: HDFS-6376-2.patch, HDFS-6376-3-branch-2.4.patch, HDFS-6376-4-branch-2.4.patch,
HDFS-6376-5-trunk.patch, HDFS-6376-6-trunk.patch, HDFS-6376-7-trunk.patch, HDFS-6376-branch-2.4.patch,
HDFS-6376-patch-1.patch, HDFS-6376.000.patch, HDFS-6376.008.patch, HDFS-6376.009.patch, HDFS-6376.010.patch,
HDFS-6376.011.patch
>
>
> User has to create a third set of configuration files for distcp when transferring data
between two HA clusters.
> Consider the scenario in [1]. You cannot put all of the required properties in core-site.xml
and hdfs-site.xml for the client to resolve the location of both active namenodes. If you
do, then the datanodes from cluster A may join cluster B. I can not find a configuration option
that tells the datanodes to federate blocks for only one of the clusters in the configuration.
> [1] http://mail-archives.apache.org/mod_mbox/hadoop-user/201404.mbox/%3CBAY172-W2133964E0C283968C161DD1520%40phx.gbl%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message