hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Haohui Mai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10016) Distcp should support copy from a secure Hadoop 1 cluster to an insecure Hadoop 2 cluster
Date Fri, 04 Oct 2013 01:05:42 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13785778#comment-13785778
] 

Haohui Mai commented on HADOOP-10016:
-------------------------------------

Indeed this is related to HADOOP-8828.

After discussions with [~sureshms], [~jingzhao], [~sanjay.radia], we believe that it is a
valid use case that distcp copies data from a secure cluster to an insecure cluster. That
is, distcp runs inside the secure cluster and writes to the insecure cluster. The set up is
the same as the one described in HADOOP-8828.

It is particularly important to support the use case of copying data from a secure Hadoop
1 cluster to an insecure Hadoop 2 cluster, since this gives users a path to migrate data from
a secure Hadoop 1 cluster to a new installation of insecure Hadoop 2 cluster.

The problem here is that in this set up, both distcp and map-reduce try to ask for delegation
tokens in order to authenticate with the insecure cluster, in which case the insecure cluster
returns an error. Currently Hadoop 2 can mostly handle this case (see HADOOP-10017), since
it uses RPCv9 which supports negotiation and fallback during authentication.

For Hadoop 1, however, the above use case is fundamentally _broken_ since RPCv8 does not support
negotiation at all. What it means is that you cannot write to the insecure Hadoop 2 cluster
directly via HDFS. You cannot write to the cluster via WebHDFS either because neither distcp
nor map-reduce can successfully get delegation tokens from the insecure cluster.

> Distcp should support copy from a secure Hadoop 1 cluster to an insecure Hadoop 2 cluster
> -----------------------------------------------------------------------------------------
>
>                 Key: HADOOP-10016
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10016
>             Project: Hadoop Common
>          Issue Type: Bug
>            Reporter: Haohui Mai
>            Assignee: Haohui Mai
>
> Distcp should be able to copy from a secure cluster to an insecure cluster. This functionality
is important for operators to migrate data to a new Hadoop installation.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message