Return-Path: X-Original-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-common-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 98EBE1024D for ; Fri, 4 Oct 2013 01:05:43 +0000 (UTC) Received: (qmail 66311 invoked by uid 500); 4 Oct 2013 01:05:43 -0000 Delivered-To: apmail-hadoop-common-issues-archive@hadoop.apache.org Received: (qmail 66269 invoked by uid 500); 4 Oct 2013 01:05:43 -0000 Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-issues@hadoop.apache.org Delivered-To: mailing list common-issues@hadoop.apache.org Received: (qmail 66217 invoked by uid 99); 4 Oct 2013 01:05:43 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Oct 2013 01:05:42 +0000 Date: Fri, 4 Oct 2013 01:05:42 +0000 (UTC) From: "Haohui Mai (JIRA)" To: common-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HADOOP-10016) Distcp should support copy from a secure Hadoop 1 cluster to an insecure Hadoop 2 cluster MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HADOOP-10016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13785778#comment-13785778 ] Haohui Mai commented on HADOOP-10016: ------------------------------------- Indeed this is related to HADOOP-8828. After discussions with [~sureshms], [~jingzhao], [~sanjay.radia], we believe that it is a valid use case that distcp copies data from a secure cluster to an insecure cluster. That is, distcp runs inside the secure cluster and writes to the insecure cluster. The set up is the same as the one described in HADOOP-8828. It is particularly important to support the use case of copying data from a secure Hadoop 1 cluster to an insecure Hadoop 2 cluster, since this gives users a path to migrate data from a secure Hadoop 1 cluster to a new installation of insecure Hadoop 2 cluster. The problem here is that in this set up, both distcp and map-reduce try to ask for delegation tokens in order to authenticate with the insecure cluster, in which case the insecure cluster returns an error. Currently Hadoop 2 can mostly handle this case (see HADOOP-10017), since it uses RPCv9 which supports negotiation and fallback during authentication. For Hadoop 1, however, the above use case is fundamentally _broken_ since RPCv8 does not support negotiation at all. What it means is that you cannot write to the insecure Hadoop 2 cluster directly via HDFS. You cannot write to the cluster via WebHDFS either because neither distcp nor map-reduce can successfully get delegation tokens from the insecure cluster. > Distcp should support copy from a secure Hadoop 1 cluster to an insecure Hadoop 2 cluster > ----------------------------------------------------------------------------------------- > > Key: HADOOP-10016 > URL: https://issues.apache.org/jira/browse/HADOOP-10016 > Project: Hadoop Common > Issue Type: Bug > Reporter: Haohui Mai > Assignee: Haohui Mai > > Distcp should be able to copy from a secure cluster to an insecure cluster. This functionality is important for operators to migrate data to a new Hadoop installation. -- This message was sent by Atlassian JIRA (v6.1#6144)