Return-Path: X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BEB6517E23 for ; Wed, 18 Mar 2015 20:15:39 +0000 (UTC) Received: (qmail 30537 invoked by uid 500); 18 Mar 2015 20:15:39 -0000 Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org Received: (qmail 30479 invoked by uid 500); 18 Mar 2015 20:15:39 -0000 Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hdfs-issues@hadoop.apache.org Delivered-To: mailing list hdfs-issues@hadoop.apache.org Received: (qmail 30467 invoked by uid 99); 18 Mar 2015 20:15:39 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Mar 2015 20:15:39 +0000 Date: Wed, 18 Mar 2015 20:15:39 +0000 (UTC) From: "Aaron T. Myers (JIRA)" To: hdfs-issues@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HDFS-7037) Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HDFS-7037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14367803#comment-14367803 ] Aaron T. Myers commented on HDFS-7037: -------------------------------------- bq. My question is how to fix all FileSystem implementations, given that there are multiple HCFS implementations (e.g., MapRFs, Ceph) that inherit the public FileSystem APIs, all of which sit outside of the repository of hadoop? Should we ask them to take care of this issue on their own? That's up to them, but it still seems obvious to me that we should fix the FileSystem implementations that are in our repository. The alternative you've proposed, as I mentioned previously, is fixing all _users of FileSystem implementations_, of which there are obviously many outside of the Hadoop repository. bq. -1 given the concern on security vulnerability. Note that in the latest patch allowing connections to fall back to an insecure cluster is configurable, and disabled by default. So given that, making this change in HFTP is no different than how Hadoop RPC currently works, and thus there is no vulnerability being introduced here. This proposed change really only amounts to addressing a bug in HFTP that even when client fallback is enabled, HFTP still can't connect to insecure clusters, since the client can't handle it when a DT can't be fetched. If the reasoning behind your -1 is really only predicated on this being a security vulnerability, then I'd ask you to please consider withdrawing it. I'd really like to get this fixed in HFTP. It's been burning plenty of users for a long time. > Using distcp to copy data from insecure to secure cluster via hftp doesn't work (branch-2 only) > ------------------------------------------------------------------------------------------------ > > Key: HDFS-7037 > URL: https://issues.apache.org/jira/browse/HDFS-7037 > Project: Hadoop HDFS > Issue Type: Bug > Components: security, tools > Affects Versions: 2.6.0 > Reporter: Yongjun Zhang > Assignee: Yongjun Zhang > Attachments: HDFS-7037.001.patch > > > This is a branch-2 only issue since hftp is only supported there. > Issuing "distcp hftp:// hdfs://" gave the following failure exception: > {code} > 14/09/13 22:07:40 INFO tools.DelegationTokenFetcher: Error when dealing remote token: > java.io.IOException: Error when dealing remote token: Internal Server Error > at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375) > at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238) > at org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252) > at org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) > at org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247) > at org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140) > at org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337) > at org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324) > at org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457) > at org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472) > at org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501) > at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) > at org.apache.hadoop.fs.Globber.glob(Globber.java:248) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623) > at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77) > at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81) > at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342) > at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154) > at org.apache.hadoop.tools.DistCp.run(DistCp.java:121) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.tools.DistCp.main(DistCp.java:390) > 14/09/13 22:07:40 WARN security.UserGroupInformation: PriviledgedActionException as:hadoopuser@xyz.COM (auth:KERBEROS) cause:java.io.IOException: Unable to obtain remote token > 14/09/13 22:07:40 ERROR tools.DistCp: Exception encountered > java.io.IOException: Unable to obtain remote token > at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:249) > at org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:252) > at org.apache.hadoop.hdfs.web.HftpFileSystem$2.run(HftpFileSystem.java:247) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554) > at org.apache.hadoop.hdfs.web.HftpFileSystem.getDelegationToken(HftpFileSystem.java:247) > at org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:140) > at org.apache.hadoop.hdfs.web.HftpFileSystem.addDelegationTokenParam(HftpFileSystem.java:337) > at org.apache.hadoop.hdfs.web.HftpFileSystem.openConnection(HftpFileSystem.java:324) > at org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.fetchList(HftpFileSystem.java:457) > at org.apache.hadoop.hdfs.web.HftpFileSystem$LsParser.getFileStatus(HftpFileSystem.java:472) > at org.apache.hadoop.hdfs.web.HftpFileSystem.getFileStatus(HftpFileSystem.java:501) > at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57) > at org.apache.hadoop.fs.Globber.glob(Globber.java:248) > at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1623) > at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77) > at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:81) > at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:342) > at org.apache.hadoop.tools.DistCp.execute(DistCp.java:154) > at org.apache.hadoop.tools.DistCp.run(DistCp.java:121) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.tools.DistCp.main(DistCp.java:390) > Caused by: java.io.IOException: Error when dealing remote token: Internal Server Error > at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.run(DelegationTokenFetcher.java:375) > at org.apache.hadoop.hdfs.tools.DelegationTokenFetcher.getDTfromRemote(DelegationTokenFetcher.java:238) > ... 22 more > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)