hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Travis Thompson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6776) distcp from insecure cluster to secure cluster doesn't work
Date Thu, 31 Jul 2014 02:02:39 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14080393#comment-14080393
] 

Travis Thompson commented on HDFS-6776:
---------------------------------------

Also seeing this in 2.3.0 so it looks like it's been broken for a while!

With hdfs://
{noformat}
$ hadoop distcp -i /tmp/motd hdfs://insecure-namenode:9000/tmp/motd
14/07/30 23:45:20 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false,
deleteMissing=false, ignoreFailures=true, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize',
sourceFileListing=null, sourcePaths=[/tmp/motd], targetPath=hdfs://insecure-namenode:9000/tmp/motd}
14/07/30 23:45:21 INFO client.RMProxy: Connecting to ResourceManager at secure-resourcemanager/xxx.xxx.xxx.xxx:8032
14/07/30 23:45:22 ERROR tools.DistCp: Exception encountered 
java.io.IOException: Failed on local exception: java.io.IOException: Server asks us to fall
back to SIMPLE auth, but this client is configured to only allow secure connections.; Host
Details : local host is: "secure-gateway/xxx.xxx.xxx.xxx"; destination host is: "insecure-namenode":9000;

        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
        at org.apache.hadoop.ipc.Client.call(Client.java:1410)
        at org.apache.hadoop.ipc.Client.call(Client.java:1359)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
        at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at com.sun.proxy.$Proxy11.getFileInfo(Unknown Source)
        at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:671)
        at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1746)
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1112)
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1108)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1108)
        at org.apache.hadoop.fs.FileSystem.isFile(FileSystem.java:1425)
        at org.apache.hadoop.tools.SimpleCopyListing.validatePaths(SimpleCopyListing.java:69)
        at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:79)
        at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90)
        at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:80)
        at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:327)
        at org.apache.hadoop.tools.DistCp.execute(DistCp.java:151)
        at org.apache.hadoop.tools.DistCp.run(DistCp.java:118)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.tools.DistCp.main(DistCp.java:375)
Caused by: java.io.IOException: Server asks us to fall back to SIMPLE auth, but this client
is configured to only allow secure connections.
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:734)
        at org.apache.hadoop.ipc.Client$Connection.access$2700(Client.java:367)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1458)
        at org.apache.hadoop.ipc.Client.call(Client.java:1377)
        ... 26 more
{noformat}

With webhdfs://
{noformat}
$ hadoop distcp -i /tmp/motd webhdfs://insecure-namenode:50070/tmp/motd
14/07/31 00:27:42 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false,
deleteMissing=false, ignoreFailures=true, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize',
sourceFileListing=null, sourcePaths=[/tmp/motd], targetPath=webhdfs://insecure-namenode:50070/tmp/motd}
14/07/31 00:27:42 INFO client.RMProxy: Connecting to ResourceManager at secure-resourcemanager/xxx.xxx.xxx.xxx:8032
14/07/31 00:27:44 ERROR tools.DistCp: Exception encountered 
java.io.IOException: Failed to get the token for tthompso, user=tthompso
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:341)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:105)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:573)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:539)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:424)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:953)
	at org.apache.hadoop.hdfs.web.TokenAspect.ensureTokenInitialized(TokenAspect.java:143)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:227)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:381)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:402)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$FsPathRunner.getUrl(WebHdfsFileSystem.java:652)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.init(WebHdfsFileSystem.java:485)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:531)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:424)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:678)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:689)
	at org.apache.hadoop.fs.FileSystem.isFile(FileSystem.java:1425)
	at org.apache.hadoop.tools.SimpleCopyListing.validatePaths(SimpleCopyListing.java:69)
	at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:79)
	at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:90)
	at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:80)
	at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:327)
	at org.apache.hadoop.tools.DistCp.execute(DistCp.java:151)
	at org.apache.hadoop.tools.DistCp.run(DistCp.java:118)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.tools.DistCp.main(DistCp.java:375)
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): Failed to get the token
for tthompso, user=tthompso
	at org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:157)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:318)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$700(WebHdfsFileSystem.java:105)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.getResponse(WebHdfsFileSystem.java:628)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:535)
	... 22 more
{noformat}

> distcp from  insecure cluster to secure cluster doesn't work
> ------------------------------------------------------------
>
>                 Key: HDFS-6776
>                 URL: https://issues.apache.org/jira/browse/HDFS-6776
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 2.5.0
>            Reporter: Yongjun Zhang
>            Assignee: Yongjun Zhang
>
> More details to come.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message