hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Greg Senia (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-13696) Distcp between 2 secure clusters and non encrypted zones fails with connection timeout
Date Wed, 29 May 2019 18:40:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-13696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851195#comment-16851195
] 

Greg Senia commented on HDFS-13696:
-----------------------------------

I have documented the cause of this issue in the HADOOP-14104 JIRA

> Distcp between 2 secure clusters and non encrypted zones fails with connection timeout
> --------------------------------------------------------------------------------------
>
>                 Key: HDFS-13696
>                 URL: https://issues.apache.org/jira/browse/HDFS-13696
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: distcp
>            Reporter: Rohit Pegallapati
>            Priority: Critical
>
> We are trying to do a distcp between 2 secure clusters. But distcp is between 2 unencrypted
zones.
> eg : hadoop distcp sourceHdfs://tmp/text1.txt  destHdfs://tmp
> throws
> {code:java}
> org.apache.oozie.action.ActionExecutorException: JA009: connect timed out
> 	at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:463)
> 	at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:437)
> 	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1247)
> 	at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1425)
> 	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:232)
> 	at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
> 	at org.apache.oozie.command.XCommand.call(XCommand.java:286)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 	at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:179)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.SocketTimeoutException: connect timed out
> 	at java.net.PlainSocketImpl.socketConnect(Native Method)
> 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> 	at java.net.Socket.connect(Socket.java:589)
> 	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
> 	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
> 	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
> 	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
> 	at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
> 	at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
> 	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
> 	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138)
> 	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032)
> 	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
> 	at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
> 	at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:184)
> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:139)
> 	at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348)
> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.doDelegationTokenOperation(DelegationTokenAuthenticator.java:308)
> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.getDelegationToken(DelegationTokenAuthenticator.java:180)
> 	at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.getDelegationToken(DelegationTokenAuthenticatedURL.java:382)
> 	at org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1014)
> 	at org.apache.hadoop.crypto.key.kms.KMSClientProvider$4.run(KMSClientProvider.java:1008)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> 	at org.apache.hadoop.crypto.key.kms.KMSClientProvider.addDelegationTokens(KMSClientProvider.java:1008)
> 	at org.apache.hadoop.crypto.key.KeyProviderDelegationTokenExtension.addDelegationTokens(KeyProviderDelegationTokenExtension.java:110)
> 	at org.apache.hadoop.hdfs.DistributedFileSystem.addDelegationTokens(DistributedFileSystem.java:2333)
> 	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:140)
> 	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
> 	at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
> 	at org.apache.hadoop.mapreduce.JobSubmitter.populateTokenCache(JobSubmitter.java:433)
> 	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:173)
> 	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)
> 	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> 	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304)
> 	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:578)
> 	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:573)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
> 	at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:573)
> 	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:564)
> 	at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1232)
> 	... 9 more
> {code}
> We need to open dest cluster kms port 16000 for communication from source, then this
works.
> This was working in cdh 5.5.1 using hadoop 2.6.0 without us opening up kms port on the
destination cluster when we upgraded to 5.13.2 cdh we started facing this issue



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message