hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rab ra <rab...@gmail.com>
Subject Re: Problem with RPC encryption over wire
Date Thu, 14 Nov 2013 06:15:45 GMT
Thank for the response.

I have removed rpc.protection parameter in all of my configuration and now
I am getting an error as below:-



Any Hint on whats going on here



13/11/14 10:10:47 INFO mapreduce.Job: Task Id :
attempt_1384339616944_0002_m_000026_0, Status : FAILED

Container launch failed for container_1384339616944_0002_01_000029 :
java.net.SocketTimeoutException: Call From U1204D32/40.221.95.97 to
U1204D32.blr.in.as.airbus.corp:45003 failed on socket timeout exception:
java.net.SocketTimeoutException: 60000 millis timeout while waiting for
channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected
local=/40.221.95.97:48268remote=U1204D32.blr.in.as.airbus.corp/
40.221.95.97:45003]; For more details see:
http://wiki.apache.org/hadoop/SocketTimeout

      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)

      at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)

      at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

      at java.lang.reflect.Constructor.newInstance(Constructor.java:526)

      at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)

      at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:749)

      at org.apache.hadoop.ipc.Client.call(Client.java:1351)

      at org.apache.hadoop.ipc.Client.call(Client.java:1300)

      at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)

      at com.sun.proxy.$Proxy30.startContainers(Unknown Source)

      at
org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)

      at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:151)

      at
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369)

      at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

      at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

      at java.lang.Thread.run(Thread.java:744)

Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected
local=/40.221.95.97:48268remote=U1204D32.blr.in.as.airbus.corp/
40.221.95.97:45003]

      at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)

      at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)

      at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)

      at java.io.FilterInputStream.read(FilterInputStream.java:133)

      at java.io.FilterInputStream.read(FilterInputStream.java:133)

      at
org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:457)

      at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)

      at java.io.BufferedInputStream.read(BufferedInputStream.java:254)

      at java.io.DataInputStream.readInt(DataInputStream.java:387)

      at
org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:995)

      at org.apache.hadoop.ipc.Client$Connection.run(Client.java:891)



13/11/14 10:10:50 INFO mapreduce.Job:  map 72% reduce 24%

13/11/14 10:19:13 INFO mapreduce.Job:  map 76% reduce 24%

13/11/14 10:19:55 INFO mapreduce.Job:  map 78% reduce 24%

13/11/14 10:20:10 INFO mapreduce.Job:  map 80% reduce 24%

13/11/14 10:20:29 INFO mapreduce.Job:  map 81% reduce 24%

13/11/14 10:21:50 INFO mapreduce.Job:  map 82% reduce 24%

13/11/14 10:22:01 INFO mapreduce.Job:  map 83% reduce 24%

13/11/14 10:22:10 INFO mapreduce.Job:  map 84% reduce 24%

13/11/14 10:22:13 INFO mapreduce.Job:  map 85% reduce 24%

13/11/14 10:22:16 INFO mapreduce.Job:  map 86% reduce 24%

13/11/14 10:22:27 INFO mapreduce.Job:  map 86% reduce 28%

13/11/14 10:22:30 INFO mapreduce.Job:  map 86% reduce 29%

13/11/14 10:28:30 INFO mapreduce.Job:  map 88% reduce 29%

13/11/14 10:28:35 INFO mapreduce.Job:  map 90% reduce 29%

13/11/14 10:28:37 INFO mapreduce.Job:  map 97% reduce 29%

13/11/14 10:28:43 INFO mapreduce.Job:  map 98% reduce 29%

13/11/14 10:28:45 INFO mapreduce.Job:  map 99% reduce 29%

13/11/14 10:28:48 INFO mapreduce.Job:  map 100% reduce 29%

13/11/14 10:28:55 INFO mapreduce.Job:  map 100% reduce 33%

13/11/14 10:29:06 INFO mapreduce.Job:  map 100% reduce 67%

13/11/14 10:29:27 INFO mapreduce.Job:  map 100% reduce 100%

13/11/14 10:29:33 INFO mapreduce.Job: Job job_1384339616944_0002 completed
successfully

13/11/14 10:29:38 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server

13/11/14 10:29:55 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server

13/11/14 10:29:56 INFO mapred.ClientServiceDelegate: Application state is
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history
server

13/11/14 10:29:56 ERROR security.UserGroupInformation:
PriviledgedActionException as:prrekapaee9f (auth:SIMPLE)
cause:java.io.IOException:
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException

      at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:220)

      at
org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)

      at
org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)

      at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)

      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)

      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)

      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)

      at java.security.AccessController.doPrivileged(Native Method)

      at javax.security.auth.Subject.doAs(Subject.java:415)

      at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)

      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)



 [Error] Exception:

java.io.IOException:
org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException):
java.lang.NullPointerException

      at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:220)

      at
org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)

      at
org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)

      at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)

      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928)

      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048)

      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)

      at java.security.AccessController.doPrivileged(Native Method)

      at javax.security.auth.Subject.doAs(Subject.java:415)

      at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)

      at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042)



      at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:331)

      at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:368)

      at
org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:511)

      at org.apache.hadoop.mapreduce.Job$7.run(Job.java:756)

      at org.apache.hadoop.mapreduce.Job$7.run(Job.java:753)

      at java.security.AccessController.doPrivileged(Native Method)

      at javax.security.auth.Subject.doAs(Subject.java:415)

      at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)

      at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:753)

      at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1361)

      at
org.apache.hadoop.mapred.JobClient$NetworkedJob.monitorAndPrintJob(JobClient.java:407)

      at
org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:855)

      at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:835)

      at com.eads.dcgo.uc4.DCGoUC4.run(DCGoUC4.java:593)

      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

      at com.eads.dcgo.uc4.DCGoUC4.main(DCGoUC4.java:694)

      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
On 13 Nov 2013 20:11, "Daryn Sharp" <daryn@yahoo-inc.com> wrote:

>  "No common protection layer between server and client " likely means the
> host for job submission does not have hadoop.rpc.protection=privacy.  In
> order for QOP to work, all client hosts (DN & others used to access the
> cluster) must have an identical setting.
>
>  A few quick questions: I'm assuming you mis-posted your configs and the
> protection setting isn't really commented out?  Your configs don't show
> security being enabled, but you do have it enabled, correct?  Otherwise QOP
> shouldn't apply.  Perhaps a bit obvious, but did you restart your NN after
> changing the QOP?  Since your defaultFS is just "master", are you using HA?
>
>  It's a bit concerning that you aren't consistently receiving the
> mismatch error.  Is the client looping on retries and then you get timeouts
> after 5 attempts?  If yes, we've got a major bug.  5 is the default number
> of RPC readers which handle SASL auth which means the protection mismatch
> is killing off the reader threads and rendering the NN unusable.  This
> shouldn't be possible, but what does your NN log show?
>
>   Daryn
>
>  On Nov 13, 2013, at 6:05 AM, rab ra <rabmdu@gmail.com> wrote:
>
>
>
> Hello,
>
> I am facing a problem in using Hadoop RPC encryption while transfer
> feature in hadoop 2.2.0. I have 3 node cluster
>
>
> Service running in node 1 (master)
> Resource manager
> Namenode
> DataNode
> SecondaryNamenode
>
> Service running in slaves ( node 2 & 3)
> NodeManager
>
>
>
> I am trying to make data transfer between master and slave secure. For
> that, I wanted to use data encryption over wire (RPC encryption) feature of
> hadoop 2.2.0
>
> When I ran the code, I get the below exception
>
> Caused by: java.net.SocketTimeoutException: 60000 millis timeout while
> waiting for channel to be ready for read.
>
>
> In another run, I saw in log the following error
>
> No common protection layer between server and client
>
> Not sure whether my configuration is inline with what I want to achieve.
>
> Can someone give me some hint on where I am going wrong?
>
> By the way, I have the below configuration setting in all of these nodes
>
> Core-site.xml
>
> <configuration>
>
>   <property>
>     <name>fs.defaultFS</name>
>     <value>hdfs://master:8020</value>
>   </property>
>
>   <property>
>     <name>hadoop.tmp.dir</name>
>     <value>/tmp</value>
>   </property>
> <!--
>   <property>
>     <name>hadoop.rpc.protection</name>
>     <value>privacy</value>
>   </property>
> -->
>   <property>
>     <name>io.file.buffer.size</name>
>     <value>131072</value>
>   </property>
>
> </configuration>
>
> Hdfs-site.xml
> <configuration>
>
>   <property>
>     <name>dfs.replication</name>
>     <value>1</value>
>    </property>
>
>   <property>
>     <name>dfs.name.dir</name>
>     <value>/app/hadoop/dfs-2.2.0/name</value>
>   </property>
>
>   <property>
>     <name>dfs.data.dir</name>
>     <value>/app/hadoop/dfs-2.2.0/data</value>
>   </property>
>
>   <property>
>     <name>dfs.encrypt.data.transfer</name>
>     <value>true</value>
>   </property>
>
>   <property>
>     <name>dfs.encrypt.data.transfer.algorithm</name>
>     <value>rc4</value>
>   </property>
>
>   <property>
>     <name>dfs.block.access.token.enable</name>
>     <value>true</value>
>   </property>
>
> </configuration>
>
> Mapred-site.xml
>
> <configuration>
>
>   <property>
>     <name>mapreduce.framework.name</name>
>     <value>yarn</value>
>   </property>
> <!--
>   <property>
>     <name>mapreduce.jobtracker.address</name>
>     <value>master:8032</value>
>   </property>
> -->
>   <property>
>     <name>mapreduce.tasktracker.map.tasks.maximum</name>
>     <value>1</value>
>   </property>
>
>   <property>
>     <name>mapreduce.tasktracker.reduce.tasks.maximum</name>
>     <value>1</value>
>   </property>
>
>   <property>
>     <name>mapreduce.map.speculative</name>
>     <value>false</value>
>   </property>
>
>   <property>
>     <name>mapreduce.reduce.speculative</name>
>     <value>false</value>
>   </property>
>
>   <property>
>     <name>mapreduce.map.java.opts</name>
>     <value>-Xmx1024m</value>
>   </property>
>
> </configuration>
>
>
> Yarn-site.xml
>
> <configuration>
>
>   <property>
>     <name>yarn.resourcemanager.hostname</name>
>     <value>master</value>
>   </property>
>
>   <property>
>     <name>yarn.log-aggregation-enable</name>
>     <value>true</value>
>   </property>
>
>   <property>
>     <name>yarn.nodemanager.aux-services</name>
>     <value>mapreduce_shuffle</value>
>   </property>
>
>   <property>
>     <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
>     <value>org.apache.hadoop.mapred.ShuffleHandler</value>
>   </property>
>
> </configuration>
>
>
>
> With thanks and regards
> Rab
>
>
>
>
>
>

Mime
View raw message