hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Siddharth Tiwari <siddharth.tiw...@live.com>
Subject RE: Strange error on Datanodes
Date Tue, 03 Dec 2013 14:46:51 GMT
Thanks Jeet
 
can you suggest me the parameter which controls the timeout value ? 

*------------------------*

Cheers !!!

Siddharth Tiwari

Have a refreshing day !!!
"Every duty is holy, and devotion to duty is the highest form of worship of God.” 

"Maybe other people will try to limit me but I don't limit myself"

 
Date: Tue, 3 Dec 2013 15:38:50 +0530
Subject: Re: Strange error on Datanodes
From: jeetuyadav200890@gmail.com
To: user@hadoop.apache.org; cdh-user@cloudera.org

Sorry for the incomplete mail.
Instead of one issue I think you may have two issues going on. I'm also adding CDH mailing
list for more inputs on the same.

1.
2013-12-02 13:11:36,441 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor
exception  for block BP-1854340821-10.238.9.151-1385733655875:blk_-2927699636194035560_63092
java.net.SocketTimeoutException: 65000 millis timeout while waiting for channel to be ready
for read. ch : java.nio.channels.SocketChannel[connected 
<> This error could be possible in a scenario where your DN process having long time
GC push, Increasing the timeout value may resolve this issue. Or your client connect could
be disconnected abnormal. 
2. 2013-12-02 13:12:06,586 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: brtlvlts0088co:50010:DataXceiver
error processing WRITE_BLOCK operation  src: /10.238.10.43:54040 dest: /10.238.10.43:50010
java.io.IOException: Premature EOF from inputStream 	at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:194)
<> Try to increase the dfs.datanode.max.xcievers conf value in the datanode hdfs-site.conf



RegardsJitendra



On Tue, Dec 3, 2013 at 3:17 PM, Jitendra Yadav <jeetuyadav200890@gmail.com> wrote:

I did some analysis on the provided logs and confs.
Instead of one issue i believe you may have two issue going on.
1.java.net.SocketTimeoutException: 65000 millis timeout while waiting for channel to be ready
for read. ch : java.nio.channels.SocketChannel[connected 
2. 2013-12-02 13:12:06,586 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: brtlvlts0088co:50010:DataXceiver
error processing WRITE_BLOCK operation  src: /10.238.10.43:54040 dest: /10.238.10.43:50010
java.io.IOException: Premature EOF from inputStream
	at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:194)






On Mon, Dec 2, 2013 at 9:30 PM, Siddharth Tiwari <siddharth.tiwari@live.com> wrote:






Hi JeetI am using CDH 4 , but I have manually installed NN and JT with HA not using cdh manager.
I am attaching NN logs here, I sent a mail just before this for other files. This is frustrating
, why is it happening.



*------------------------*

Cheers !!!

Siddharth Tiwari

Have a refreshing day !!!
"Every duty is holy, and devotion to duty is the highest form of worship of God.” 



"Maybe other people will try to limit me but I don't limit myself"


Date: Mon, 2 Dec 2013 21:24:43 +0530


Subject: Re: Strange error on Datanodes
From: jeetuyadav200890@gmail.com
To: user@hadoop.apache.org



Which hadoop destro you are using?, It would be good if you share the logs from data node
on which the data block(blk_-2927699636194035560_63092) exist and from name nodes also.
Regards


Jitendra

On Mon, Dec 2, 2013 at 9:13 PM, Siddharth Tiwari <siddharth.tiwari@live.com> wrote:






Hi Jeet
I have a cluster of size 25, 4 Admin nodes and 21 datanodes.


2 NN 2 JT 3 Zookeepers and 3 QJNs
if you could help me in understanding what kind of logs you want I will provide it to you.
Do you need hdfs-site.xml, core-site.xml and mapred-site.xmls ?





*------------------------*

Cheers !!!

Siddharth Tiwari

Have a refreshing day !!!
"Every duty is holy, and devotion to duty is the highest form of worship of God.” 




"Maybe other people will try to limit me but I don't limit myself"


Date: Mon, 2 Dec 2013 21:09:03 +0530



Subject: Re: Strange error on Datanodes
From: jeetuyadav200890@gmail.com
To: user@hadoop.apache.org




Hi,
Can you share some more logs from Data nodes? could you please also share the conf and cluster
size?
RegardsJitendra  




On Mon, Dec 2, 2013 at 8:49 PM, Siddharth Tiwari <siddharth.tiwari@live.com> wrote:







Hi team
I see following errors on datanodes. What is the reason for this and how can it will be resolved:-




2013-12-02 13:11:36,441 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor
exception  for block BP-1854340821-10.238.9.151-1385733655875:blk_-2927699636194035560_63092
java.net.SocketTimeoutException: 65000 millis timeout while waiting for channel to be ready
for read. ch : java.nio.channels.SocketChannel[connected local=/10.238.10.43:54040 remote=/10.238.10.43:50010]
	at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:165)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:156)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:129)
	at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:117)
	at java.io.FilterInputStream.read(FilterInputStream.java:83)
	at java.io.FilterInputStream.read(FilterInputStream.java:83)
	at org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:169)
	at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:114)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:694)
2013-12-02 13:12:06,572 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs'
truncater with mapRetainSize=-1 and reduceRetainSize=-1
2013-12-02 13:12:06,581 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:hadoop (auth:SIMPLE) cause:java.io.IOException: All datanodes 10.238.10.43:50010 are bad.
Aborting...
2013-12-02 13:12:06,581 WARN org.apache.hadoop.mapred.Child: Error running child
java.io.IOException: All datanodes 10.238.10.43:50010 are bad. Aborting...
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:959)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:779)
	at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:448)
2013-12-02 13:12:06,583 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task

*------------------------*

Cheers !!!

Siddharth Tiwari

Have a refreshing day !!!
"Every duty is holy, and devotion to duty is the highest form of worship of God.” 





"Maybe other people will try to limit me but I don't limit myself"
 		 	   		  





 		 	   		  

 		 	   		  



 		 	   		  
Mime
View raw message