hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sagar Naik <sn...@attributor.com>
Subject Re: occasional createBlockException in Hadoop .18.1
Date Mon, 15 Dec 2008 19:20:40 GMT
Hi,
Some data points on this issue.
1) "du" runs for 20-30 secs.
2) after some time , I dont see any activity in datanode logs
3) I cant even jstack the datanode (forced it , gave me a 
DebuggerException, double checked the pid), the datanode:50075/stacks 
takes forever to respond

I can telnet to datanode:50010

I think, the disk is bad or something

Pl suggest some pointers to analyze this problem

-Sagar
Sagar Naik wrote:
>
>
> CLIENT EXCEPTION:
>
> 2008-12-14 08:41:46,919 [Thread-90] INFO 
> org.apache.hadoop.dfs.DFSClient: Exception in createBlockOutputStream 
> java.net.SocketTimeoutException: 69000 millis timeout while waiting 
> for channel to be ready for read. ch : 
> java.nio.channels.SocketChannel[connected local=/10.50.80.133:54045 
> remote=/10.50.80.108:50010] 2008-12-14 08:41:46,919 [Thread-90] INFO 
> org.apache.hadoop.dfs.DFSClient: Abandoning block 
> blk_-7364265396616885025_5870078 2008-12-14 08:41:46,920 [Thread-90] 
> INFO org.apache.hadoop.dfs.DFSClient: Waiting to find target node: 
> 10.50.80.108:50010
>
>
> DATANODE
>
> 2008-12-14 08:40:39,215 INFO org.apache.hadoop.dfs.DataNode: Receiving 
> block blk_-7364265396616885025_5870078 src: /10.50.80.133:54045 dest: 
> /10.50.80.133:50010
> .
> .
> .
> .
> . 
> I occasionally see the datanode as deadnode. When the datanode is 
> deadnode, I see the du forked from datanode. The "du" is seen in "D" 
> state
>
>
>
> Any pointers to debug this information would help me
>
> -Sagar


Mime
View raw message