hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From himanshu chandola <himanshu_cool...@yahoo.com>
Subject hadoop hangs on reduce
Date Thu, 17 Sep 2009 21:08:05 GMT
Hi,
Has anyone seen hadoop getting stuck on reduces? 
I'm using a compiled version of hadoop from cloudera:

Hadoop 0.18.3-14.cloudera.CH0_3
Subversion  -r HEAD
Compiled by root on Mon Jul  6 15:02:31 EDT 2009

I've a map reduce job and hadoop gets stuck at 96.49% with no progress since the last 1 hour.
I've tried to look into the logs and there isn't anything interesting there. Here's the log
from the last  1 hour:

>>>>
2009-09-17 14:51:47,033 ERROR org.apache.hadoop.dfs.DataNode: DatanodeRegistration(10.42.1.1:50010,
storageID=DS-1052225239-129.170.192.42-50010-1252112893659, infoPort=50075, ipcPort=50020):DataXceiver:
java.net.SocketTimeoutException: 480000 millis timeout while waiting for channel to be ready
for write. ch : java.nio.channels.SocketChannel[connected local=/10.42.1.1:50010 remote=/10.42.255.247:41915]
    at org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246)
    at org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)
    at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)
    at org.apache.hadoop.dfs.DataNode$BlockSender.sendChunks(DataNode.java:1938)
    at org.apache.hadoop.dfs.DataNode$BlockSender.sendBlock(DataNode.java:2032)
    at org.apache.hadoop.dfs.DataNode$DataXceiver.readBlock(DataNode.java:1159)
    at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:1087)
    at java.lang.Thread.run(Thread.java:619)

2009-09-17 14:52:29,916 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 40 blocks got
processed in 7 msecs
2009-09-17 15:15:06,586 INFO org.apache.hadoop.dfs.DataBlockScanner: Verification succeeded
for blk_4472685249728744796_2145
2009-09-17 15:52:28,686 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 40 blocks got
processed in 8 msecs
2009-09-17 16:52:30,397 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 40 blocks got
processed in 8 msecs
>>>>
while the hadoop reduce is stuck at 96.49 for the last 1 hr:
09/09/17 15:49:49 INFO mapred.JobClient:  map 100% reduce 96%

This is the second time I've tried to run this code and both of the times I've seen it hitting
a barrier on reduce. My reduce step is just aggregating all of the keys and dumping them into
files using MultipleTextOutputFormat. I've run a problem of similar size before where the
input to reduce was of the same size as this one.

Any help would be greatly appreciated. I can't seem to find any reason why this is happening.

Thanks

Himanshu

 Morpheus: Do you believe in fate, Neo?
Neo: No.
Morpheus: Why Not?
Neo: Because I don't like the idea that I'm not in control of my life.



      

Mime
View raw message