Hi,
Has anyone seen hadoop getting stuck on reduces?
I'm using a compiled version of hadoop from cloudera:
Hadoop 0.18.3-14.cloudera.CH0_3
Subversion -r HEAD
Compiled by root on Mon Jul 6 15:02:31 EDT 2009
I've a map reduce job and hadoop gets stuck at 96.49% with no progress since the last 1 hour.
I've tried to look into the logs and there isn't anything interesting there. Here's the log
from the last 1 hour:
>>>>
2009-09-17 14:51:47,033 ERROR org.apache.hadoop.dfs.DataNode: DatanodeRegistration(10.42.1.1:50010,
storageID=DS-1052225239-129.170.192.42-50010-1252112893659, infoPort=50075, ipcPort=50020):DataXceiver:
java.net.SocketTimeoutException: 480000 millis timeout while waiting for channel to be ready
for write. ch : java.nio.channels.SocketChannel[connected local=/10.42.1.1:50010 remote=/10.42.255.247:41915]
at org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246)
at org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159)
at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198)
at org.apache.hadoop.dfs.DataNode$BlockSender.sendChunks(DataNode.java:1938)
at org.apache.hadoop.dfs.DataNode$BlockSender.sendBlock(DataNode.java:2032)
at org.apache.hadoop.dfs.DataNode$DataXceiver.readBlock(DataNode.java:1159)
at org.apache.hadoop.dfs.DataNode$DataXceiver.run(DataNode.java:1087)
at java.lang.Thread.run(Thread.java:619)
2009-09-17 14:52:29,916 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 40 blocks got
processed in 7 msecs
2009-09-17 15:15:06,586 INFO org.apache.hadoop.dfs.DataBlockScanner: Verification succeeded
for blk_4472685249728744796_2145
2009-09-17 15:52:28,686 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 40 blocks got
processed in 8 msecs
2009-09-17 16:52:30,397 INFO org.apache.hadoop.dfs.DataNode: BlockReport of 40 blocks got
processed in 8 msecs
>>>>
while the hadoop reduce is stuck at 96.49 for the last 1 hr:
09/09/17 15:49:49 INFO mapred.JobClient: map 100% reduce 96%
This is the second time I've tried to run this code and both of the times I've seen it hitting
a barrier on reduce. My reduce step is just aggregating all of the keys and dumping them into
files using MultipleTextOutputFormat. I've run a problem of similar size before where the
input to reduce was of the same size as this one.
Any help would be greatly appreciated. I can't seem to find any reason why this is happening.
Thanks
Himanshu
Morpheus: Do you believe in fate, Neo?
Neo: No.
Morpheus: Why Not?
Neo: Because I don't like the idea that I'm not in control of my life.
|