hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pallavi Palleti <pallavip...@gmail.com>
Subject Map Task is failing with out of memory issue
Date Thu, 28 Aug 2008 15:04:06 GMT

Hi,
 I have running map-reduce task on huge data on 40 Quad Core(16GB RAM)
Machine hadoop cluster. As my mapper machines sufficiently run on 1GB heap
size. I had set map reduce tasks heap size to 1GB. And, total number of
tasks per node is set to 9 ( 5 map tasks and 4 reduce tasks). It took huge
amount of time and finally after mapper completed 98%, the job failed with
below error:

java.lang.OutOfMemoryError: Java heap space
	at
org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:52)
	at org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:90)
	at
org.apache.hadoop.io.SequenceFile$Reader.nextRawKey(SequenceFile.java:1974)
	at
org.apache.hadoop.io.SequenceFile$Sorter$SegmentDescriptor.nextRawKey(SequenceFile.java:3002)
	at
org.apache.hadoop.io.SequenceFile$Sorter$MergeQueue.merge(SequenceFile.java:2802)
	at org.apache.hadoop.io.SequenceFile$Sorter.merge(SequenceFile.java:2511)
	at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.mergeParts(MapTask.java:1040)
	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:698)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:220)
	at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)

I am not able to figure out where things went wrong. Can some one please
help me in this?

Thanks
Pallavi

-- 
View this message in context: http://www.nabble.com/Map-Task-is-failing-with-out-of-memory-issue-tp19202548p19202548.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.


Mime
View raw message