hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ruben Quintero <rfq_...@yahoo.com>
Subject Out of Memory during Reduce Merge
Date Fri, 11 Jun 2010 21:19:46 GMT
Hi all,

We have a MapReduce job writing a Lucene index (modeled closely after the example in contrib),
and we keep hitting out of memory exceptions in the reduce phase once the number of files
grows large.

Here are the relevant non-default values in our mapred-site.xml:

mapred.child.java.opts: -Xmx1024M -XX:+UseConcMarkSweepGC
mapred.reduce.parallel.copies: 20
io.sort.factor: 100
io.sort.mb: 200

Looking through the output logs, I think the problem occurs here:

INFO org.apache.hadoop.mapred.ReduceTask: Merging 7 files, 3005057577 bytes from disk

>From what I can tell, it is trying to merge more than twice the heapsize from disk, thus
triggering the OOM. I've posted part of the log file below.

We were looking at various mapreduce settings (mapred.job.shuffle.input.buffer.percent, mapred.job.shuffle.merge.percent,
mapred.inmem.merge.threshold), but weren't sure which settings might cause this issue. Does
anyone have any insight or suggestions as to where to start?

Thank you,

- Ruben

2010-06-11 15:27:11,519 INFO org.apache.hadoop.mapred.Merger: Merging 25 sorted segments
2010-06-11 15:27:11,528 INFO org.apache.hadoop.mapred.Merger: Down to the last merge-pass,
with 25 segments left of total size: 321695767 bytes
2010-06-11 15:27:12,783 INFO org.apache.hadoop.mapred.ReduceTask: Merged 25 segments, 321695767
bytes to disk to satisfy reduce memory limit
2010-06-11 15:27:12,785 INFO org.apache.hadoop.mapred.ReduceTask: Merging 7 files, 3005057577
bytes from disk
2010-06-11 15:27:12,799 INFO org.apache.hadoop.mapred.ReduceTask: Merging 0 segments, 0 bytes
from memory into reduce
2010-06-11 15:27:12,799 INFO org.apache.hadoop.mapred.Merger: Merging 7 sorted segments
2010-06-11 15:27:20,891 FATAL org.apache.hadoop.mapred.TaskTracker: Error running child :
java.lang.OutOfMemoryError: Java heap space
	at org.apache.hadoop.mapred.IFile$Reader.readNextBlock(IFile.java:342)
	at org.apache.hadoop.mapred.IFile$Reader.next(IFile.java:404)
	at org.apache.hadoop.mapred.Merger$Segment.next(Merger.java:220)
	at org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:420)
	at org.apache.hadoop.mapred.Merger.merge(Merger.java:142)
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.createKVIterator(ReduceTask.java:2298)
	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier.access$400(ReduceTask.java:570)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
	at org.apache.hadoop.mapred.Child.main(Child.java:170)

View raw message