hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mayuran Yogarajah <mayuran.yogara...@casalemedia.com>
Subject What can cause: Map output copy failure
Date Thu, 07 Jan 2010 21:16:50 GMT
I'm seeing this error when a job runs:

Shuffling 35338524 bytes (35338524 raw bytes) into RAM from attempt_201001051549_0036_m_000003_0
Map output copy failure: java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMemory(ReduceTask.java:1198)

I originally had mapred.child.java.opts set to 200M.  If I boost this up 
to 512M the error goes away.
I'm trying to understand whats going on though.  Can anyone explain? 
Also are there any other parameters that
I should be tweaking to help with this?

thank you very much,

View raw message