hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David Parks" <davidpark...@yahoo.com>
Subject RE: Map output copy failure
Date Wed, 12 Dec 2012 06:15:54 GMT
I had the same problem yesterday, it sure does look to be dead on that
issue. I found another forum discussion on AWS that suggested more memory as
a stop-gap way to deal with it, or apply the patch. I checked the code on
hadoop 1.0.3 (the version on AWS) and it didn't have the fix, so it looks
like it's only in the newer builds. I actually have an AWS ticket opened for
it seeing if their engineers can offer any guidance as well.

 

My understanding is that it should be doing a shuffle on disk in this case,
it appeared to be just a small fix (a few lines) to apply the patch to
src/mapred/org/apache/hadoop/mapred/ReduceTask.java

 

Dave

 

 

 

From: Manoj Babu [mailto:manoj444@gmail.com] 
Sent: Monday, December 10, 2012 8:09 PM
To: user@hadoop.apache.org
Subject: Reg: Map output copy failure

 

Hi All

 

I got the below exception, Is the issue related to
https://issues.apache.org/jira/browse/MAPREDUCE-1182 ?

 

Am using CDH3U1

2012-12-10 06:22:39,688 FATAL org.apache.hadoop.mapred.Task:
attempt_201211120903_9197_r_000024_0 : Map output copy failure :
java.lang.OutOfMemoryError: Java heap space
        at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.shuffleInMe
mory(ReduceTask.java:1593)
        at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutpu
t(ReduceTask.java:1453)
        at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(
ReduceTask.java:1302)
        at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceT
ask.java:1234)

Cheers!

Manoj.

 


Mime
View raw message