hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Florian Leibert <...@leibert.de>
Subject Re: Hadoop (0.18) Spill Failed, out of Heap Space error
Date Wed, 10 Sep 2008 20:50:34 GMT
Hi Chris,
where do you find those values? I don't seem to see them in the  
userlogs nor in the tasktracker logs...

Thanks!

Florian
On Sep 3, 2008, at 2:50 PM, Chris Douglas wrote:

> InMemValBytes::reset can perform an allocation, but it should be  
> only as large as the value. When you look at the log for the failed  
> task, what does it report as the values of bufstart, bufend,  
> kvstart, kvend, etc. before the spill? -C
>
> On Sep 3, 2008, at 9:49 AM, Paco NATHAN wrote:
>
>> Also, that almost always happens early in the map phase of the first
>> MR job which runs on our cluster.
>>
>> Hadoop 0.18.1 on EC2 m1.xl instances.
>>
>> We run 10 MR jobs in sequence, 6hr duration, not seeing the problem
>> repeated after that 1 heap space exception.
>>
>> Paco
>>
>>
>> On Wed, Sep 3, 2008 at 11:42 AM, Florian Leibert <flo@leibert.de>  
>> wrote:
>>> Hi,
>>> we're running 100 XLarge instances (ec2), with a gig of heap space  
>>> for each
>>> task - and are seeing the following error frequently (but not  
>>> always):
>>> ##### BEGIN PASTE #####
>>>   [exec] 08/09/03 11:21:09 INFO mapred.JobClient:  map 43% reduce 5%
>>>   [exec] 08/09/03 11:21:16 INFO mapred.JobClient: Task Id :
>>> attempt_200809031101_0001_m_000220_0, Status : FAILED
>>>   [exec] java.io.IOException: Spill failed
>>>   [exec]     at
>>> org.apache.hadoop.mapred.MapTask 
>>> $MapOutputBuffer.flush(MapTask.java:688)
>>>   [exec]     at org.apache.hadoop.mapred.MapTask.run(MapTask.java: 
>>> 228)
>>>   [exec]     at
>>> org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java: 
>>> 2209)
>>>   [exec] Caused by: java.lang.OutOfMemoryError: Java heap space
>>>   [exec]     at
>>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer 
>>> $InMemValBytes.reset(MapTask.java:928)
>>>   [exec]     at
>>> org.apache.hadoop.mapred.MapTask 
>>> $MapOutputBuffer.getVBytesForOffset(MapTask.java:891)
>>>   [exec]     at
>>> org.apache.hadoop.mapred.MapTask 
>>> $MapOutputBuffer.sortAndSpill(MapTask.java:765)
>>>   [exec]     at
>>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access 
>>> $1600(MapTask.java:286)
>>>   [exec]     at
>>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer 
>>> $SpillThread.run(MapTask.java:712)
>>> ##### END #####
>>>
>>> Has anyone seen this? Thanks,
>>>
>>> Florian Leibert
>>> Sr. Software Engineer
>>> Adknowledge Inc.
>>>
>


Mime
View raw message