hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-2053) OutOfMemoryError : Java heap space errors in hadoop 0.14
Date Mon, 15 Oct 2007 10:17:50 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-2053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Arun C Murthy updated HADOOP-2053:
----------------------------------

    Attachment: HADOOP-2053_1_20071015.patch

Here is a patch which frees the reference to the large {{DataOutputBuffer}} that {{BasicTypeSorterBase}}
has in it's {{close}} method... this lets the GC collect away the keyValBuffer. 

In absence of this patch, there is a window where both the currently active keyValBuffer and
the one that should have been freed in the previous iteration are both active i.e. doubling
the required amount of memory, which leads to the OutOfMemoryException.

All credit to this goes to Koji!

> OutOfMemoryError : Java heap space errors in hadoop 0.14
> --------------------------------------------------------
>
>                 Key: HADOOP-2053
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2053
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.14.0, 0.14.1, 0.14.2
>            Reporter: lohit vijayarenu
>            Assignee: Arun C Murthy
>            Priority: Blocker
>             Fix For: 0.14.3
>
>         Attachments: HADOOP-2053_1_20071015.patch
>
>
> In recent hadoop 0.14 we are seeing few jobs where map taskf fail with java.lang.OutOfMemoryError:
Java heap space problem
> These were the same jobs which used to work fine with 0.13
> <stack>
> task_200710112103_0001_m_000015_1: java.lang.OutOfMemoryError: Java heap space
> 	at java.util.Arrays.copyOf(Arrays.java:2786)
> 	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> 	at java.io.DataOutputStream.write(DataOutputStream.java:90)
> 	at org.apache.hadoop.io.Text.write(Text.java:243)
> 	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:340)
> </stack>

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message