hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3550) Reduce tasks failing with OOM
Date Fri, 13 Jun 2008 07:46:46 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12604764#action_12604764
] 

Arun C Murthy commented on HADOOP-3550:
---------------------------------------

Interestingly, when I ran this with -XX:+HeapDumpOnOutOfMemoryError I see 2 _very large_ byte
buffers: each greater than 100MB amounting to a total of ~250M. I'm currently testing it with
hprof and heap=sites...

> Reduce tasks failing with OOM
> -----------------------------
>
>                 Key: HADOOP-3550
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3550
>             Project: Hadoop Core
>          Issue Type: Bug
>    Affects Versions: 0.17.0
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>            Priority: Blocker
>             Fix For: 0.17.1, 0.18.0
>
>
> Map-Reduce jobs which worked with 0.16.* are failing with OOM with the following trace:
> {noformat}
> java.lang.OutOfMemoryError: Java heap space
>         at java.util.Arrays.copyOf(Arrays.java:2786)
>         at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
>         at java.io.DataOutputStream.write(DataOutputStream.java:90)
>         at org.apache.hadoop.io.SequenceFile$UncompressedBytes.writeUncompressedBytes(SequenceFile.java:617)
>         at org.apache.hadoop.mapred.ReduceTask$ValuesIterator.readNextValue(ReduceTask.java:289)
>         at org.apache.hadoop.mapred.ReduceTask$ValuesIterator.next(ReduceTask.java:232)
>         at org.apache.hadoop.mapred.ReduceTask$ReduceValuesIterator.next(ReduceTask.java:311)
>         at org.apache.hadoop.streaming.PipeReducer.reduce(PipeReducer.java:67)
>         at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:391)
>         at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2124)
> {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message