hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Lowe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4730) AM crashes due to OOM while serving up map task completion events
Date Fri, 19 Oct 2012 19:23:11 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480289#comment-13480289

Jason Lowe commented on MAPREDUCE-4730:

Update on testing, I was able to test this (along with the fix for MAPREDUCE-4733) using a
sleep job with 20000 maps and 3000 reduces on a cluster big enough to mass-launch the map
and reduce phases.  The AM with a 1.5GB slot size stayed up during the job, where previously
it failed even with a larger slot.

The only issue I ran into was a significant number of maps and reduces failed because they
timed out trying to establish a connection to the AM.  I suspected the AM could have been
busy garbage collecting and causing the delays, so I bumped up the AM size to 3G and it ran
smoothly with no connection timeout failures from any tasks.
> AM crashes due to OOM while serving up map task completion events
> -----------------------------------------------------------------
>                 Key: MAPREDUCE-4730
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4730
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster, mrv2
>    Affects Versions: 0.23.3
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>            Priority: Blocker
>         Attachments: MAPREDUCE-4730.patch, MAPREDUCE-4730.patch
> We're seeing a repeatable OOM crash in the AM for a task with around 30000 maps and 3000
reducers.  Details to follow.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message