hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liyin Liang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-143) OOM in the TaskTracker while serving map outputs
Date Thu, 12 May 2011 11:56:47 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032367#comment-13032367
] 

Liyin Liang commented on MAPREDUCE-143:
---------------------------------------

The QueuedThreadPool of jetty6 own an array of jobs. If an idle thread is available a job
is directly dispatched, otherwise the job is queued to the array. At first the size of the
array is _maxThreads(40). When its full, the size grow to array.length() + _maxThreads. Because
the grow has no limit, this array can occupy too many memory when there are lots of fetch
request from reduce task. So is this jetty6's bug?

> OOM in the TaskTracker while serving map outputs
> ------------------------------------------------
>
>                 Key: MAPREDUCE-143
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-143
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Devaraj Das
>
> Saw this exception in the TT logs:
> 2009-02-06 06:18:08,553 ERROR org.mortbay.log: EXCEPTION
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> 2009-02-06 06:18:11,247 ERROR org.mortbay.log: Error for /mapOutput
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> 2009-02-06 06:18:11,247 ERROR org.mortbay.log: Error for /mapOutput
> java.lang.OutOfMemoryError: Java heap space
>         at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
>         at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
>         at org.mortbay.io.nio.IndirectNIOBuffer.<init>(IndirectNIOBuffer.java:28)
>         at org.mortbay.jetty.nio.AbstractNIOConnector.newBuffer(AbstractNIOConnector.java:71)
>         at org.mortbay.jetty.AbstractBuffers.getBuffer(AbstractBuffers.java:131)
>         at org.mortbay.jetty.HttpGenerator.addContent(HttpGenerator.java:145)
>         at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:642)
>         at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:577)
>         at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2879)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message