hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liyin Liang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-143) OOM in the TaskTracker while serving map outputs
Date Thu, 12 May 2011 05:01:47 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032254#comment-13032254
] 

Liyin Liang commented on MAPREDUCE-143:
---------------------------------------

Our cluster met the similar problem after upgrade to jetty6.
related log:
2011-05-11 16:24:26,914 ERROR org.mortbay.log: Error for /mapOutput

java.lang.OutOfMemoryError: Java heap space

        at java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)

        at org.apache.hadoop.fs.BufferedFSInputStream.<init>(BufferedFSInputStream.java:44)


        at org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:176)

        at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:359)

        at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3040)

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) 

        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)

        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)

        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)

        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)

        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)

        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)

        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)

        at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)


        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)

        at org.mortbay.jetty.Server.handle(Server.java:324)

        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)

        at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)

        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)

        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)

        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)

        at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)

        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
2011-05-11 17:31:39,376 ERROR org.mortbay.log: Error for /mapOutput

2011-05-11 17:31:45,523 ERROR org.mortbay.log: Error for /mapOutput

java.lang.OutOfMemoryError: Java heap space

jmap -heap result:
Heap Configuration:

   MinHeapFreeRatio = 40

   MaxHeapFreeRatio = 70

   MaxHeapSize      = 1610612736 (1536.0MB)

   NewSize          = 1310720 (1.25MB)

   MaxNewSize       = 17592186044415 MB

   OldSize          = 5439488 (5.1875MB)

   NewRatio         = 2

   SurvivorRatio    = 8

   PermSize         = 21757952 (20.75MB)

   MaxPermSize      = 85983232 (82.0MB)



Heap Usage:

PS Young Generation

Eden Space:

   capacity = 61865984 (59.0MB)

   used     = 61865984 (59.0MB)

   free     = 0 (0.0MB)

   100.0% used

>From Space:

   capacity = 178913280 (170.625MB)

   used     = 11205368 (10.686271667480469MB)

   free     = 167707912 (159.93872833251953MB)

   6.263016361893315% used

To Space:

   capacity = 178913280 (170.625MB)

   used     = 0 (0.0MB)

   free     = 178913280 (170.625MB)

   0.0% used

PS Old Generation

   capacity = 1073741824 (1024.0MB)

   used     = 1073710024 (1023.9696731567383MB)

   free     = 31800 (0.03032684326171875MB)

   99.99703839421272% used

PS Perm Generation

   capacity = 21757952 (20.75MB)

   used     = 17614112 (16.798126220703125MB)

   free     = 4143840 (3.951873779296875MB)

   80.95482516001506% used


We dump the heap of TaskTracker and analyze it with MAT. We found one instance of "org.mortbay.thread.QueuedThreadPool"
occupies 853,258,184 (72.51%) bytes. This object contain a "java.lang.Runnable[]" which has
7200 elements.

> OOM in the TaskTracker while serving map outputs
> ------------------------------------------------
>
>                 Key: MAPREDUCE-143
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-143
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Devaraj Das
>
> Saw this exception in the TT logs:
> 2009-02-06 06:18:08,553 ERROR org.mortbay.log: EXCEPTION
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> 2009-02-06 06:18:11,247 ERROR org.mortbay.log: Error for /mapOutput
> java.lang.OutOfMemoryError: GC overhead limit exceeded
> 2009-02-06 06:18:11,247 ERROR org.mortbay.log: Error for /mapOutput
> java.lang.OutOfMemoryError: Java heap space
>         at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:39)
>         at java.nio.ByteBuffer.allocate(ByteBuffer.java:312)
>         at org.mortbay.io.nio.IndirectNIOBuffer.<init>(IndirectNIOBuffer.java:28)
>         at org.mortbay.jetty.nio.AbstractNIOConnector.newBuffer(AbstractNIOConnector.java:71)
>         at org.mortbay.jetty.AbstractBuffers.getBuffer(AbstractBuffers.java:131)
>         at org.mortbay.jetty.HttpGenerator.addContent(HttpGenerator.java:145)
>         at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:642)
>         at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:577)
>         at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2879)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message