hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chris Douglas (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2510) TaskTracker throw OutOfMemoryError after upgrade to jetty6
Date Wed, 18 May 2011 18:56:49 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035585#comment-13035585
] 

Chris Douglas commented on MAPREDUCE-2510:
------------------------------------------

bq. Is there an issue about upgrading Jetty to 6.1.26?

None that I'm aware of. Upgrading from Jetty5 to Jetty6 was painful, but upgrading within
Jetty6 has been very easy.

bq. Why Jetty 6.1.26 does not have this behavior?

This showed up in a cluster while I was on call. The heap dump of the TaskTracker showed exactly
what you found, i.e. a large number of Runnable instances in an array. Looking at the 6.1.14
source, it was obvious that the circular buffer was failing to free references to requests
queued in this buffer, which expanded whenever its throughput lagged behind the request rate.
Looking at the 6.1.25 source, references to requests in this buffer were correctly set to
{{null}} after dispatching them. I don't know when this was fixed, only that we've been running
6.1.26 and it's the smoothest version of Jetty we've deployed.

> TaskTracker throw OutOfMemoryError after upgrade to jetty6
> ----------------------------------------------------------
>
>                 Key: MAPREDUCE-2510
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2510
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Liyin Liang
>
> Our product cluster's TaskTracker sometimes throw OutOfMemoryError after upgrade to jetty6.
The exception in TT's log is as follows:
> 2011-05-17 19:16:40,756 ERROR org.mortbay.log: Error for /mapOutput
> java.lang.OutOfMemoryError: Java heap space
>         at java.io.BufferedInputStream.<init>(BufferedInputStream.java:178)
>         at org.apache.hadoop.fs.BufferedFSInputStream.<init>(BufferedFSInputStream.java:44)
>         at org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:176)
>         at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:359)
>         at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3040)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>         at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>         at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>         at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>         at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>         at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>         at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>         at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>         at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>         at org.mortbay.jetty.Server.handle(Server.java:324)
>         at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>         at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>         at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>         at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
> Exceptions in .out file:
> java.lang.OutOfMemoryError: Java heap space
> Exception in thread "process reaper" java.lang.OutOfMemoryError: Java heap space
> Exception in thread "pool-1-thread-1" java.lang.OutOfMemoryError: Java heap space
> java.lang.OutOfMemoryError: Java heap space
> java.lang.reflect.InvocationTargetException
> Exception in thread "IPC Server handler 6 on 50050"     at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown
Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.mortbay.log.Slf4jLog.warn(Slf4jLog.java:126)
>         at org.mortbay.log.Log.warn(Log.java:181)
>         at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:449)
>         at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>         at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>         at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>         at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>         at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>         at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>         at org.mortbay.jetty.Server.handle(Server.java:324)
>         at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>         at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>         at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>         at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message