hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Björn-Elmar Macek <ma...@cs.uni-kassel.de>
Subject Re: mortbay, huge files and the ulimit
Date Fri, 31 Aug 2012 12:08:54 GMT
Hi all,

i got some more information:

It is always only one TT, where mortbay errors do not occur: all other 
slaves fail relatively fast after the reduce process started: maybe it 
is a comparator issue (since the operation is MAPRED_SHUFFLE)
And it has nothing to do with the local configuration of the server, since
1) if i rerun the job on the same set of servers, the ONE working 
differs from one job to another.
2) the same happens even when i am using a completely different set of 
servers.

Who is capable to solve this unsolved mystery? :D

Am 30.08.2012 12:27, schrieb Björn-Elmar Macek:
> Hi,
>
> is there anybody, who knows more about this issue: it has already been 
> recently marked here:
> https://issues.apache.org/jira/browse/MAPREDUCE-5
>
> I really want to do something about it, if i knew how... i tried so 
> many different setup parameters and JVM options and nothing did the 
> trick...
>
> It was opened 2009 and gets still updated (2012): I mean - this does 
> NOT look OK, does it? The whole TT seems like a big time waster:
>
> 2012-08-30 12:05:22,582 INFO 
> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
> 141.51.205.130:35497, dest: 141.51.205.126:39586, bytes: 146086410, 
> op: MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000189_0, 
> duration: 5456122852
> 2012-08-30 12:05:56,275 INFO org.mortbay.log: 
> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
> injecting delay2 times
> 2012-08-30 12:05:56,276 INFO org.mortbay.log: 
> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
> recreating selector 2 times, canceled keys 70 times
> 2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker: 
> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
> org.mortbay.jetty.EofException
>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>     at 
> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>     at 
> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>     at 
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>     at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>     at 
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>     at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>     at 
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>     at 
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>     at 
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>     at 
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>     at 
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>     at 
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>     at 
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>     at org.mortbay.jetty.Server.handle(Server.java:326)
>     at 
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>     at 
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>     at 
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>     at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Caused by: java.io.IOException: Broken pipe
>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>     at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>     at 
> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>     ... 27 more
>
> 2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410 
> getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
> org.mortbay.jetty.EofException
>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>     at 
> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>     at 
> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>     at 
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>     at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>     at 
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>     at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>     at 
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>     at 
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>     at 
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>     at 
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>     at 
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>     at 
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>     at 
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>     at org.mortbay.jetty.Server.handle(Server.java:326)
>     at 
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>     at 
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>     at 
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>     at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Caused by: java.io.IOException: Broken pipe
>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>     at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>     at 
> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>     ... 27 more
>
> 2012-08-30 12:08:39,236 INFO 
> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
> 141.51.205.130:35497, dest: 141.51.205.126:39614, bytes: 65536, op: 
> MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000204_0, duration: 
> 3209410
> 2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
> java.lang.IllegalStateException: Committed
>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>     at 
> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>     at 
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>     at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>     at 
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>     at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>     at 
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>     at 
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>     at 
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>     at 
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>     at 
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>     at 
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>     at 
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>     at org.mortbay.jetty.Server.handle(Server.java:326)
>     at 
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>     at 
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>     at 
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>     at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> 2012-08-30 12:10:27,109 INFO 
> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
> 141.51.205.130:35497, dest: 141.51.205.126:39618, bytes: 145730308, 
> op: MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000204_0, 
> duration: 5686096517
> 2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker: 
> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
> org.mortbay.jetty.EofException
>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>     at 
> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>     at 
> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>     at 
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>     at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>     at 
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>     at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>     at 
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>     at 
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>     at 
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>     at 
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>     at 
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>     at 
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>     at 
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>     at org.mortbay.jetty.Server.handle(Server.java:326)
>     at 
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>     at 
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>     at 
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>     at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Caused by: java.io.IOException: Broken pipe
>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>     at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>     at 
> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>     ... 27 more
>
> 2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410 
> getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
> org.mortbay.jetty.EofException
>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>     at 
> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>     at 
> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>     at 
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>     at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>     at 
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>     at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>     at 
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>     at 
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>     at 
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>     at 
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>     at 
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>     at 
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>     at 
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>     at org.mortbay.jetty.Server.handle(Server.java:326)
>     at 
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>     at 
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>     at 
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>     at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Caused by: java.io.IOException: Broken pipe
>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>     at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>     at 
> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>     ... 27 more
>
> 2012-08-30 12:14:29,740 INFO 
> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
> 141.51.205.130:35497, dest: 141.51.205.126:39648, bytes: 65536, op: 
> MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000236_0, duration: 
> 3746212
> 2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
> java.lang.IllegalStateException: Committed
>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>     at 
> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>     at 
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>     at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>     at 
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>     at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>     at 
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>     at 
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>     at 
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>     at 
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>     at 
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>     at 
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>     at 
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>     at org.mortbay.jetty.Server.handle(Server.java:326)
>     at 
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>     at 
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>     at 
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>     at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> 2012-08-30 12:16:28,503 INFO 
> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
> 141.51.205.130:35497, dest: 141.51.205.126:39652, bytes: 145506420, 
> op: MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000236_0, 
> duration: 7793391393
> 2012-08-30 12:16:59,421 INFO org.mortbay.log: 
> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
> injecting delay1 times
> 2012-08-30 12:16:59,422 INFO org.mortbay.log: 
> org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
> recreating selector 1 times, canceled keys 69 times
> 2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker: 
> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
> org.mortbay.jetty.EofException
>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>     at 
> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>     at 
> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>     at 
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>     at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>     at 
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>     at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>     at 
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>     at 
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>     at 
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>     at 
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>     at 
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>     at 
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>     at 
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>     at org.mortbay.jetty.Server.handle(Server.java:326)
>     at 
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>     at 
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>     at 
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>     at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Caused by: java.io.IOException: Broken pipe
>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>     at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>     at 
> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>     ... 27 more
>
> 2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410 
> getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
> org.mortbay.jetty.EofException
>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
>     at 
> org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
>     at 
> org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
>     at 
> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>     at 
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>     at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>     at 
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>     at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>     at 
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>     at 
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>     at 
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>     at 
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>     at 
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>     at 
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>     at 
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>     at org.mortbay.jetty.Server.handle(Server.java:326)
>     at 
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>     at 
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>     at 
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>     at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Caused by: java.io.IOException: Broken pipe
>     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
>     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
>     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
>     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
>     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
>     at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
>     at 
> org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
>     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
>     ... 27 more
>
> 2012-08-30 12:20:01,205 INFO 
> org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
> 141.51.205.130:35497, dest: 141.51.205.126:39683, bytes: 65536, op: 
> MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000241_0, duration: 
> 5405031
> 2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
> java.lang.IllegalStateException: Committed
>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>     at 
> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>     at 
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>     at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>     at 
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>     at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>     at 
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>     at 
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>     at 
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>     at 
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>     at 
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>     at 
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>     at 
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>     at org.mortbay.jetty.Server.handle(Server.java:326)
>     at 
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>     at 
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>     at 
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>     at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>
> Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
>> Hi there,
>>
>> i am currently running a job where i selfjoin a 63 gigabyte big csv 
>> file on 20 physically distinct nodes with 15GB each:
>>
>> While the mapping works just fine and is low cost, the reducer does 
>> the main work: holding a hashmap with elements to join with and 
>> finding join tuples for evry incoming key-value-pair.
>>
>> The jobs works perfectly on small files with 2 gigabytes, but starts 
>> to get "unstable" as the file size goes up: this becomes evident with 
>> a look into the tasktracker's logs saying:
>>
>> ERROR org.mortbay.log: /mapOutput
>> java.lang.IllegalStateException: Committed
>>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>>     at 
>> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>     at 
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>>     at 
>> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>>     at 
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>>     at 
>> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>     at 
>> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>>     at 
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>     at 
>> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>>     at 
>> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>     at 
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>     at org.mortbay.jetty.Server.handle(Server.java:326)
>>     at 
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>>     at 
>> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>     at 
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>>     at 
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>>
>>
>> And while it is no problem at the beginning of the reduce process, 
>> where this happens only on a few nodes and rarely, it becomes crucial 
>> as the progress rises. The reason for this (afaik from reading 
>> articles), is that there are memory or file handle problems. I 
>> addressed the memory problem by conitiously purging the map of 
>> outdated elements evry 5 million processed key-value-pairs. And i 
>> set  mapred.child.ulimit to 100000000 (ulimit in the shell tells me 
>> it is 400000000).
>>
>> Anyway i am still running into those mortbay errors and i start to 
>> wonder, if hadoop can manage the job with this algorithmn anyways. By 
>> pure naive math it should be:
>> i explicily assigned 10GB memory to each JVM on each node and set 
>> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops 
>> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large 
>> datastructures cause the GC to throw exceptions). This would naively 
>> make 18 slave machines with 10GB each resulting in an overall memory 
>> of 180GB - three times as much as needed... i would think. So if the 
>> Partitioner distributes them just about equally to all nodes i should 
>> not run into any errors, do i?
>>
>> Can anybody help me with this issue?
>>
>> Best regards,
>> Elmar
>>
>>
>
>


Mime
View raw message