hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Björn-Elmar Macek <...@cs.uni-kassel.de>
Subject Re: mortbay, huge files and the ulimit
Date Thu, 30 Aug 2012 10:27:22 GMT
Hi,

is there anybody, who knows more about this issue: it has already been 
recently marked here:
https://issues.apache.org/jira/browse/MAPREDUCE-5

I really want to do something about it, if i knew how... i tried so many 
different setup parameters and JVM options and nothing did the trick...

It was opened 2009 and gets still updated (2012): I mean - this does NOT 
look OK, does it? The whole TT seems like a big time waster:

2012-08-30 12:05:22,582 INFO 
org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
141.51.205.130:35497, dest: 141.51.205.126:39586, bytes: 146086410, op: 
MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000189_0, duration: 
5456122852
2012-08-30 12:05:56,275 INFO org.mortbay.log: 
org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
injecting delay2 times
2012-08-30 12:05:56,276 INFO org.mortbay.log: 
org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
recreating selector 2 times, canceled keys 70 times
2012-08-30 12:08:39,236 WARN org.apache.hadoop.mapred.TaskTracker: 
getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
org.mortbay.jetty.EofException
     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
     at 
org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
     at 
org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
     at 
org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
     at 
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
     at 
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
     at 
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
     at 
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
     at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
     at 
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
     at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
     at 
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
     at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
     at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
     at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
     at 
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
     at 
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
     at 
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
     at org.mortbay.jetty.Server.handle(Server.java:326)
     at 
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
     at 
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
     at 
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
     at 
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.io.IOException: Broken pipe
     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
     at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
     at 
org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
     ... 27 more

2012-08-30 12:08:39,236 WARN org.mortbay.log: Committed before 410 
getMapOutput(attempt_201208301039_0001_m_000204_0,0) failed :
org.mortbay.jetty.EofException
     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
     at 
org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
     at 
org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
     at 
org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
     at 
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
     at 
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
     at 
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
     at 
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
     at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
     at 
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
     at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
     at 
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
     at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
     at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
     at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
     at 
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
     at 
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
     at 
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
     at org.mortbay.jetty.Server.handle(Server.java:326)
     at 
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
     at 
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
     at 
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
     at 
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.io.IOException: Broken pipe
     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
     at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
     at 
org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
     ... 27 more

2012-08-30 12:08:39,236 INFO 
org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
141.51.205.130:35497, dest: 141.51.205.126:39614, bytes: 65536, op: 
MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000204_0, duration: 
3209410
2012-08-30 12:08:39,237 ERROR org.mortbay.log: /mapOutput
java.lang.IllegalStateException: Committed
     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
     at org.mortbay.jetty.Response.sendError(Response.java:240)
     at 
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
     at 
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
     at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
     at 
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
     at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
     at 
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
     at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
     at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
     at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
     at 
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
     at 
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
     at 
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
     at org.mortbay.jetty.Server.handle(Server.java:326)
     at 
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
     at 
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
     at 
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
     at 
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2012-08-30 12:10:27,109 INFO 
org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
141.51.205.130:35497, dest: 141.51.205.126:39618, bytes: 145730308, op: 
MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000204_0, duration: 
5686096517
2012-08-30 12:14:29,739 WARN org.apache.hadoop.mapred.TaskTracker: 
getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
org.mortbay.jetty.EofException
     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
     at 
org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
     at 
org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
     at 
org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
     at 
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
     at 
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
     at 
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
     at 
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
     at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
     at 
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
     at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
     at 
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
     at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
     at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
     at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
     at 
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
     at 
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
     at 
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
     at org.mortbay.jetty.Server.handle(Server.java:326)
     at 
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
     at 
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
     at 
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
     at 
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.io.IOException: Broken pipe
     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
     at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
     at 
org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
     ... 27 more

2012-08-30 12:14:29,740 WARN org.mortbay.log: Committed before 410 
getMapOutput(attempt_201208301039_0001_m_000236_0,0) failed :
org.mortbay.jetty.EofException
     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
     at 
org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
     at 
org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
     at 
org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
     at 
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
     at 
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
     at 
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
     at 
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
     at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
     at 
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
     at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
     at 
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
     at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
     at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
     at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
     at 
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
     at 
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
     at 
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
     at org.mortbay.jetty.Server.handle(Server.java:326)
     at 
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
     at 
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
     at 
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
     at 
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.io.IOException: Broken pipe
     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
     at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
     at 
org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
     ... 27 more

2012-08-30 12:14:29,740 INFO 
org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
141.51.205.130:35497, dest: 141.51.205.126:39648, bytes: 65536, op: 
MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000236_0, duration: 
3746212
2012-08-30 12:14:29,740 ERROR org.mortbay.log: /mapOutput
java.lang.IllegalStateException: Committed
     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
     at org.mortbay.jetty.Response.sendError(Response.java:240)
     at 
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
     at 
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
     at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
     at 
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
     at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
     at 
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
     at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
     at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
     at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
     at 
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
     at 
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
     at 
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
     at org.mortbay.jetty.Server.handle(Server.java:326)
     at 
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
     at 
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
     at 
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
     at 
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2012-08-30 12:16:28,503 INFO 
org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
141.51.205.130:35497, dest: 141.51.205.126:39652, bytes: 145506420, op: 
MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000236_0, duration: 
7793391393
2012-08-30 12:16:59,421 INFO org.mortbay.log: 
org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
injecting delay1 times
2012-08-30 12:16:59,422 INFO org.mortbay.log: 
org.mortbay.io.nio.SelectorManager$SelectSet@6bfd67ba JVM BUG(s) - 
recreating selector 1 times, canceled keys 69 times
2012-08-30 12:20:01,203 WARN org.apache.hadoop.mapred.TaskTracker: 
getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
org.mortbay.jetty.EofException
     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
     at 
org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
     at 
org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
     at 
org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
     at 
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
     at 
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
     at 
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
     at 
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
     at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
     at 
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
     at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
     at 
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
     at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
     at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
     at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
     at 
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
     at 
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
     at 
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
     at org.mortbay.jetty.Server.handle(Server.java:326)
     at 
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
     at 
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
     at 
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
     at 
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.io.IOException: Broken pipe
     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
     at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
     at 
org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
     ... 27 more

2012-08-30 12:20:01,205 WARN org.mortbay.log: Committed before 410 
getMapOutput(attempt_201208301039_0001_m_000241_0,0) failed :
org.mortbay.jetty.EofException
     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791)
     at 
org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551)
     at 
org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572)
     at 
org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012)
     at 
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651)
     at 
org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580)
     at 
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3916)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
     at 
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
     at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
     at 
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
     at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
     at 
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
     at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
     at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
     at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
     at 
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
     at 
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
     at 
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
     at org.mortbay.jetty.Server.handle(Server.java:326)
     at 
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
     at 
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
     at 
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
     at 
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.io.IOException: Broken pipe
     at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
     at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
     at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
     at sun.nio.ch.IOUtil.write(IOUtil.java:60)
     at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
     at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:170)
     at 
org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
     at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:725)
     ... 27 more

2012-08-30 12:20:01,205 INFO 
org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 
141.51.205.130:35497, dest: 141.51.205.126:39683, bytes: 65536, op: 
MAPRED_SHUFFLE, cliID: attempt_201208301039_0001_m_000241_0, duration: 
5405031
2012-08-30 12:20:01,205 ERROR org.mortbay.log: /mapOutput
java.lang.IllegalStateException: Committed
     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
     at org.mortbay.jetty.Response.sendError(Response.java:240)
     at 
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
     at 
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
     at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
     at 
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
     at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
     at 
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
     at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
     at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
     at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
     at 
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
     at 
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
     at 
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
     at org.mortbay.jetty.Server.handle(Server.java:326)
     at 
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
     at 
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
     at 
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
     at 
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Am 29.08.12 15:53, schrieb Björn-Elmar Macek:
> Hi there,
>
> i am currently running a job where i selfjoin a 63 gigabyte big csv 
> file on 20 physically distinct nodes with 15GB each:
>
> While the mapping works just fine and is low cost, the reducer does 
> the main work: holding a hashmap with elements to join with and 
> finding join tuples for evry incoming key-value-pair.
>
> The jobs works perfectly on small files with 2 gigabytes, but starts 
> to get "unstable" as the file size goes up: this becomes evident with 
> a look into the tasktracker's logs saying:
>
> ERROR org.mortbay.log: /mapOutput
> java.lang.IllegalStateException: Committed
>     at org.mortbay.jetty.Response.resetBuffer(Response.java:1023)
>     at org.mortbay.jetty.Response.sendError(Response.java:240)
>     at 
> org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3945)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>     at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>     at 
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>     at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>     at 
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
>     at 
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>     at 
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>     at 
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>     at 
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>     at 
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>     at 
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>     at 
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>     at 
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>     at org.mortbay.jetty.Server.handle(Server.java:326)
>     at 
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>     at 
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>     at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
>     at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
>     at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>     at 
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
>     at 
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
>
>
> And while it is no problem at the beginning of the reduce process, 
> where this happens only on a few nodes and rarely, it becomes crucial 
> as the progress rises. The reason for this (afaik from reading 
> articles), is that there are memory or file handle problems. I 
> addressed the memory problem by conitiously purging the map of 
> outdated elements evry 5 million processed key-value-pairs. And i set  
> mapred.child.ulimit to 100000000 (ulimit in the shell tells me it is 
> 400000000).
>
> Anyway i am still running into those mortbay errors and i start to 
> wonder, if hadoop can manage the job with this algorithmn anyways. By 
> pure naive math it should be:
> i explicily assigned 10GB memory to each JVM on each node and set 
> mapred.child.java.opts to "-Xmx10240m -XX:+UseCompressedOops 
> -XX:-UseGCOverheadLimit" (its a 64 bit environment and large 
> datastructures cause the GC to throw exceptions). This would naively 
> make 18 slave machines with 10GB each resulting in an overall memory 
> of 180GB - three times as much as needed... i would think. So if the 
> Partitioner distributes them just about equally to all nodes i should 
> not run into any errors, do i?
>
> Can anybody help me with this issue?
>
> Best regards,
> Elmar
>
>


Mime
View raw message