hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "George Porter (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5865) Shuffle's getMapOutput() fails with EofException, followed by IllegalStateException
Date Mon, 18 May 2009 22:13:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12710515#action_12710515
] 

George Porter commented on HADOOP-5865:
---------------------------------------

The relevant sections of the TaskTracker log are:

2009-05-18 14:59:05,699 INFO org.apache.hadoop.mapred.JvmManager: No new JVM spawned for jobId/taskid:
job_200905181452_0002/attempt_200905181452_0002_m_000110_0. Attempting to reuse: jvm_200905181452_0002_m_-1082179983
2009-05-18 14:59:05,718 INFO org.apache.hadoop.mapred.TaskTracker: JVM with ID: jvm_200905181452_0002_m_-1082179983
given task: attempt_200905181452_0002_m_000110_0
2009-05-18 14:59:07,536 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_200905181452_0002_m_000010_0,0)
failed :
org.mortbay.jetty.EofException
	at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:787)
	at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:566)
	at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:946)
	at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:646)
	at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:577)
	at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2979)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:324)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
Caused by: java.io.IOException: Broken pipe
	at sun.nio.ch.FileDispatcher.write0(Native Method)
	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
	at sun.nio.ch.IOUtil.write(IOUtil.java:75)
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
	at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:169)
	at org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
	at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:721)
	... 23 more

2009-05-18 14:59:07,537 WARN org.mortbay.log: Committed before 410 getMapOutput(attempt_200905181452_0002_m_000010_0,0)
failed :
org.mortbay.jetty.EofException
	at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:787)
	at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:566)
	at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:946)
	at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:646)
	at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:577)
	at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2979)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:324)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
Caused by: java.io.IOException: Broken pipe
	at sun.nio.ch.FileDispatcher.write0(Native Method)
	at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
	at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
	at sun.nio.ch.IOUtil.write(IOUtil.java:75)
	at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
	at org.mortbay.io.nio.ChannelEndPoint.flush(ChannelEndPoint.java:169)
	at org.mortbay.io.nio.SelectChannelEndPoint.flush(SelectChannelEndPoint.java:221)
	at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:721)
	... 23 more

2009-05-18 14:59:07,537 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 0.0.0.0:0,
dest: 10.7.2.97:55398, bytes: 65536, op: MAPRED_SHUFFLE, cliID: attempt_200905181452_0002_m_000010_0
2009-05-18 14:59:07,552 ERROR org.mortbay.log: /mapOutput
java.lang.IllegalStateException: Committed
	at org.mortbay.jetty.Response.resetBuffer(Response.java:994)
	at org.mortbay.jetty.Response.sendError(Response.java:240)
	at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:3002)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:324)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
2009-05-18 14:59:08,479 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200905181452_0002_m_000101_0
1.0% hdfs://rep4150-90h:9000/sortinput/part-00006:671088640+134217728
2009-05-18 14:59:09,284 INFO org.apache.hadoop.mapred.TaskTracker: attempt_200905181452_0002_m_000101_0
1.0% hdfs://rep4150-90h:9000/sortinput/part-00006:671088640+134217728
2009-05-18 14:59:09,285 INFO org.apache.hadoop.mapred.TaskTracker: Task attempt_200905181452_0002_m_000101_0
is done.
2009-05-18 14:59:09,285 INFO org.apache.hadoop.mapred.TaskTracker: reported output size for
attempt_200905181452_0002_m_000101_0  was 0
2009-05-18 14:59:09,285 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free
slots : 1


> Shuffle's getMapOutput() fails with EofException, followed by IllegalStateException
> -----------------------------------------------------------------------------------
>
>                 Key: HADOOP-5865
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5865
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>         Environment: Sun Java 1.6.0_13, OpenSolaris, running on a SunFire 4150 (x64)
10 node cluster
>            Reporter: George Porter
>
> During the shuffle phase, I'm seeing a large sequence of the following actions:
> 1) WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_200905181452_0002_m_000010_0,0)
failed : org.mortbay.jetty.EofException
> 2) WARN org.mortbay.log: Committed before 410 getMapOutput(attempt_200905181452_0002_m_000010_0,0)
failed : org.mortbay.jetty.EofException
> 3) ERROR org.mortbay.log: /mapOutput java.lang.IllegalStateException: Committed
> The map phase completes with 100%, and then the reduce phase crawls along with the above
errors in each of the TaskTracker logs.  None of the tasktrackers get lost.  When I run non-data
jobs like the 'pi' test from the example jar, everything works fine.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message