hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1176) Reduce hang on huge map output
Date Thu, 05 Apr 2007 17:41:32 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12487062
] 

Devaraj Das commented on HADOOP-1176:
-------------------------------------

I know the patch is preliminary. I just wanted to point out that we should use standard HTTP
headers as opposed to Jetty's constants. We should not rely on Jetty's constants as far as
possible especially when we are using the standard HTTP.header strings.

> Reduce hang on huge map output
> ------------------------------
>
>                 Key: HADOOP-1176
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1176
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.12.2
>            Reporter: Hairong Kuang
>         Assigned To: Arun C Murthy
>             Fix For: 0.13.0
>
>         Attachments: HADOOP-1176_20070405_1.patch
>
>
> I ran a job with one map and one reduce. The map succeeded, producing 3G output. The
reducer hang on fetching the map output. The log shows the following error message:
> WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(task_0401_m_000000_0,0) failed
:
> java.net.SocketException: Connection reset
>         at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
>         at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
>         at org.mortbay.http.ChunkingOutputStream.bypassWrite(ChunkingOutputStream.java:151)
>         at org.mortbay.http.BufferedOutputStream.write(BufferedOutputStream.java:139)
>         at org.mortbay.http.HttpOutputStream.write(HttpOutputStream.java:423)
>         at org.mortbay.jetty.servlet.ServletOut.write(ServletOut.java:54)
>         at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1664)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
>         at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
>         at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475)
>         at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
>         at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
>         at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635)
>         at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
>         at org.mortbay.http.HttpServer.service(HttpServer.java:954)
>         at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
>         at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
>         at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
>         at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
>         at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
>         at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
> WARN /: /mapOutput?map=task_0401_m_000000_0&reduce=0:
> java.lang.IllegalStateException: Committed
>         at org.mortbay.jetty.servlet.ServletHttpResponse.resetBuffer(ServletHttpResponse.java:212)
>         at org.mortbay.jetty.servlet.ServletHttpResponse.sendError(ServletHttpResponse.java:375)
>         at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1694)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
>         at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
>         at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475)
>         at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
>         at org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
>         at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635)
>         at org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
>         at org.mortbay.http.HttpServer.service(HttpServer.java:954)
>         at org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
>         at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
>         at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
>         at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
>         at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
>         at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message