Return-Path: Delivered-To: apmail-lucene-hadoop-dev-archive@locus.apache.org Received: (qmail 35363 invoked from network); 12 Apr 2007 18:11:39 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 12 Apr 2007 18:11:39 -0000 Received: (qmail 11870 invoked by uid 500); 12 Apr 2007 18:11:43 -0000 Delivered-To: apmail-lucene-hadoop-dev-archive@lucene.apache.org Received: (qmail 11826 invoked by uid 500); 12 Apr 2007 18:11:43 -0000 Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-dev@lucene.apache.org Delivered-To: mailing list hadoop-dev@lucene.apache.org Received: (qmail 11803 invoked by uid 99); 12 Apr 2007 18:11:42 -0000 Received: from herse.apache.org (HELO herse.apache.org) (140.211.11.133) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Apr 2007 11:11:42 -0700 X-ASF-Spam-Status: No, hits=-100.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO brutus.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 12 Apr 2007 11:11:35 -0700 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id 8E3E9714071 for ; Thu, 12 Apr 2007 11:11:15 -0700 (PDT) Message-ID: <1533830.1176401475570.JavaMail.jira@brutus> Date: Thu, 12 Apr 2007 11:11:15 -0700 (PDT) From: "Hairong Kuang (JIRA)" To: hadoop-dev@lucene.apache.org Subject: [jira] Commented: (HADOOP-1176) Reduce hang on huge map output In-Reply-To: <1192251.1175108605436.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org [ https://issues.apache.org/jira/browse/HADOOP-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12488458 ] Hairong Kuang commented on HADOOP-1176: --------------------------------------- +1 test passed. > Reduce hang on huge map output > ------------------------------ > > Key: HADOOP-1176 > URL: https://issues.apache.org/jira/browse/HADOOP-1176 > Project: Hadoop > Issue Type: Bug > Components: mapred > Affects Versions: 0.12.2 > Reporter: Hairong Kuang > Assigned To: Arun C Murthy > Fix For: 0.13.0 > > Attachments: HADOOP-1176_20070405_1.patch, HADOOP-1176_20070406_2.patch, HADOOP-1176_20070411_3.patch, jetty-exception.log > > > I ran a job with one map and one reduce. The map succeeded, producing 3G output. The reducer hang on fetching the map output. The log shows the following error message: > WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(task_0401_m_000000_0,0) failed : > java.net.SocketException: Connection reset > at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96) > at java.net.SocketOutputStream.write(SocketOutputStream.java:136) > at org.mortbay.http.ChunkingOutputStream.bypassWrite(ChunkingOutputStream.java:151) > at org.mortbay.http.BufferedOutputStream.write(BufferedOutputStream.java:139) > at org.mortbay.http.HttpOutputStream.write(HttpOutputStream.java:423) > at org.mortbay.jetty.servlet.ServletOut.write(ServletOut.java:54) > at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1664) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:689) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) > at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427) > at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475) > at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567) > at org.mortbay.http.HttpContext.handle(HttpContext.java:1565) > at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635) > at org.mortbay.http.HttpContext.handle(HttpContext.java:1517) > at org.mortbay.http.HttpServer.service(HttpServer.java:954) > at org.mortbay.http.HttpConnection.service(HttpConnection.java:814) > at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981) > at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831) > at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244) > at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357) > at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534) > WARN /: /mapOutput?map=task_0401_m_000000_0&reduce=0: > java.lang.IllegalStateException: Committed > at org.mortbay.jetty.servlet.ServletHttpResponse.resetBuffer(ServletHttpResponse.java:212) > at org.mortbay.jetty.servlet.ServletHttpResponse.sendError(ServletHttpResponse.java:375) > at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1694) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:689) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:802) > at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427) > at org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475) > at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567) > at org.mortbay.http.HttpContext.handle(HttpContext.java:1565) > at org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635) > at org.mortbay.http.HttpContext.handle(HttpContext.java:1517) > at org.mortbay.http.HttpServer.service(HttpServer.java:954) > at org.mortbay.http.HttpConnection.service(HttpConnection.java:814) > at org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981) > at org.mortbay.http.HttpConnection.handle(HttpConnection.java:831) > at org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244) > at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357) > at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.