Return-Path: Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: (qmail 50307 invoked from network); 7 Jan 2011 12:44:22 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 7 Jan 2011 12:44:22 -0000 Received: (qmail 75198 invoked by uid 500); 7 Jan 2011 12:44:19 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 75103 invoked by uid 500); 7 Jan 2011 12:44:19 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 75095 invoked by uid 99); 7 Jan 2011 12:44:18 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jan 2011 12:44:18 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=10.0 tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [207.97.245.131] (HELO smtp131.iad.emailsrvr.com) (207.97.245.131) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 07 Jan 2011 12:44:12 +0000 Received: from localhost (localhost.localdomain [127.0.0.1]) by smtp43.relay.iad1a.emailsrvr.com (SMTP Server) with ESMTP id 1BD8A2D02CE; Fri, 7 Jan 2011 07:43:51 -0500 (EST) X-Virus-Scanned: OK Received: by smtp43.relay.iad1a.emailsrvr.com (Authenticated sender: adarsh.sharma-AT-orkash.com) with ESMTPSA id BDB4E2D07A4 for ; Fri, 7 Jan 2011 07:43:49 -0500 (EST) Message-ID: <4D270B71.3040001@orkash.com> Date: Fri, 07 Jan 2011 18:17:45 +0530 From: Adarsh Sharma User-Agent: Thunderbird 2.0.0.22 (X11/20090625) MIME-Version: 1.0 To: common-user@hadoop.apache.org Subject: Too-many fetch failure Reduce Error Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Dear all, I am researching about the below error and could not able to find the reason : Data Size : 3.4 GB Hadoop-0.20.0 hadoop@ws32-test-lin:~/project/hadoop-0.20.2$ bin/hadoop jar hadoop-0.20.2-examples.jar wordcount /user/hadoop/page_content.txt page_content_output.txt 11/01/07 16:11:14 INFO input.FileInputFormat: Total input paths to process : 1 11/01/07 16:11:15 INFO mapred.JobClient: Running job: job_201101071129_0001 11/01/07 16:11:16 INFO mapred.JobClient: map 0% reduce 0% 11/01/07 16:11:41 INFO mapred.JobClient: map 1% reduce 0% 11/01/07 16:11:45 INFO mapred.JobClient: map 2% reduce 0% 11/01/07 16:11:48 INFO mapred.JobClient: map 3% reduce 0% 11/01/07 16:11:52 INFO mapred.JobClient: map 4% reduce 0% 11/01/07 16:11:56 INFO mapred.JobClient: map 5% reduce 0% 11/01/07 16:12:00 INFO mapred.JobClient: map 6% reduce 0% 11/01/07 16:12:05 INFO mapred.JobClient: map 7% reduce 0% 11/01/07 16:12:08 INFO mapred.JobClient: map 8% reduce 0% 11/01/07 16:12:11 INFO mapred.JobClient: map 9% reduce 0% 11/01/07 16:12:14 INFO mapred.JobClient: map 10% reduce 0% 11/01/07 16:12:17 INFO mapred.JobClient: map 11% reduce 0% 11/01/07 16:12:21 INFO mapred.JobClient: map 12% reduce 0% 11/01/07 16:12:24 INFO mapred.JobClient: map 13% reduce 0% 11/01/07 16:12:27 INFO mapred.JobClient: map 14% reduce 0% 11/01/07 16:12:30 INFO mapred.JobClient: map 15% reduce 0% 11/01/07 16:12:33 INFO mapred.JobClient: map 16% reduce 0% 11/01/07 16:12:36 INFO mapred.JobClient: map 17% reduce 0% 11/01/07 16:12:40 INFO mapred.JobClient: map 18% reduce 0% 11/01/07 16:12:45 INFO mapred.JobClient: map 19% reduce 0% 11/01/07 16:12:48 INFO mapred.JobClient: map 20% reduce 0% 11/01/07 16:12:54 INFO mapred.JobClient: map 21% reduce 0% 11/01/07 16:13:00 INFO mapred.JobClient: map 22% reduce 0% 11/01/07 16:13:04 INFO mapred.JobClient: map 22% reduce 1% 11/01/07 16:13:13 INFO mapred.JobClient: map 23% reduce 1% 11/01/07 16:13:19 INFO mapred.JobClient: map 24% reduce 1% 11/01/07 16:13:25 INFO mapred.JobClient: map 25% reduce 1% 11/01/07 16:13:30 INFO mapred.JobClient: map 26% reduce 1% 11/01/07 16:13:34 INFO mapred.JobClient: map 26% reduce 3% 11/01/07 16:13:36 INFO mapred.JobClient: map 27% reduce 3% 11/01/07 16:13:37 INFO mapred.JobClient: map 27% reduce 4% 11/01/07 16:13:39 INFO mapred.JobClient: map 28% reduce 4% 11/01/07 16:13:43 INFO mapred.JobClient: map 29% reduce 4% 11/01/07 16:13:46 INFO mapred.JobClient: map 30% reduce 4% 11/01/07 16:13:49 INFO mapred.JobClient: map 31% reduce 4% 11/01/07 16:13:52 INFO mapred.JobClient: map 32% reduce 4% 11/01/07 16:13:55 INFO mapred.JobClient: map 33% reduce 4% 11/01/07 16:13:58 INFO mapred.JobClient: map 34% reduce 4% 11/01/07 16:14:02 INFO mapred.JobClient: map 35% reduce 4% 11/01/07 16:14:05 INFO mapred.JobClient: map 36% reduce 4% 11/01/07 16:14:08 INFO mapred.JobClient: map 37% reduce 4% 11/01/07 16:14:11 INFO mapred.JobClient: map 38% reduce 4% 11/01/07 16:14:15 INFO mapred.JobClient: map 39% reduce 4% 11/01/07 16:14:19 INFO mapred.JobClient: map 40% reduce 4% 11/01/07 16:14:20 INFO mapred.JobClient: map 40% reduce 5% 11/01/07 16:14:25 INFO mapred.JobClient: map 41% reduce 5% 11/01/07 16:14:32 INFO mapred.JobClient: map 42% reduce 5% 11/01/07 16:14:38 INFO mapred.JobClient: map 43% reduce 5% 11/01/07 16:14:41 INFO mapred.JobClient: map 43% reduce 6% 11/01/07 16:14:43 INFO mapred.JobClient: map 44% reduce 6% 11/01/07 16:14:47 INFO mapred.JobClient: map 45% reduce 6% 11/01/07 16:14:50 INFO mapred.JobClient: map 46% reduce 6% 11/01/07 16:14:54 INFO mapred.JobClient: map 47% reduce 7% 11/01/07 16:14:59 INFO mapred.JobClient: map 48% reduce 7% 11/01/07 16:15:02 INFO mapred.JobClient: map 49% reduce 7% 11/01/07 16:15:05 INFO mapred.JobClient: map 50% reduce 7% 11/01/07 16:15:11 INFO mapred.JobClient: map 51% reduce 7% 11/01/07 16:15:14 INFO mapred.JobClient: map 52% reduce 7% 11/01/07 16:15:16 INFO mapred.JobClient: map 52% reduce 8% 11/01/07 16:15:20 INFO mapred.JobClient: map 53% reduce 8% 11/01/07 16:15:25 INFO mapred.JobClient: map 54% reduce 8% 11/01/07 16:15:29 INFO mapred.JobClient: map 55% reduce 8% 11/01/07 16:15:31 INFO mapred.JobClient: map 55% reduce 9% 11/01/07 16:15:33 INFO mapred.JobClient: map 56% reduce 9% 11/01/07 16:15:38 INFO mapred.JobClient: map 57% reduce 9% 11/01/07 16:15:42 INFO mapred.JobClient: map 58% reduce 9% 11/01/07 16:15:43 INFO mapred.JobClient: map 58% reduce 10% 11/01/07 16:15:46 INFO mapred.JobClient: map 59% reduce 10% 11/01/07 16:15:49 INFO mapred.JobClient: map 60% reduce 10% 11/01/07 16:15:53 INFO mapred.JobClient: map 61% reduce 10% 11/01/07 16:15:56 INFO mapred.JobClient: map 62% reduce 10% 11/01/07 16:16:00 INFO mapred.JobClient: map 63% reduce 10% 11/01/07 16:16:06 INFO mapred.JobClient: map 64% reduce 10% 11/01/07 16:16:10 INFO mapred.JobClient: map 65% reduce 10% 11/01/07 16:16:15 INFO mapred.JobClient: map 66% reduce 10% 11/01/07 16:16:18 INFO mapred.JobClient: map 67% reduce 10% 11/01/07 16:16:19 INFO mapred.JobClient: map 67% reduce 12% 11/01/07 16:16:21 INFO mapred.JobClient: map 68% reduce 12% 11/01/07 16:16:25 INFO mapred.JobClient: map 69% reduce 12% 11/01/07 16:16:28 INFO mapred.JobClient: map 70% reduce 12% 11/01/07 16:16:31 INFO mapred.JobClient: map 71% reduce 12% 11/01/07 16:16:35 INFO mapred.JobClient: map 72% reduce 12% 11/01/07 16:16:38 INFO mapred.JobClient: map 73% reduce 12% 11/01/07 16:16:43 INFO mapred.JobClient: map 74% reduce 12% 11/01/07 16:16:47 INFO mapred.JobClient: map 75% reduce 12% 11/01/07 16:16:53 INFO mapred.JobClient: map 75% reduce 13% 11/01/07 16:16:54 INFO mapred.JobClient: map 76% reduce 13% 11/01/07 16:17:01 INFO mapred.JobClient: map 77% reduce 13% 11/01/07 16:17:03 INFO mapred.JobClient: map 77% reduce 14% 11/01/07 16:17:06 INFO mapred.JobClient: map 78% reduce 15% 11/01/07 16:17:11 INFO mapred.JobClient: map 79% reduce 15% 11/01/07 16:17:15 INFO mapred.JobClient: map 80% reduce 15% 11/01/07 16:17:23 INFO mapred.JobClient: map 81% reduce 15% 11/01/07 16:17:27 INFO mapred.JobClient: map 82% reduce 15% 11/01/07 16:17:31 INFO mapred.JobClient: map 83% reduce 15% 11/01/07 16:17:33 INFO mapred.JobClient: map 84% reduce 15% 11/01/07 16:17:38 INFO mapred.JobClient: map 85% reduce 15% 11/01/07 16:17:41 INFO mapred.JobClient: map 86% reduce 15% 11/01/07 16:17:45 INFO mapred.JobClient: map 87% reduce 15% 11/01/07 16:17:50 INFO mapred.JobClient: map 88% reduce 15% 11/01/07 16:17:54 INFO mapred.JobClient: map 89% reduce 15% 11/01/07 16:18:00 INFO mapred.JobClient: map 90% reduce 15% 11/01/07 16:18:02 INFO mapred.JobClient: map 90% reduce 16% 11/01/07 16:18:03 INFO mapred.JobClient: map 91% reduce 16% 11/01/07 16:18:06 INFO mapred.JobClient: map 92% reduce 16% 11/01/07 16:18:09 INFO mapred.JobClient: map 93% reduce 16% 11/01/07 16:18:14 INFO mapred.JobClient: map 94% reduce 16% 11/01/07 16:18:17 INFO mapred.JobClient: map 95% reduce 16% 11/01/07 16:18:21 INFO mapred.JobClient: map 96% reduce 16% 11/01/07 16:18:27 INFO mapred.JobClient: map 97% reduce 16% 11/01/07 16:18:29 INFO mapred.JobClient: map 97% reduce 17% 11/01/07 16:18:35 INFO mapred.JobClient: map 98% reduce 17% 11/01/07 16:18:48 INFO mapred.JobClient: map 99% reduce 18% 11/01/07 16:18:55 INFO mapred.JobClient: map 99% reduce 19% 11/01/07 16:19:12 INFO mapred.JobClient: map 100% reduce 19% 11/01/07 16:19:13 INFO mapred.JobClient: map 100% reduce 20% 11/01/07 16:19:34 INFO mapred.JobClient: map 100% reduce 21% 11/01/07 16:22:54 INFO mapred.JobClient: Task Id : attempt_201101071129_0001_m_000012_0, Status : FAILED Too many fetch-failures 11/01/07 16:22:54 WARN mapred.JobClient: Error reading task outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000012_0&filter=stdout 11/01/07 16:22:54 WARN mapred.JobClient: Error reading task outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000012_0&filter=stderr 11/01/07 16:22:58 INFO mapred.JobClient: map 98% reduce 21% 11/01/07 16:23:16 INFO mapred.JobClient: map 99% reduce 21% 11/01/07 16:23:34 INFO mapred.JobClient: map 100% reduce 21% 11/01/07 16:25:28 INFO mapred.JobClient: map 100% reduce 23% 11/01/07 16:35:17 INFO mapred.JobClient: Task Id : attempt_201101071129_0001_m_000011_0, Status : FAILED Too many fetch-failures 11/01/07 16:35:17 WARN mapred.JobClient: Error reading task outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000011_0&filter=stdout 11/01/07 16:35:17 WARN mapred.JobClient: Error reading task outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000011_0&filter=stderr 11/01/07 16:35:21 INFO mapred.JobClient: map 98% reduce 23% 11/01/07 16:35:39 INFO mapred.JobClient: map 99% reduce 23% 11/01/07 16:35:54 INFO mapred.JobClient: map 100% reduce 23% 11/01/07 16:37:49 INFO mapred.JobClient: map 100% reduce 24% 11/01/07 16:47:32 INFO mapred.JobClient: Task Id : attempt_201101071129_0001_m_000015_0, Status : FAILED Too many fetch-failures 11/01/07 16:47:32 WARN mapred.JobClient: Error reading task outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000015_0&filter=stdout 11/01/07 16:47:32 WARN mapred.JobClient: Error reading task outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000015_0&filter=stderr 11/01/07 16:47:36 INFO mapred.JobClient: map 98% reduce 24% 11/01/07 16:47:54 INFO mapred.JobClient: map 99% reduce 24% 11/01/07 16:48:12 INFO mapred.JobClient: map 100% reduce 24% 11/01/07 16:50:03 INFO mapred.JobClient: map 100% reduce 25% 11/01/07 16:59:48 INFO mapred.JobClient: Task Id : attempt_201101071129_0001_m_000019_0, Status : FAILED Too many fetch-failures 11/01/07 16:59:48 WARN mapred.JobClient: Error reading task outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000019_0&filter=stdout 11/01/07 16:59:48 WARN mapred.JobClient: Error reading task outputhttp://hadoop2:50060/tasklog?plaintext=true&taskid=attempt_201101071129_0001_m_000019_0&filter=stderr 11/01/07 16:59:52 INFO mapred.JobClient: map 98% reduce 25% 11/01/07 17:00:10 INFO mapred.JobClient: map 99% reduce 25% 11/01/07 17:00:26 INFO mapred.JobClient: map 100% reduce 25% 11/01/07 17:00:38 INFO mapred.JobClient: map 100% reduce 26% Would anyone Please send me the reason as I am interested to find the cause to clear my understanding. I configured properly Hadoop on 2 clusters. I am sure there is no configuration problem because in my other cluster it runs fine. One Cluster is on Standalone Servers ( 4 nodes ). Job is executed successfully. Other Cluster is on VM's (Cloud) . My tasktracker log says this 2011-01-07 16:20:23,234 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201101071129_0001_m_000012_0,0) failed : org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201101071129_0001/attempt_201101071129_0001_m_000012_0/output/file.out.index in any of the configured local directories at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:324) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522) 2011-01-07 16:20:23,236 WARN org.apache.hadoop.mapred.TaskTracker: Unknown child with bad map output: attempt_201101071129_0001_m_000012_0. Ignored. 2011-01-07 16:20:23,239 INFO org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 172.16.1.3:50060, dest: 172.16.1.5:47135, bytes: 0, op: MAPRED_SHUFFLE, cliID: attempt_201101071129_0001_m_000012_0 2011-01-07 16:20:23,239 WARN org.mortbay.log: /mapOutput: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201101071129_0001/attempt_201101071129_0001_m_000012_0/output/file.out.index in any of the configured local directories 2011-01-07 16:22:53,266 WARN org.apache.hadoop.mapred.TaskTracker: getMapOutput(attempt_201101071129_0001_m_000012_0,0) failed : org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/jobcache/job_201101071129_0001/attempt_201101071129_0001_m_000012_0/output/file.out.index in any of the configured local directories at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:138) at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2887) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:324) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522) Let's have some discussion. Thanks & Regards Adarsh Sharma