Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 7701878CC for ; Wed, 28 Sep 2011 08:57:33 +0000 (UTC) Received: (qmail 49092 invoked by uid 500); 28 Sep 2011 08:57:30 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 49045 invoked by uid 500); 28 Sep 2011 08:57:30 -0000 Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-user@hadoop.apache.org Delivered-To: mailing list common-user@hadoop.apache.org Received: (qmail 49032 invoked by uid 99); 28 Sep 2011 08:57:29 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Sep 2011 08:57:29 +0000 X-ASF-Spam-Status: No, hits=4.0 required=5.0 tests=FREEMAIL_FROM,FREEMAIL_REPLY,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of abdoukamel@gmail.com designates 209.85.161.48 as permitted sender) Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Sep 2011 08:57:22 +0000 Received: by fxd23 with SMTP id 23so474772fxd.35 for ; Wed, 28 Sep 2011 01:57:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=JMoP9/pa15COYAc7YLFQr5ZHnDho1VdDJ6x+mR6EvGw=; b=YiKd8ICHtXlXwqWTdI5LcqiiRDeBqcdqQ0S4iL5zKQF/heHNL783ez2NsR0tL72EZG CWQM/9x+ON8WpymnZTmoOkUdTPypS6X0Ntfll2nZE3ZcspnOvC3e0rEDV9Eyc0zbb7/a w2z+njeUUwE4iWM4Aue6TxDEZ9uyq0F7UWqeE= MIME-Version: 1.0 Received: by 10.223.20.69 with SMTP id e5mr3438069fab.3.1317200220405; Wed, 28 Sep 2011 01:57:00 -0700 (PDT) Received: by 10.223.108.78 with HTTP; Wed, 28 Sep 2011 01:57:00 -0700 (PDT) In-Reply-To: References: <06006DDA5A27D541991944AC4117E7A9131D06EB@szxeml524-mbs.china.huawei.com> Date: Wed, 28 Sep 2011 11:57:00 +0300 Message-ID: Subject: Re: Too many fetch failures. Help! From: Abdelrahman Kamel To: common-user@hadoop.apache.org Content-Type: multipart/alternative; boundary=001517440f8a3c3fc704adfc9393 --001517440f8a3c3fc704adfc9393 Content-Type: text/plain; charset=ISO-8859-1 Thanks again. I have solved my problem by commenting out "127.0.1.1 hdslave" from /etc/hosts in all nodes. On Wed, Sep 28, 2011 at 10:15 AM, praveenesh kumar wrote: > Try commenting out "127.0.0.1 localhost" line from /etc/hosts in all our > systems. > > > On Tue, Sep 27, 2011 at 11:59 PM, Abdelrahman Kamel >wrote: > > > Thanks very much for all your fast replies. > > > > *Here is my terminal output:* > > > > hduser@hdmaster:/usr/local/hadoop$ bin/hadoop jar > > hadoop-0.20.2-examples.jar > > wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output > > 11/09/28 07:38:23 INFO input.FileInputFormat: Total input paths to > process > > : > > 5 > > 11/09/28 07:38:23 INFO mapred.JobClient: Running job: > job_201109280735_0002 > > 11/09/28 07:38:24 INFO mapred.JobClient: map 0% reduce 0% > > 11/09/28 07:38:42 INFO mapred.JobClient: map 20% reduce 0% > > 11/09/28 07:38:44 INFO mapred.JobClient: map 40% reduce 0% > > 11/09/28 07:38:45 INFO mapred.JobClient: map 60% reduce 0% > > 11/09/28 07:38:47 INFO mapred.JobClient: map 80% reduce 0% > > 11/09/28 07:38:51 INFO mapred.JobClient: map 100% reduce 0% > > 11/09/28 07:38:54 INFO mapred.JobClient: map 100% reduce 13% > > 11/09/28 07:39:01 INFO mapred.JobClient: map 100% reduce 20% > > > > *The terminal is stuck here.* > > > > *And here is my JobTracker log:* > > > > 2011-09-28 07:35:43,185 INFO org.apache.hadoop.mapred.JobTracker: > > STARTUP_MSG: > > /************************************************************ > > STARTUP_MSG: Starting JobTracker > > STARTUP_MSG: host = hdmaster/127.0.1.1 > > STARTUP_MSG: args = [] > > STARTUP_MSG: version = 0.20.2 > > STARTUP_MSG: build = > > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r > > 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010 > > ************************************************************/ > > 2011-09-28 07:35:43,256 INFO org.apache.hadoop.mapred.JobTracker: > Scheduler > > configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, > > limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1) > > 2011-09-28 07:35:43,310 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: > > Initializing RPC Metrics with hostName=JobTracker, port=54311 > > 2011-09-28 07:35:53,431 INFO org.mortbay.log: Logging to > > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via > > org.mortbay.log.Slf4jLog > > 2011-09-28 07:35:53,510 INFO org.apache.hadoop.http.HttpServer: Port > > returned by webServer.getConnectors()[0].getLocalPort() before open() is > > -1. > > Opening the listener on 50030 > > 2011-09-28 07:35:53,511 INFO org.apache.hadoop.http.HttpServer: > > listener.getLocalPort() returned 50030 > > webServer.getConnectors()[0].getLocalPort() returned 50030 > > 2011-09-28 07:35:53,511 INFO org.apache.hadoop.http.HttpServer: Jetty > bound > > to port 50030 > > 2011-09-28 07:35:53,511 INFO org.mortbay.log: jetty-6.1.14 > > 2011-09-28 07:35:53,816 INFO org.mortbay.log: Started > > SelectChannelConnector@0.0.0.0:50030 > > 2011-09-28 07:35:53,817 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: > > Initializing JVM Metrics with processName=JobTracker, sessionId= > > 2011-09-28 07:35:53,818 INFO org.apache.hadoop.mapred.JobTracker: > > JobTracker > > up at: 54311 > > 2011-09-28 07:35:53,818 INFO org.apache.hadoop.mapred.JobTracker: > > JobTracker > > webserver: 50030 > > 2011-09-28 07:35:53,926 INFO org.apache.hadoop.mapred.JobTracker: > Cleaning > > up the system directory > > 2011-09-28 07:35:53,930 INFO org.apache.hadoop.mapred.JobTracker: problem > > cleaning system directory: > hdfs://master:54310/app/hadoop/tmp/mapred/system > > org.apache.hadoop.ipc.RemoteException: > > org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete > > /app/hadoop/tmp/mapred/system. Name node is in safe mode. > > The ratio of reported blocks 0.0000 has not reached the threshold 0.9990. > > Safe mode will be turned off automatically. > > at > > > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700) > > at > > > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680) > > at > > org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:396) > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) > > > > at org.apache.hadoop.ipc.Client.call(Client.java:740) > > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > > at $Proxy4.delete(Unknown Source) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at > > > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > > at > > > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > > at $Proxy4.delete(Unknown Source) > > at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582) > > at > > > > > org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227) > > at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:1695) > > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183) > > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175) > > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702) > > 2011-09-28 07:36:03,934 INFO org.apache.hadoop.mapred.JobTracker: > Cleaning > > up the system directory > > 2011-09-28 07:36:03,935 INFO org.apache.hadoop.mapred.JobTracker: problem > > cleaning system directory: > hdfs://master:54310/app/hadoop/tmp/mapred/system > > org.apache.hadoop.ipc.RemoteException: > > org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete > > /app/hadoop/tmp/mapred/system. Name node is in safe mode. > > The ratio of reported blocks 0.0000 has not reached the threshold 0.9990. > > Safe mode will be turned off automatically. > > at > > > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700) > > at > > > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680) > > at > > org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:396) > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) > > > > at org.apache.hadoop.ipc.Client.call(Client.java:740) > > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > > at $Proxy4.delete(Unknown Source) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at > > > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > > at > > > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > > at $Proxy4.delete(Unknown Source) > > at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582) > > at > > > > > org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227) > > at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:1695) > > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183) > > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175) > > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702) > > 2011-09-28 07:36:13,942 INFO org.apache.hadoop.mapred.JobTracker: > Cleaning > > up the system directory > > 2011-09-28 07:36:13,944 INFO org.apache.hadoop.mapred.JobTracker: problem > > cleaning system directory: > hdfs://master:54310/app/hadoop/tmp/mapred/system > > org.apache.hadoop.ipc.RemoteException: > > org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete > > /app/hadoop/tmp/mapred/system. Name node is in safe mode. > > The ratio of reported blocks 1.0000 has reached the threshold 0.9990. > Safe > > mode will be turned off automatically in 25 seconds. > > at > > > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700) > > at > > > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680) > > at > > org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:396) > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) > > > > at org.apache.hadoop.ipc.Client.call(Client.java:740) > > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > > at $Proxy4.delete(Unknown Source) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at > > > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > > at > > > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > > at $Proxy4.delete(Unknown Source) > > at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582) > > at > > > > > org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227) > > at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:1695) > > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183) > > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175) > > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702) > > 2011-09-28 07:36:23,947 INFO org.apache.hadoop.mapred.JobTracker: > Cleaning > > up the system directory > > 2011-09-28 07:36:23,949 INFO org.apache.hadoop.mapred.JobTracker: problem > > cleaning system directory: > hdfs://master:54310/app/hadoop/tmp/mapred/system > > org.apache.hadoop.ipc.RemoteException: > > org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete > > /app/hadoop/tmp/mapred/system. Name node is in safe mode. > > The ratio of reported blocks 1.0000 has reached the threshold 0.9990. > Safe > > mode will be turned off automatically in 15 seconds. > > at > > > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700) > > at > > > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680) > > at > > org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:396) > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) > > > > at org.apache.hadoop.ipc.Client.call(Client.java:740) > > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > > at $Proxy4.delete(Unknown Source) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at > > > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > > at > > > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > > at $Proxy4.delete(Unknown Source) > > at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582) > > at > > > > > org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227) > > at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:1695) > > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183) > > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175) > > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702) > > 2011-09-28 07:36:33,952 INFO org.apache.hadoop.mapred.JobTracker: > Cleaning > > up the system directory > > 2011-09-28 07:36:33,954 INFO org.apache.hadoop.mapred.JobTracker: problem > > cleaning system directory: > hdfs://master:54310/app/hadoop/tmp/mapred/system > > org.apache.hadoop.ipc.RemoteException: > > org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete > > /app/hadoop/tmp/mapred/system. Name node is in safe mode. > > The ratio of reported blocks 1.0000 has reached the threshold 0.9990. > Safe > > mode will be turned off automatically in 5 seconds. > > at > > > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700) > > at > > > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680) > > at > > org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:396) > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) > > > > at org.apache.hadoop.ipc.Client.call(Client.java:740) > > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > > at $Proxy4.delete(Unknown Source) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at > > > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82) > > at > > > > > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59) > > at $Proxy4.delete(Unknown Source) > > at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582) > > at > > > > > org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227) > > at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:1695) > > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183) > > at > org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175) > > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702) > > 2011-09-28 07:37:09,922 INFO org.apache.hadoop.mapred.JobTracker: > Cleaning > > up the system directory > > 2011-09-28 07:37:10,107 INFO > > org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is > > inactive > > 2011-09-28 07:37:11,461 INFO org.apache.hadoop.ipc.Server: IPC Server > > Responder: starting > > 2011-09-28 07:37:11,462 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 0 on 54311: starting > > 2011-09-28 07:37:11,462 INFO org.apache.hadoop.ipc.Server: IPC Server > > listener on 54311: starting > > 2011-09-28 07:37:11,463 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 1 on 54311: starting > > 2011-09-28 07:37:11,463 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 2 on 54311: starting > > 2011-09-28 07:37:11,463 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 3 on 54311: starting > > 2011-09-28 07:37:11,470 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 4 on 54311: starting > > 2011-09-28 07:37:11,470 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 6 on 54311: starting > > 2011-09-28 07:37:11,470 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 7 on 54311: starting > > 2011-09-28 07:37:11,472 INFO org.apache.hadoop.mapred.JobTracker: > Starting > > RUNNING > > 2011-09-28 07:37:11,472 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 5 on 54311: starting > > 2011-09-28 07:37:11,472 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 8 on 54311: starting > > 2011-09-28 07:37:11,477 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 9 on 54311: starting > > 2011-09-28 07:37:11,577 INFO org.apache.hadoop.net.NetworkTopology: > Adding > > a > > new node: /default-rack/hdmaster > > 2011-09-28 07:37:43,188 INFO org.apache.hadoop.net.NetworkTopology: > Adding > > a > > new node: /default-rack/hdslave.ubuntu-domain > > 2011-09-28 07:38:23,832 INFO org.apache.hadoop.mapred.JobTracker: > > Initializing job_201109280735_0002 > > 2011-09-28 07:38:23,838 INFO org.apache.hadoop.mapred.JobInProgress: > > Initializing job_201109280735_0002 > > 2011-09-28 07:38:24,013 INFO org.apache.hadoop.mapred.JobInProgress: > Input > > size for job job_201109280735_0002 = 3430958. Number of splits = 5 > > 2011-09-28 07:38:24,013 INFO org.apache.hadoop.mapred.JobInProgress: > > tip:task_201109280735_0002_m_000000 has split on > > node:/default-rack/hdmaster > > 2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress: > > tip:task_201109280735_0002_m_000000 has split on > > node:/default-rack/hdslave.ubuntu-domain > > 2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress: > > tip:task_201109280735_0002_m_000001 has split on > > node:/default-rack/hdmaster > > 2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress: > > tip:task_201109280735_0002_m_000001 has split on > > node:/default-rack/hdslave.ubuntu-domain > > 2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress: > > tip:task_201109280735_0002_m_000002 has split on > > node:/default-rack/hdmaster > > 2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress: > > tip:task_201109280735_0002_m_000002 has split on > > node:/default-rack/hdslave.ubuntu-domain > > 2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress: > > tip:task_201109280735_0002_m_000003 has split on > > node:/default-rack/hdmaster > > 2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress: > > tip:task_201109280735_0002_m_000003 has split on > > node:/default-rack/hdslave.ubuntu-domain > > 2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress: > > tip:task_201109280735_0002_m_000004 has split on > > node:/default-rack/hdmaster > > 2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress: > > tip:task_201109280735_0002_m_000004 has split on > > node:/default-rack/hdslave.ubuntu-domain > > 2011-09-28 07:38:35,674 INFO org.apache.hadoop.mapred.JobTracker: Adding > > task 'attempt_201109280735_0002_m_000006_0' to tip > > task_201109280735_0002_m_000006, for tracker > > 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342' > > 2011-09-28 07:38:38,710 INFO org.apache.hadoop.mapred.JobInProgress: Task > > 'attempt_201109280735_0002_m_000006_0' has completed > > task_201109280735_0002_m_000006 successfully. > > 2011-09-28 07:38:38,714 INFO org.apache.hadoop.mapred.JobTracker: Adding > > task 'attempt_201109280735_0002_m_000000_0' to tip > > task_201109280735_0002_m_000000, for tracker > > 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342' > > 2011-09-28 07:38:38,717 INFO org.apache.hadoop.mapred.JobInProgress: > > Choosing data-local task task_201109280735_0002_m_000000 > > 2011-09-28 07:38:38,717 INFO org.apache.hadoop.mapred.JobTracker: Adding > > task 'attempt_201109280735_0002_m_000001_0' to tip > > task_201109280735_0002_m_000001, for tracker > > 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342' > > 2011-09-28 07:38:38,718 INFO org.apache.hadoop.mapred.JobInProgress: > > Choosing data-local task task_201109280735_0002_m_000001 > > 2011-09-28 07:38:40,233 INFO org.apache.hadoop.mapred.JobTracker: Adding > > task 'attempt_201109280735_0002_m_000002_0' to tip > > task_201109280735_0002_m_000002, for tracker 'tracker_hdmaster:localhost/ > > 127.0.0.1:54873' > > 2011-09-28 07:38:40,233 INFO org.apache.hadoop.mapred.JobInProgress: > > Choosing data-local task task_201109280735_0002_m_000002 > > 2011-09-28 07:38:40,233 INFO org.apache.hadoop.mapred.JobTracker: Adding > > task 'attempt_201109280735_0002_m_000003_0' to tip > > task_201109280735_0002_m_000003, for tracker 'tracker_hdmaster:localhost/ > > 127.0.0.1:54873' > > 2011-09-28 07:38:40,234 INFO org.apache.hadoop.mapred.JobInProgress: > > Choosing data-local task task_201109280735_0002_m_000003 > > 2011-09-28 07:38:41,724 INFO org.apache.hadoop.mapred.JobInProgress: Task > > 'attempt_201109280735_0002_m_000000_0' has completed > > task_201109280735_0002_m_000000 successfully. > > 2011-09-28 07:38:41,724 INFO org.apache.hadoop.mapred.ResourceEstimator: > > completedMapsUpdates:1 completedMapsInputSize:1423802 > > completedMapsOutputSize:466814 > > 2011-09-28 07:38:41,726 INFO org.apache.hadoop.mapred.JobTracker: Adding > > task 'attempt_201109280735_0002_m_000004_0' to tip > > task_201109280735_0002_m_000004, for tracker > > 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342' > > 2011-09-28 07:38:41,726 INFO org.apache.hadoop.mapred.JobInProgress: > > Choosing data-local task task_201109280735_0002_m_000004 > > 2011-09-28 07:38:41,730 INFO org.apache.hadoop.mapred.JobTracker: Adding > > task 'attempt_201109280735_0002_r_000000_0' to tip > > task_201109280735_0002_r_000000, for tracker > > 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342' > > 2011-09-28 07:38:43,272 INFO org.apache.hadoop.mapred.JobInProgress: Task > > 'attempt_201109280735_0002_m_000003_0' has completed > > task_201109280735_0002_m_000003 successfully. > > 2011-09-28 07:38:43,272 INFO org.apache.hadoop.mapred.ResourceEstimator: > > completedMapsUpdates:2 completedMapsInputSize:1817766 > > completedMapsOutputSize:719434 > > 2011-09-28 07:38:44,746 INFO org.apache.hadoop.mapred.JobInProgress: Task > > 'attempt_201109280735_0002_m_000001_0' has completed > > task_201109280735_0002_m_000001 successfully. > > 2011-09-28 07:38:44,746 INFO org.apache.hadoop.mapred.ResourceEstimator: > > completedMapsUpdates:3 completedMapsInputSize:2492333 > > completedMapsOutputSize:986447 > > 2011-09-28 07:38:46,284 INFO org.apache.hadoop.mapred.JobInProgress: Task > > 'attempt_201109280735_0002_m_000002_0' has completed > > task_201109280735_0002_m_000002 successfully. > > 2011-09-28 07:38:46,284 INFO org.apache.hadoop.mapred.ResourceEstimator: > > completedMapsUpdates:4 completedMapsInputSize:3087267 > > completedMapsOutputSize:1204101 > > 2011-09-28 07:38:50,874 INFO org.apache.hadoop.mapred.JobInProgress: Task > > 'attempt_201109280735_0002_m_000004_0' has completed > > task_201109280735_0002_m_000004 successfully. > > 2011-09-28 07:38:50,874 INFO org.apache.hadoop.mapred.ResourceEstimator: > > completedMapsUpdates:5 completedMapsInputSize:3430963 > > completedMapsOutputSize:1367489 > > 2011-09-28 07:39:06,502 INFO org.apache.hadoop.mapred.JobInProgress: > Failed > > fetch notification #1 for task attempt_201109280735_0002_m_000003_0 > > 2011-09-28 07:41:53,093 INFO org.apache.hadoop.mapred.JobInProgress: > Failed > > fetch notification #2 for task attempt_201109280735_0002_m_000003_0 > > 2011-09-28 07:47:06,741 INFO org.apache.hadoop.mapred.JobInProgress: > Failed > > fetch notification #3 for task attempt_201109280735_0002_m_000003_0 > > 2011-09-28 07:47:06,742 INFO org.apache.hadoop.mapred.JobInProgress: Too > > many fetch-failures for output of task: > > attempt_201109280735_0002_m_000003_0 > > ... killing it > > 2011-09-28 07:47:06,742 INFO org.apache.hadoop.mapred.TaskInProgress: > Error > > from attempt_201109280735_0002_m_000003_0: Too many fetch-failures > > 2011-09-28 07:47:06,743 INFO org.apache.hadoop.mapred.JobTracker: Adding > > task 'attempt_201109280735_0002_m_000003_1' to tip > > task_201109280735_0002_m_000003, for tracker > > 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342' > > 2011-09-28 07:47:06,743 INFO org.apache.hadoop.mapred.JobInProgress: > > Choosing data-local task task_201109280735_0002_m_000003 > > 2011-09-28 07:47:07,607 INFO org.apache.hadoop.mapred.JobTracker: Removed > > completed task 'attempt_201109280735_0002_m_000003_0' from > > 'tracker_hdmaster:localhost/127.0.0.1:54873' > > 2011-09-28 07:47:13,361 INFO org.apache.hadoop.mapred.JobInProgress: Task > > 'attempt_201109280735_0002_m_000003_1' has completed > > task_201109280735_0002_m_000003 successfully. > > 2011-09-28 07:47:13,361 INFO org.apache.hadoop.mapred.ResourceEstimator: > > completedMapsUpdates:6 completedMapsInputSize:3824927 > > completedMapsOutputSize:1620109 > > > > *And here is my TaskTracker #1 log:* > > > > 2011-09-28 07:35:44,452 INFO org.apache.hadoop.mapred.TaskTracker: > > STARTUP_MSG: > > /************************************************************ > > STARTUP_MSG: Starting TaskTracker > > STARTUP_MSG: host = hdmaster/127.0.1.1 > > STARTUP_MSG: args = [] > > STARTUP_MSG: version = 0.20.2 > > STARTUP_MSG: build = > > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r > > 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010 > > ************************************************************/ > > 2011-09-28 07:35:54,661 INFO org.mortbay.log: Logging to > > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via > > org.mortbay.log.Slf4jLog > > 2011-09-28 07:35:54,744 INFO org.apache.hadoop.http.HttpServer: Port > > returned by webServer.getConnectors()[0].getLocalPort() before open() is > > -1. > > Opening the listener on 50060 > > 2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer: > > listener.getLocalPort() returned 50060 > > webServer.getConnectors()[0].getLocalPort() returned 50060 > > 2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer: Jetty > bound > > to port 50060 > > 2011-09-28 07:35:54,750 INFO org.mortbay.log: jetty-6.1.14 > > 2011-09-28 07:35:55,025 INFO org.mortbay.log: Started > > SelectChannelConnector@0.0.0.0:50060 > > 2011-09-28 07:35:55,064 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: > > Initializing JVM Metrics with processName=TaskTracker, sessionId= > > 2011-09-28 07:35:55,082 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: > > Initializing RPC Metrics with hostName=TaskTracker, port=54873 > > 2011-09-28 07:35:55,127 INFO org.apache.hadoop.ipc.Server: IPC Server > > Responder: starting > > 2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker: > > TaskTracker up at: localhost/127.0.0.1:54873 > > 2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker: > Starting > > tracker tracker_hdmaster:localhost/127.0.0.1:54873 > > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 0 on 54873: starting > > 2011-09-28 07:35:55,131 INFO org.apache.hadoop.ipc.Server: IPC Server > > listener on 54873: starting > > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 1 on 54873: starting > > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 2 on 54873: starting > > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 3 on 54873: starting > > 2011-09-28 07:37:11,483 INFO org.apache.hadoop.mapred.TaskTracker: > Starting > > thread: Map-events fetcher for all reduce tasks on > > tracker_hdmaster:localhost/127.0.0.1:54873 > > 2011-09-28 07:37:11,484 INFO org.apache.hadoop.mapred.TaskTracker: Using > > MemoryCalculatorPlugin : > > org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@5a3923 > > 2011-09-28 07:37:11,487 WARN org.apache.hadoop.mapred.TaskTracker: > > TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is > > disabled. > > 2011-09-28 07:37:11,488 INFO org.apache.hadoop.mapred.IndexCache: > > IndexCache > > created with max memory = 10485760 > > 2011-09-28 07:38:40,269 INFO org.apache.hadoop.mapred.TaskTracker: > > LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000002_0 > > task's > > state:UNASSIGNED > > 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: > > LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000003_0 > > task's > > state:UNASSIGNED > > 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: Trying > > to > > launch : attempt_201109280735_0002_m_000002_0 > > 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: In > > TaskLauncher, current free slots : 2 and trying to launch > > attempt_201109280735_0002_m_000002_0 > > 2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: Trying > > to > > launch : attempt_201109280735_0002_m_000003_0 > > 2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: In > > TaskLauncher, current free slots : 1 and trying to launch > > attempt_201109280735_0002_m_000003_0 > > 2011-09-28 07:38:40,538 INFO org.apache.hadoop.mapred.JvmManager: In > > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_1909679810 > > 2011-09-28 07:38:40,540 INFO org.apache.hadoop.mapred.JvmManager: JVM > > Runner > > jvm_201109280735_0002_m_1909679810 spawned. > > 2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: In > > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-622388657 > > 2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: JVM > > Runner > > jvm_201109280735_0002_m_-622388657 spawned. > > 2011-09-28 07:38:41,046 INFO org.apache.hadoop.mapred.TaskTracker: JVM > with > > ID: jvm_201109280735_0002_m_1909679810 given task: > > attempt_201109280735_0002_m_000002_0 > > 2011-09-28 07:38:41,099 INFO org.apache.hadoop.mapred.TaskTracker: JVM > with > > ID: jvm_201109280735_0002_m_-622388657 given task: > > attempt_201109280735_0002_m_000003_0 > > 2011-09-28 07:38:43,264 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_m_000003_0 1.0% > > 2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker: Task > > attempt_201109280735_0002_m_000003_0 is done. > > 2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker: > reported > > output size for attempt_201109280735_0002_m_000003_0 was 0 > > 2011-09-28 07:38:43,266 INFO org.apache.hadoop.mapred.TaskTracker: > > addFreeSlot : current free slots : 1 > > 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_m_000002_0 1.0% > > 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker: Task > > attempt_201109280735_0002_m_000002_0 is done. > > 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker: > reported > > output size for attempt_201109280735_0002_m_000002_0 was 0 > > 2011-09-28 07:38:43,322 INFO org.apache.hadoop.mapred.TaskTracker: > > addFreeSlot : current free slots : 2 > > 2011-09-28 07:38:43,362 INFO org.apache.hadoop.mapred.JvmManager: JVM : > > jvm_201109280735_0002_m_1909679810 exited. Number of tasks it ran: 1 > > 2011-09-28 07:38:43,412 INFO org.apache.hadoop.mapred.JvmManager: JVM : > > jvm_201109280735_0002_m_-622388657 exited. Number of tasks it ran: 1 > > > > > > *And here is my TaskTracker #2 log:* > > > > 2011-09-28 07:35:44,452 INFO org.apache.hadoop.mapred.TaskTracker: > > STARTUP_MSG: > > /************************************************************ > > STARTUP_MSG: Starting TaskTracker > > STARTUP_MSG: host = hdmaster/127.0.1.1 > > STARTUP_MSG: args = [] > > STARTUP_MSG: version = 0.20.2 > > STARTUP_MSG: build = > > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r > > 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010 > > ************************************************************/ > > 2011-09-28 07:35:54,661 INFO org.mortbay.log: Logging to > > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via > > org.mortbay.log.Slf4jLog > > 2011-09-28 07:35:54,744 INFO org.apache.hadoop.http.HttpServer: Port > > returned by webServer.getConnectors()[0].getLocalPort() before open() is > > -1. > > Opening the listener on 50060 > > 2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer: > > listener.getLocalPort() returned 50060 > > webServer.getConnectors()[0].getLocalPort() returned 50060 > > 2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer: Jetty > bound > > to port 50060 > > 2011-09-28 07:35:54,750 INFO org.mortbay.log: jetty-6.1.14 > > 2011-09-28 07:35:55,025 INFO org.mortbay.log: Started > > SelectChannelConnector@0.0.0.0:50060 > > 2011-09-28 07:35:55,064 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: > > Initializing JVM Metrics with processName=TaskTracker, sessionId= > > 2011-09-28 07:35:55,082 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: > > Initializing RPC Metrics with hostName=TaskTracker, port=54873 > > 2011-09-28 07:35:55,127 INFO org.apache.hadoop.ipc.Server: IPC Server > > Responder: starting > > 2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker: > > TaskTracker up at: localhost/127.0.0.1:54873 > > 2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker: > Starting > > tracker tracker_hdmaster:localhost/127.0.0.1:54873 > > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 0 on 54873: starting > > 2011-09-28 07:35:55,131 INFO org.apache.hadoop.ipc.Server: IPC Server > > listener on 54873: starting > > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 1 on 54873: starting > > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 2 on 54873: starting > > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server > > handler 3 on 54873: starting > > 2011-09-28 07:37:11,483 INFO org.apache.hadoop.mapred.TaskTracker: > Starting > > thread: Map-events fetcher for all reduce tasks on > > tracker_hdmaster:localhost/127.0.0.1:54873 > > 2011-09-28 07:37:11,484 INFO org.apache.hadoop.mapred.TaskTracker: Using > > MemoryCalculatorPlugin : > > org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@5a3923 > > 2011-09-28 07:37:11,487 WARN org.apache.hadoop.mapred.TaskTracker: > > TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is > > disabled. > > 2011-09-28 07:37:11,488 INFO org.apache.hadoop.mapred.IndexCache: > > IndexCache > > created with max memory = 10485760 > > 2011-09-28 07:38:40,269 INFO org.apache.hadoop.mapred.TaskTracker: > > LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000002_0 > > task's > > state:UNASSIGNED > > 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: > > LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000003_0 > > task's > > state:UNASSIGNED > > 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: Trying > > to > > launch : attempt_201109280735_0002_m_000002_0 > > 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: In > > TaskLauncher, current free slots : 2 and trying to launch > > attempt_201109280735_0002_m_000002_0 > > 2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: Trying > > to > > launch : attempt_201109280735_0002_m_000003_0 > > 2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: In > > TaskLauncher, current free slots : 1 and trying to launch > > attempt_201109280735_0002_m_000003_0 > > 2011-09-28 07:38:40,538 INFO org.apache.hadoop.mapred.JvmManager: In > > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_1909679810 > > 2011-09-28 07:38:40,540 INFO org.apache.hadoop.mapred.JvmManager: JVM > > Runner > > jvm_201109280735_0002_m_1909679810 spawned. > > 2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: In > > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-622388657 > > 2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: JVM > > Runner > > jvm_201109280735_0002_m_-622388657 spawned. > > 2011-09-28 07:38:41,046 INFO org.apache.hadoop.mapred.TaskTracker: JVM > with > > ID: jvm_201109280735_0002_m_1909679810 given task: > > attempt_201109280735_0002_m_000002_0 > > 2011-09-28 07:38:41,099 INFO org.apache.hadoop.mapred.TaskTracker: JVM > with > > ID: jvm_201109280735_0002_m_-622388657 given task: > > attempt_201109280735_0002_m_000003_0 > > 2011-09-28 07:38:43,264 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_m_000003_0 1.0% > > 2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker: Task > > attempt_201109280735_0002_m_000003_0 is done. > > 2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker: > reported > > output size for attempt_201109280735_0002_m_000003_0 was 0 > > 2011-09-28 07:38:43,266 INFO org.apache.hadoop.mapred.TaskTracker: > > addFreeSlot : current free slots : 1 > > 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_m_000002_0 1.0% > > 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker: Task > > attempt_201109280735_0002_m_000002_0 is done. > > 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker: > reported > > output size for attempt_201109280735_0002_m_000002_0 was 0 > > 2011-09-28 07:38:43,322 INFO org.apache.hadoop.mapred.TaskTracker: > > addFreeSlot : current free slots : 2 > > 2011-09-28 07:38:43,362 INFO org.apache.hadoop.mapred.JvmManager: JVM : > > jvm_201109280735_0002_m_1909679810 exited. Number of tasks it ran: 1 > > 2011-09-28 07:38:43,412 INFO org.apache.hadoop.mapred.JvmManager: JVM : > > jvm_201109280735_0002_m_-622388657 exited. Number of tasks it ran: 1 > > 2011-09-28 07:36:10,343 INFO org.apache.hadoop.ipc.Client: Retrying > connect > > to server: master/10.0.0.1:54311. Already tried 0 time(s). > > 2011-09-28 07:36:31,346 INFO org.apache.hadoop.ipc.Client: Retrying > connect > > to server: master/10.0.0.1:54311. Already tried 1 time(s). > > 2011-09-28 07:36:31,581 INFO org.apache.hadoop.mapred.TaskTracker: > Starting > > thread: Map-events fetcher for all reduce tasks on > > tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342 > > 2011-09-28 07:36:31,583 INFO org.apache.hadoop.mapred.TaskTracker: Using > > MemoryCalculatorPlugin : > > org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@ce16ad > > 2011-09-28 07:36:31,588 WARN org.apache.hadoop.mapred.TaskTracker: > > TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is > > disabled. > > 2011-09-28 07:36:31,589 INFO org.apache.hadoop.mapred.IndexCache: > > IndexCache > > created with max memory = 10485760 > > 2011-09-28 07:37:35,032 INFO org.apache.hadoop.mapred.TaskTracker: > > LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000006_0 > > task's > > state:UNASSIGNED > > 2011-09-28 07:37:35,034 INFO org.apache.hadoop.mapred.TaskTracker: Trying > > to > > launch : attempt_201109280735_0002_m_000006_0 > > 2011-09-28 07:37:35,034 INFO org.apache.hadoop.mapred.TaskTracker: In > > TaskLauncher, current free slots : 2 and trying to launch > > attempt_201109280735_0002_m_000006_0 > > 2011-09-28 07:37:35,374 INFO org.apache.hadoop.mapred.JvmManager: In > > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_774270481 > > 2011-09-28 07:37:35,374 INFO org.apache.hadoop.mapred.JvmManager: JVM > > Runner > > jvm_201109280735_0002_m_774270481 spawned. > > 2011-09-28 07:37:35,773 INFO org.apache.hadoop.mapred.TaskTracker: JVM > with > > ID: jvm_201109280735_0002_m_774270481 given task: > > attempt_201109280735_0002_m_000006_0 > > 2011-09-28 07:37:36,667 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_m_000006_0 0.0% setup > > 2011-09-28 07:37:36,668 INFO org.apache.hadoop.mapred.TaskTracker: Task > > attempt_201109280735_0002_m_000006_0 is done. > > 2011-09-28 07:37:36,668 INFO org.apache.hadoop.mapred.TaskTracker: > reported > > output size for attempt_201109280735_0002_m_000006_0 was 0 > > 2011-09-28 07:37:36,670 INFO org.apache.hadoop.mapred.TaskTracker: > > addFreeSlot : current free slots : 2 > > 2011-09-28 07:37:36,818 INFO org.apache.hadoop.mapred.JvmManager: JVM : > > jvm_201109280735_0002_m_774270481 exited. Number of tasks it ran: 1 > > 2011-09-28 07:37:38,033 INFO org.apache.hadoop.mapred.TaskTracker: > > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find > > > > > taskTracker/jobcache/job_201109280735_0002/attempt_201109280735_0002_m_000006_0/output/file.out > > in any of the configured local directories > > 2011-09-28 07:37:38,045 INFO org.apache.hadoop.mapred.TaskTracker: > > LaunchTaskAction > > (registerTask): attempt_201109280735_0002_m_000000_0 task's > > state:UNASSIGNED > > 2011-09-28 07:37:38,045 INFO org.apache.hadoop.mapred.TaskTracker: > > LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000001_0 > > task's > > state:UNASSIGNED > > 2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker: Trying > > to > > launch : attempt_201109280735_0002_m_000000_0 > > 2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker: In > > TaskLauncher, current free slots : 2 and trying to launch > > attempt_201109280735_0002_m_000000_0 > > 2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker: > Received > > KillTaskAction for task: attempt_201109280735_0002_m_000006_0 > > 2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker: About > to > > purge task: attempt_201109280735_0002_m_000006_0 > > 2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskRunner: > > attempt_201109280735_0002_m_000006_0 done; removing files. > > 2011-09-28 07:37:38,047 INFO org.apache.hadoop.mapred.IndexCache: Map ID > > attempt_201109280735_0002_m_000006_0 not found in cache > > 2011-09-28 07:37:38,080 INFO org.apache.hadoop.mapred.TaskTracker: Trying > > to > > launch : attempt_201109280735_0002_m_000001_0 > > 2011-09-28 07:37:38,080 INFO org.apache.hadoop.mapred.TaskTracker: In > > TaskLauncher, current free slots : 1 and trying to launch > > attempt_201109280735_0002_m_000001_0 > > 2011-09-28 07:37:38,082 INFO org.apache.hadoop.mapred.JvmManager: In > > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-1878969571 > > 2011-09-28 07:37:38,086 INFO org.apache.hadoop.mapred.JvmManager: JVM > > Runner > > jvm_201109280735_0002_m_-1878969571 spawned. > > 2011-09-28 07:37:38,129 INFO org.apache.hadoop.mapred.JvmManager: In > > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-364970930 > > 2011-09-28 07:37:38,129 INFO org.apache.hadoop.mapred.JvmManager: JVM > > Runner > > jvm_201109280735_0002_m_-364970930 spawned. > > 2011-09-28 07:37:38,590 INFO org.apache.hadoop.mapred.TaskTracker: JVM > with > > ID: jvm_201109280735_0002_m_-364970930 given task: > > attempt_201109280735_0002_m_000001_0 > > 2011-09-28 07:37:38,604 INFO org.apache.hadoop.mapred.TaskTracker: JVM > with > > ID: jvm_201109280735_0002_m_-1878969571 given task: > > attempt_201109280735_0002_m_000000_0 > > 2011-09-28 07:37:40,644 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_m_000000_0 1.0% > > 2011-09-28 07:37:40,645 INFO org.apache.hadoop.mapred.TaskTracker: Task > > attempt_201109280735_0002_m_000000_0 is done. > > 2011-09-28 07:37:40,645 INFO org.apache.hadoop.mapred.TaskTracker: > reported > > output size for attempt_201109280735_0002_m_000000_0 was 0 > > 2011-09-28 07:37:40,645 INFO org.apache.hadoop.mapred.TaskTracker: > > addFreeSlot : current free slots : 1 > > 2011-09-28 07:37:40,697 INFO org.apache.hadoop.mapred.JvmManager: JVM : > > jvm_201109280735_0002_m_-1878969571 exited. Number of tasks it ran: 1 > > 2011-09-28 07:37:41,067 INFO org.apache.hadoop.mapred.TaskTracker: > > LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000004_0 > > task's > > state:UNASSIGNED > > 2011-09-28 07:37:41,067 INFO org.apache.hadoop.mapred.TaskTracker: Trying > > to > > launch : attempt_201109280735_0002_m_000004_0 > > 2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker: In > > TaskLauncher, current free slots : 1 and trying to launch > > attempt_201109280735_0002_m_000004_0 > > 2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker: > > LaunchTaskAction (registerTask): attempt_201109280735_0002_r_000000_0 > > task's > > state:UNASSIGNED > > 2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker: Trying > > to > > launch : attempt_201109280735_0002_r_000000_0 > > 2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker: In > > TaskLauncher, current free slots : 2 and trying to launch > > attempt_201109280735_0002_r_000000_0 > > 2011-09-28 07:37:41,118 INFO org.apache.hadoop.mapred.JvmManager: In > > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_531367236 > > 2011-09-28 07:37:41,118 INFO org.apache.hadoop.mapred.JvmManager: JVM > > Runner > > jvm_201109280735_0002_m_531367236 spawned. > > 2011-09-28 07:37:41,165 INFO org.apache.hadoop.mapred.JvmManager: In > > JvmRunner constructed JVM ID: jvm_201109280735_0002_r_774270481 > > 2011-09-28 07:37:41,166 INFO org.apache.hadoop.mapred.JvmManager: JVM > > Runner > > jvm_201109280735_0002_r_774270481 spawned. > > 2011-09-28 07:37:41,635 INFO org.apache.hadoop.mapred.TaskTracker: JVM > with > > ID: jvm_201109280735_0002_r_774270481 given task: > > attempt_201109280735_0002_r_000000_0 > > 2011-09-28 07:37:41,660 INFO org.apache.hadoop.mapred.TaskTracker: JVM > with > > ID: jvm_201109280735_0002_m_531367236 given task: > > attempt_201109280735_0002_m_000004_0 > > 2011-09-28 07:37:43,417 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_m_000001_0 1.0% > > 2011-09-28 07:37:43,418 INFO org.apache.hadoop.mapred.TaskTracker: Task > > attempt_201109280735_0002_m_000001_0 is done. > > 2011-09-28 07:37:43,418 INFO org.apache.hadoop.mapred.TaskTracker: > reported > > output size for attempt_201109280735_0002_m_000001_0 was 0 > > 2011-09-28 07:37:43,419 INFO org.apache.hadoop.mapred.TaskTracker: > > addFreeSlot : current free slots : 1 > > 2011-09-28 07:37:43,456 INFO org.apache.hadoop.mapred.JvmManager: JVM : > > jvm_201109280735_0002_m_-364970930 exited. Number of tasks it ran: 1 > > 2011-09-28 07:37:46,712 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_m_000004_0 1.0% > > 2011-09-28 07:37:46,713 INFO org.apache.hadoop.mapred.TaskTracker: Task > > attempt_201109280735_0002_m_000004_0 is done.2011-09-28 07:37:46,713 INFO > > org.apache.hadoop.mapred.TaskTracker: reported output size for > > attempt_201109280735_0002_m_000004_0 was 0 > > 2011-09-28 07:37:46,714 INFO org.apache.hadoop.mapred.TaskTracker: > > addFreeSlot : current free slots : 2 > > 2011-09-28 07:37:46,847 INFO org.apache.hadoop.mapred.JvmManager: JVM : > > jvm_201109280735_0002_m_531367236 exited. Number of tasks it ran: 1 > > 2011-09-28 07:37:47,276 INFO org.apache.hadoop.mapred.TaskTracker: Sent > out > > 466814 bytes for reduce: 0 from map: attempt_201109280735_0002_m_000000_0 > > given 466814/466810 > > 2011-09-28 07:37:47,276 INFO > > org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.1.1:50060, > > dest: 127.0.0.1:52662, bytes: 466814, op: MAPRED_SHUFFLE, cliID: > > attempt_201109280735_0002_m_000000_0 > > 2011-09-28 07:37:47,287 INFO org.apache.hadoop.mapred.TaskTracker: Sent > out > > 267013 bytes for reduce: 0 from map: attempt_201109280735_0002_m_000001_0 > > given 267013/267009 > > 2011-09-28 07:37:47,287 INFO > > org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.1.1:50060, > > dest: 127.0.0.1:52663, bytes: 267013, op: MAPRED_SHUFFLE, cliID: > > attempt_201109280735_0002_m_000001_0 > > 2011-09-28 07:37:48,128 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.13333334% reduce > copy (2 of 5 at > > 0.12 MB/s) > > > 2011-09-28 07:37:51,130 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.13333334% reduce > copy (2 of 5 at > > 0.12 MB/s) > > > 2011-09-28 07:37:51,176 INFO org.apache.hadoop.mapred.TaskTracker: Sent > out > > 163388 bytes for reduce: 0 from map: attempt_201109280735_0002_m_000004_0 > > given 163388/163384 > > 2011-09-28 07:37:51,176 INFO > > org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.1.1:50060, > > dest: 127.0.0.1:52664, bytes: 163388, op: MAPRED_SHUFFLE, cliID: > > attempt_201109280735_0002_m_000004_0 > > 2011-09-28 07:37:57,132 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:38:03,135 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:38:06,136 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:38:18,140 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:38:21,142 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:38:27,144 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:38:33,146 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:38:36,148 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) >2011-09-28 07:38:42,150 INFO > > org.apache.hadoop.mapred.TaskTracker: > attempt_201109280735_0002_r_000000_0 > > 0.20000002% reduce > copy (3 of 5 at 0.09 MB/s) > > > 2011-09-28 07:38:48,153 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:38:51,163 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:38:57,166 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:39:03,168 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:39:06,170 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:39:12,172 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:39:18,175 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:39:21,176 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:39:27,178 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:39:33,181 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:39:36,183 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:39:42,185 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:39:48,187 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:39:51,189 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:39:57,191 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:40:03,193 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:40:06,195 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:40:12,197 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:40:18,199 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:40:21,201 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 > > 0.20000002% reduce > copy (3 of 5 at 0.09 MB/s) > > > 2011-09-28 07:40:27,203 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:40:33,205 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:40:36,206 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:40:42,209 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:40:51,212 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:40:57,214 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:41:03,217 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:41:06,218 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:41:12,221 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:41:18,223 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:41:21,225 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:41:27,227 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:41:33,229 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:41:36,230 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:41:42,233 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:41:48,238 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:41:51,240 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:41:57,242 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:42:03,244 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:42:06,246 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:42:12,248 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:42:18,250 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:42:21,252 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:42:27,254 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:42:33,256 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:42:36,258 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:42:42,261 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:42:48,263 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:42:51,265 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:42:57,267 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:43:03,269 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:43:06,271 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:43:12,273 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:43:18,275 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:43:27,278 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:43:33,281 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:43:36,282 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:43:42,285 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:43:48,287 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:43:51,289 INFO org.apache.hadoop.mapred.TaskTracker: > > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at > > 0.09 MB/s) > > > 2011-09-28 07:43:57,291 INFO org.apache.hadoop.mapred.TaskTracker: > > > > > > *Great thanks hoping you could help me.* > > > > > > On Mon, Sep 26, 2011 at 6:39 PM, Devaraj k wrote: > > > > > Hi Bharath, > > > > > > There are few reasons to cause this problem. I have listed below some > > > reasons with solutions. This might help you to solve this. If you post > > the > > > logs, the problem can be figured out. > > > > > > > > > Reason 1: > > > It could be that the mapping in the /etc/hosts file is not present. > > > The DNS server is down as a result of which the hostnames cannot be > > > resolved. > > > The DNS server is in-correctly configured. > > > > > > Solution: Setting the slave.host.name property can be one solution. > > > Appropriate changes need to be done based on the problem. > > > > > > > > > Reason 2: If the map outputs are larger, we may get > > > java.lang.OutOfMemoryError: Java heap space. Because of this there are > > too > > > many fetch failures. > > > > > > Solution: The error, java.lang.OutOfMemoryError: Java heap space in > task > > > tracker logs can be solved by any of the following methods: > > > > > > By decreasing the value configured for > > > mapred.job.shuffle.input.buffer.percent. > > > By increasing the heap memory of child JVM options for the property > > > mapred.child.java.opts. > > > > > > > > > > > > > > > Thanks > > > Devaraj > > > > > > ________________________________________ > > > From: bharath vissapragada [bharathvissapragada1990@gmail.com] > > > Sent: Monday, September 26, 2011 8:54 PM > > > To: common-user@hadoop.apache.org > > > Subject: Re: Too many fetch failures. Help! > > > > > > Hey, > > > > > > Try configuring your cluster with hostnames instead of ips and add > > > those entries to /etc/hosts and sync it across all the nodes in the > > > cluster. You need to restart the cluster after making these changes. > > > > > > Hope this helps, > > > > > > On Mon, Sep 26, 2011 at 8:46 PM, Abdelrahman Kamel < > abdoukamel@gmail.com > > > > > > wrote: > > > > Hi, > > > > This is my first post here. > > > > I'm new to Hadoop. > > > > I've already installed Hadoop on 2 Ubuntu boxes (one is both master > and > > > > slave and the other is only slave). > > > > When I run a Wordcount example on 5 small txt files, the process > never > > > > completes and I get a "Too many fetch failures" error on my terminal. > > > > If you can help me, I cant post my terminal's output and any log > files > > > > needed. > > > > Great thanks. > > > > > > > > -- > > > > Abdelrahman Kamel > > > > > > > > > > > > > > > > -- > > > Regards, > > > Bharath .V > > > w:http://researchweb.iiit.ac.in/~bharath.v > > > > > > > > > > > -- > > Abdelrahman Kamel > > > -- Abdelrahman Kamel --001517440f8a3c3fc704adfc9393--