Return-Path: 
 <common-user-return-30394-apmail-hadoop-common-user-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-common-user-archive@www.apache.org
Delivered-To: apmail-hadoop-common-user-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 130D2778D
	for <apmail-hadoop-common-user-archive@www.apache.org>;
 Wed, 28 Sep 2011 06:59:43 +0000 (UTC)
Received: (qmail 30073 invoked by uid 500); 28 Sep 2011 06:59:39 -0000
Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org
Received: (qmail 30030 invoked by uid 500); 28 Sep 2011 06:59:39 -0000
Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:common-user-help@hadoop.apache.org>
List-Unsubscribe: <mailto:common-user-unsubscribe@hadoop.apache.org>
List-Post: <mailto:common-user@hadoop.apache.org>
List-Id: <common-user.hadoop.apache.org>
Reply-To: common-user@hadoop.apache.org
Delivered-To: mailing list common-user@hadoop.apache.org
Received: (qmail 30007 invoked by uid 99); 28 Sep 2011 06:59:39 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Sep 2011 06:59:39 +0000
X-ASF-Spam-Status: No, hits=4.0 required=5.0
	tests=FREEMAIL_FROM,FREEMAIL_REPLY,HTML_FONT_SIZE_LARGE,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,WEIRD_PORT
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of abdoukamel@gmail.com
 designates 209.85.161.48 as permitted sender)
Received: from [209.85.161.48] (HELO mail-fx0-f48.google.com) (209.85.161.48)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Sep 2011 06:59:31 +0000
Received: by fxd23 with SMTP id 23so388828fxd.35
        for <common-user@hadoop.apache.org>;
 Tue, 27 Sep 2011 23:59:09 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        bh=wJB+D8I4bKhVf2zk6bEj9dUfIOK0jEZscZ1Hbaq2GIA=;
        b=HvrtOMluWW8gEphRbYWHK733VhHYGLQKoRbNVD0HT1gGdzMOm1uGWZ7FIuCv7plNUb
         a47525k5gtcjiUd6glkzt5fRXjcvSppnP4MHt69pP4O7fRTyrFHPNElU1pfAu2sV5whz
         60e2waNq99AzwXnZ+SQKKm6foKHnoYaUSoQ10=
MIME-Version: 1.0
Received: by 10.223.23.197 with SMTP id s5mr518221fab.22.1317193148928; Tue,
 27 Sep 2011 23:59:08 -0700 (PDT)
Received: by 10.223.108.78 with HTTP; Tue, 27 Sep 2011 23:59:08 -0700 (PDT)
In-Reply-To: 
 <06006DDA5A27D541991944AC4117E7A9131D06EB@szxeml524-mbs.china.huawei.com>
References: 
 <CA+xnphW04HMmpXoLCVmFC=ZfM6b_RfvepiJdY2KEoFcQgT0DPw@mail.gmail.com>
	<CAK3hZ7SXVSaptVyMkBHtVqNvHy5-SFbRXQAU5SXiHney9DJd7w@mail.gmail.com>
	<06006DDA5A27D541991944AC4117E7A9131D06EB@szxeml524-mbs.china.huawei.com>
Date: Wed, 28 Sep 2011 09:59:08 +0300
Message-ID: 
 <CA+xnphUkL7G=LO0zCaKUcypsANc4UaZhpeJ6=qkeNaT2As5hQA@mail.gmail.com>
Subject: Re: Too many fetch failures. Help!
From: Abdelrahman Kamel <abdoukamel@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=00151747bb4cbe14b404adfaed3d

--00151747bb4cbe14b404adfaed3d
Content-Type: text/plain; charset=ISO-8859-1

Thanks very much for all your fast replies.

*Here is my terminal output:*

hduser@hdmaster:/usr/local/hadoop$ bin/hadoop jar hadoop-0.20.2-examples.jar
wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output
11/09/28 07:38:23 INFO input.FileInputFormat: Total input paths to process :
5
11/09/28 07:38:23 INFO mapred.JobClient: Running job: job_201109280735_0002
11/09/28 07:38:24 INFO mapred.JobClient:  map 0% reduce 0%
11/09/28 07:38:42 INFO mapred.JobClient:  map 20% reduce 0%
11/09/28 07:38:44 INFO mapred.JobClient:  map 40% reduce 0%
11/09/28 07:38:45 INFO mapred.JobClient:  map 60% reduce 0%
11/09/28 07:38:47 INFO mapred.JobClient:  map 80% reduce 0%
11/09/28 07:38:51 INFO mapred.JobClient:  map 100% reduce 0%
11/09/28 07:38:54 INFO mapred.JobClient:  map 100% reduce 13%
11/09/28 07:39:01 INFO mapred.JobClient:  map 100% reduce 20%

*The terminal is stuck here.*

*And here is my JobTracker log:*

2011-09-28 07:35:43,185 INFO org.apache.hadoop.mapred.JobTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG:   host = hdmaster/127.0.1.1
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2011-09-28 07:35:43,256 INFO org.apache.hadoop.mapred.JobTracker: Scheduler
configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2011-09-28 07:35:43,310 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=JobTracker, port=54311
2011-09-28 07:35:53,431 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2011-09-28 07:35:53,510 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open() is -1.
Opening the listener on 50030
2011-09-28 07:35:53,511 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50030
webServer.getConnectors()[0].getLocalPort() returned 50030
2011-09-28 07:35:53,511 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 50030
2011-09-28 07:35:53,511 INFO org.mortbay.log: jetty-6.1.14
2011-09-28 07:35:53,816 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50030
2011-09-28 07:35:53,817 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=JobTracker, sessionId=
2011-09-28 07:35:53,818 INFO org.apache.hadoop.mapred.JobTracker: JobTracker
up at: 54311
2011-09-28 07:35:53,818 INFO org.apache.hadoop.mapred.JobTracker: JobTracker
webserver: 50030
2011-09-28 07:35:53,926 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
up the system directory
2011-09-28 07:35:53,930 INFO org.apache.hadoop.mapred.JobTracker: problem
cleaning system directory: hdfs://master:54310/app/hadoop/tmp/mapred/system
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
/app/hadoop/tmp/mapred/system. Name node is in safe mode.
The ratio of reported blocks 0.0000 has not reached the threshold 0.9990.
Safe mode will be turned off automatically.
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
    at
org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

    at org.apache.hadoop.ipc.Client.call(Client.java:740)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
    at $Proxy4.delete(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
    at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
    at $Proxy4.delete(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
    at
org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
2011-09-28 07:36:03,934 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
up the system directory
2011-09-28 07:36:03,935 INFO org.apache.hadoop.mapred.JobTracker: problem
cleaning system directory: hdfs://master:54310/app/hadoop/tmp/mapred/system
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
/app/hadoop/tmp/mapred/system. Name node is in safe mode.
The ratio of reported blocks 0.0000 has not reached the threshold 0.9990.
Safe mode will be turned off automatically.
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
    at
org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

    at org.apache.hadoop.ipc.Client.call(Client.java:740)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
    at $Proxy4.delete(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
    at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
    at $Proxy4.delete(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
    at
org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
2011-09-28 07:36:13,942 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
up the system directory
2011-09-28 07:36:13,944 INFO org.apache.hadoop.mapred.JobTracker: problem
cleaning system directory: hdfs://master:54310/app/hadoop/tmp/mapred/system
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
/app/hadoop/tmp/mapred/system. Name node is in safe mode.
The ratio of reported blocks 1.0000 has reached the threshold 0.9990. Safe
mode will be turned off automatically in 25 seconds.
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
    at
org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

    at org.apache.hadoop.ipc.Client.call(Client.java:740)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
    at $Proxy4.delete(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
    at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
    at $Proxy4.delete(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
    at
org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
2011-09-28 07:36:23,947 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
up the system directory
2011-09-28 07:36:23,949 INFO org.apache.hadoop.mapred.JobTracker: problem
cleaning system directory: hdfs://master:54310/app/hadoop/tmp/mapred/system
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
/app/hadoop/tmp/mapred/system. Name node is in safe mode.
The ratio of reported blocks 1.0000 has reached the threshold 0.9990. Safe
mode will be turned off automatically in 15 seconds.
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
    at
org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

    at org.apache.hadoop.ipc.Client.call(Client.java:740)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
    at $Proxy4.delete(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
    at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
    at $Proxy4.delete(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
    at
org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
2011-09-28 07:36:33,952 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
up the system directory
2011-09-28 07:36:33,954 INFO org.apache.hadoop.mapred.JobTracker: problem
cleaning system directory: hdfs://master:54310/app/hadoop/tmp/mapred/system
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
/app/hadoop/tmp/mapred/system. Name node is in safe mode.
The ratio of reported blocks 1.0000 has reached the threshold 0.9990. Safe
mode will be turned off automatically in 5 seconds.
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
    at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
    at
org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)

    at org.apache.hadoop.ipc.Client.call(Client.java:740)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
    at $Proxy4.delete(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
    at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
    at $Proxy4.delete(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
    at
org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
    at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
    at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
    at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
2011-09-28 07:37:09,922 INFO org.apache.hadoop.mapred.JobTracker: Cleaning
up the system directory
2011-09-28 07:37:10,107 INFO
org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
inactive
2011-09-28 07:37:11,461 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2011-09-28 07:37:11,462 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 54311: starting
2011-09-28 07:37:11,462 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 54311: starting
2011-09-28 07:37:11,463 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 54311: starting
2011-09-28 07:37:11,463 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 54311: starting
2011-09-28 07:37:11,463 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 54311: starting
2011-09-28 07:37:11,470 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 4 on 54311: starting
2011-09-28 07:37:11,470 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 6 on 54311: starting
2011-09-28 07:37:11,470 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 7 on 54311: starting
2011-09-28 07:37:11,472 INFO org.apache.hadoop.mapred.JobTracker: Starting
RUNNING
2011-09-28 07:37:11,472 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 5 on 54311: starting
2011-09-28 07:37:11,472 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 8 on 54311: starting
2011-09-28 07:37:11,477 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 9 on 54311: starting
2011-09-28 07:37:11,577 INFO org.apache.hadoop.net.NetworkTopology: Adding a
new node: /default-rack/hdmaster
2011-09-28 07:37:43,188 INFO org.apache.hadoop.net.NetworkTopology: Adding a
new node: /default-rack/hdslave.ubuntu-domain
2011-09-28 07:38:23,832 INFO org.apache.hadoop.mapred.JobTracker:
Initializing job_201109280735_0002
2011-09-28 07:38:23,838 INFO org.apache.hadoop.mapred.JobInProgress:
Initializing job_201109280735_0002
2011-09-28 07:38:24,013 INFO org.apache.hadoop.mapred.JobInProgress: Input
size for job job_201109280735_0002 = 3430958. Number of splits = 5
2011-09-28 07:38:24,013 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201109280735_0002_m_000000 has split on node:/default-rack/hdmaster
2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201109280735_0002_m_000000 has split on
node:/default-rack/hdslave.ubuntu-domain
2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201109280735_0002_m_000001 has split on node:/default-rack/hdmaster
2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201109280735_0002_m_000001 has split on
node:/default-rack/hdslave.ubuntu-domain
2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201109280735_0002_m_000002 has split on node:/default-rack/hdmaster
2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201109280735_0002_m_000002 has split on
node:/default-rack/hdslave.ubuntu-domain
2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201109280735_0002_m_000003 has split on node:/default-rack/hdmaster
2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201109280735_0002_m_000003 has split on
node:/default-rack/hdslave.ubuntu-domain
2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201109280735_0002_m_000004 has split on node:/default-rack/hdmaster
2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201109280735_0002_m_000004 has split on
node:/default-rack/hdslave.ubuntu-domain
2011-09-28 07:38:35,674 INFO org.apache.hadoop.mapred.JobTracker: Adding
task 'attempt_201109280735_0002_m_000006_0' to tip
task_201109280735_0002_m_000006, for tracker
'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
2011-09-28 07:38:38,710 INFO org.apache.hadoop.mapred.JobInProgress: Task
'attempt_201109280735_0002_m_000006_0' has completed
task_201109280735_0002_m_000006 successfully.
2011-09-28 07:38:38,714 INFO org.apache.hadoop.mapred.JobTracker: Adding
task 'attempt_201109280735_0002_m_000000_0' to tip
task_201109280735_0002_m_000000, for tracker
'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
2011-09-28 07:38:38,717 INFO org.apache.hadoop.mapred.JobInProgress:
Choosing data-local task task_201109280735_0002_m_000000
2011-09-28 07:38:38,717 INFO org.apache.hadoop.mapred.JobTracker: Adding
task 'attempt_201109280735_0002_m_000001_0' to tip
task_201109280735_0002_m_000001, for tracker
'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
2011-09-28 07:38:38,718 INFO org.apache.hadoop.mapred.JobInProgress:
Choosing data-local task task_201109280735_0002_m_000001
2011-09-28 07:38:40,233 INFO org.apache.hadoop.mapred.JobTracker: Adding
task 'attempt_201109280735_0002_m_000002_0' to tip
task_201109280735_0002_m_000002, for tracker 'tracker_hdmaster:localhost/
127.0.0.1:54873'
2011-09-28 07:38:40,233 INFO org.apache.hadoop.mapred.JobInProgress:
Choosing data-local task task_201109280735_0002_m_000002
2011-09-28 07:38:40,233 INFO org.apache.hadoop.mapred.JobTracker: Adding
task 'attempt_201109280735_0002_m_000003_0' to tip
task_201109280735_0002_m_000003, for tracker 'tracker_hdmaster:localhost/
127.0.0.1:54873'
2011-09-28 07:38:40,234 INFO org.apache.hadoop.mapred.JobInProgress:
Choosing data-local task task_201109280735_0002_m_000003
2011-09-28 07:38:41,724 INFO org.apache.hadoop.mapred.JobInProgress: Task
'attempt_201109280735_0002_m_000000_0' has completed
task_201109280735_0002_m_000000 successfully.
2011-09-28 07:38:41,724 INFO org.apache.hadoop.mapred.ResourceEstimator:
completedMapsUpdates:1  completedMapsInputSize:1423802
completedMapsOutputSize:466814
2011-09-28 07:38:41,726 INFO org.apache.hadoop.mapred.JobTracker: Adding
task 'attempt_201109280735_0002_m_000004_0' to tip
task_201109280735_0002_m_000004, for tracker
'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
2011-09-28 07:38:41,726 INFO org.apache.hadoop.mapred.JobInProgress:
Choosing data-local task task_201109280735_0002_m_000004
2011-09-28 07:38:41,730 INFO org.apache.hadoop.mapred.JobTracker: Adding
task 'attempt_201109280735_0002_r_000000_0' to tip
task_201109280735_0002_r_000000, for tracker
'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
2011-09-28 07:38:43,272 INFO org.apache.hadoop.mapred.JobInProgress: Task
'attempt_201109280735_0002_m_000003_0' has completed
task_201109280735_0002_m_000003 successfully.
2011-09-28 07:38:43,272 INFO org.apache.hadoop.mapred.ResourceEstimator:
completedMapsUpdates:2  completedMapsInputSize:1817766
completedMapsOutputSize:719434
2011-09-28 07:38:44,746 INFO org.apache.hadoop.mapred.JobInProgress: Task
'attempt_201109280735_0002_m_000001_0' has completed
task_201109280735_0002_m_000001 successfully.
2011-09-28 07:38:44,746 INFO org.apache.hadoop.mapred.ResourceEstimator:
completedMapsUpdates:3  completedMapsInputSize:2492333
completedMapsOutputSize:986447
2011-09-28 07:38:46,284 INFO org.apache.hadoop.mapred.JobInProgress: Task
'attempt_201109280735_0002_m_000002_0' has completed
task_201109280735_0002_m_000002 successfully.
2011-09-28 07:38:46,284 INFO org.apache.hadoop.mapred.ResourceEstimator:
completedMapsUpdates:4  completedMapsInputSize:3087267
completedMapsOutputSize:1204101
2011-09-28 07:38:50,874 INFO org.apache.hadoop.mapred.JobInProgress: Task
'attempt_201109280735_0002_m_000004_0' has completed
task_201109280735_0002_m_000004 successfully.
2011-09-28 07:38:50,874 INFO org.apache.hadoop.mapred.ResourceEstimator:
completedMapsUpdates:5  completedMapsInputSize:3430963
completedMapsOutputSize:1367489
2011-09-28 07:39:06,502 INFO org.apache.hadoop.mapred.JobInProgress: Failed
fetch notification #1 for task attempt_201109280735_0002_m_000003_0
2011-09-28 07:41:53,093 INFO org.apache.hadoop.mapred.JobInProgress: Failed
fetch notification #2 for task attempt_201109280735_0002_m_000003_0
2011-09-28 07:47:06,741 INFO org.apache.hadoop.mapred.JobInProgress: Failed
fetch notification #3 for task attempt_201109280735_0002_m_000003_0
2011-09-28 07:47:06,742 INFO org.apache.hadoop.mapred.JobInProgress: Too
many fetch-failures for output of task: attempt_201109280735_0002_m_000003_0
... killing it
2011-09-28 07:47:06,742 INFO org.apache.hadoop.mapred.TaskInProgress: Error
from attempt_201109280735_0002_m_000003_0: Too many fetch-failures
2011-09-28 07:47:06,743 INFO org.apache.hadoop.mapred.JobTracker: Adding
task 'attempt_201109280735_0002_m_000003_1' to tip
task_201109280735_0002_m_000003, for tracker
'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
2011-09-28 07:47:06,743 INFO org.apache.hadoop.mapred.JobInProgress:
Choosing data-local task task_201109280735_0002_m_000003
2011-09-28 07:47:07,607 INFO org.apache.hadoop.mapred.JobTracker: Removed
completed task 'attempt_201109280735_0002_m_000003_0' from
'tracker_hdmaster:localhost/127.0.0.1:54873'
2011-09-28 07:47:13,361 INFO org.apache.hadoop.mapred.JobInProgress: Task
'attempt_201109280735_0002_m_000003_1' has completed
task_201109280735_0002_m_000003 successfully.
2011-09-28 07:47:13,361 INFO org.apache.hadoop.mapred.ResourceEstimator:
completedMapsUpdates:6  completedMapsInputSize:3824927
completedMapsOutputSize:1620109

*And here is my TaskTracker #1 log:*

2011-09-28 07:35:44,452 INFO org.apache.hadoop.mapred.TaskTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG:   host = hdmaster/127.0.1.1
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2011-09-28 07:35:54,661 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2011-09-28 07:35:54,744 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open() is -1.
Opening the listener on 50060
2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50060
webServer.getConnectors()[0].getLocalPort() returned 50060
2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 50060
2011-09-28 07:35:54,750 INFO org.mortbay.log: jetty-6.1.14
2011-09-28 07:35:55,025 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50060
2011-09-28 07:35:55,064 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=TaskTracker, sessionId=
2011-09-28 07:35:55,082 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=TaskTracker, port=54873
2011-09-28 07:35:55,127 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker:
TaskTracker up at: localhost/127.0.0.1:54873
2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker: Starting
tracker tracker_hdmaster:localhost/127.0.0.1:54873
2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 54873: starting
2011-09-28 07:35:55,131 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 54873: starting
2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 54873: starting
2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 54873: starting
2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 54873: starting
2011-09-28 07:37:11,483 INFO org.apache.hadoop.mapred.TaskTracker: Starting
thread: Map-events fetcher for all reduce tasks on
tracker_hdmaster:localhost/127.0.0.1:54873
2011-09-28 07:37:11,484 INFO org.apache.hadoop.mapred.TaskTracker:  Using
MemoryCalculatorPlugin :
org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@5a3923
2011-09-28 07:37:11,487 WARN org.apache.hadoop.mapred.TaskTracker:
TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
disabled.
2011-09-28 07:37:11,488 INFO org.apache.hadoop.mapred.IndexCache: IndexCache
created with max memory = 10485760
2011-09-28 07:38:40,269 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000002_0 task's
state:UNASSIGNED
2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000003_0 task's
state:UNASSIGNED
2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
launch : attempt_201109280735_0002_m_000002_0
2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 2 and trying to launch
attempt_201109280735_0002_m_000002_0
2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
launch : attempt_201109280735_0002_m_000003_0
2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 1 and trying to launch
attempt_201109280735_0002_m_000003_0
2011-09-28 07:38:40,538 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201109280735_0002_m_1909679810
2011-09-28 07:38:40,540 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201109280735_0002_m_1909679810 spawned.
2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-622388657
2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201109280735_0002_m_-622388657 spawned.
2011-09-28 07:38:41,046 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
ID: jvm_201109280735_0002_m_1909679810 given task:
attempt_201109280735_0002_m_000002_0
2011-09-28 07:38:41,099 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
ID: jvm_201109280735_0002_m_-622388657 given task:
attempt_201109280735_0002_m_000003_0
2011-09-28 07:38:43,264 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_m_000003_0 1.0%
2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker: Task
attempt_201109280735_0002_m_000003_0 is done.
2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker: reported
output size for attempt_201109280735_0002_m_000003_0  was 0
2011-09-28 07:38:43,266 INFO org.apache.hadoop.mapred.TaskTracker:
addFreeSlot : current free slots : 1
2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_m_000002_0 1.0%
2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker: Task
attempt_201109280735_0002_m_000002_0 is done.
2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker: reported
output size for attempt_201109280735_0002_m_000002_0  was 0
2011-09-28 07:38:43,322 INFO org.apache.hadoop.mapred.TaskTracker:
addFreeSlot : current free slots : 2
2011-09-28 07:38:43,362 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201109280735_0002_m_1909679810 exited. Number of tasks it ran: 1
2011-09-28 07:38:43,412 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201109280735_0002_m_-622388657 exited. Number of tasks it ran: 1


*And here is my TaskTracker #2 log:*

2011-09-28 07:35:44,452 INFO org.apache.hadoop.mapred.TaskTracker:
STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting TaskTracker
STARTUP_MSG:   host = hdmaster/127.0.1.1
STARTUP_MSG:   args = []
STARTUP_MSG:   version = 0.20.2
STARTUP_MSG:   build =
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2011-09-28 07:35:54,661 INFO org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
org.mortbay.log.Slf4jLog
2011-09-28 07:35:54,744 INFO org.apache.hadoop.http.HttpServer: Port
returned by webServer.getConnectors()[0].getLocalPort() before open() is -1.
Opening the listener on 50060
2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer:
listener.getLocalPort() returned 50060
webServer.getConnectors()[0].getLocalPort() returned 50060
2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer: Jetty bound
to port 50060
2011-09-28 07:35:54,750 INFO org.mortbay.log: jetty-6.1.14
2011-09-28 07:35:55,025 INFO org.mortbay.log: Started
SelectChannelConnector@0.0.0.0:50060
2011-09-28 07:35:55,064 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
Initializing JVM Metrics with processName=TaskTracker, sessionId=
2011-09-28 07:35:55,082 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
Initializing RPC Metrics with hostName=TaskTracker, port=54873
2011-09-28 07:35:55,127 INFO org.apache.hadoop.ipc.Server: IPC Server
Responder: starting
2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker:
TaskTracker up at: localhost/127.0.0.1:54873
2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker: Starting
tracker tracker_hdmaster:localhost/127.0.0.1:54873
2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 0 on 54873: starting
2011-09-28 07:35:55,131 INFO org.apache.hadoop.ipc.Server: IPC Server
listener on 54873: starting
2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 1 on 54873: starting
2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 2 on 54873: starting
2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
handler 3 on 54873: starting
2011-09-28 07:37:11,483 INFO org.apache.hadoop.mapred.TaskTracker: Starting
thread: Map-events fetcher for all reduce tasks on
tracker_hdmaster:localhost/127.0.0.1:54873
2011-09-28 07:37:11,484 INFO org.apache.hadoop.mapred.TaskTracker:  Using
MemoryCalculatorPlugin :
org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@5a3923
2011-09-28 07:37:11,487 WARN org.apache.hadoop.mapred.TaskTracker:
TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
disabled.
2011-09-28 07:37:11,488 INFO org.apache.hadoop.mapred.IndexCache: IndexCache
created with max memory = 10485760
2011-09-28 07:38:40,269 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000002_0 task's
state:UNASSIGNED
2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000003_0 task's
state:UNASSIGNED
2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
launch : attempt_201109280735_0002_m_000002_0
2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 2 and trying to launch
attempt_201109280735_0002_m_000002_0
2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
launch : attempt_201109280735_0002_m_000003_0
2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 1 and trying to launch
attempt_201109280735_0002_m_000003_0
2011-09-28 07:38:40,538 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201109280735_0002_m_1909679810
2011-09-28 07:38:40,540 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201109280735_0002_m_1909679810 spawned.
2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-622388657
2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201109280735_0002_m_-622388657 spawned.
2011-09-28 07:38:41,046 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
ID: jvm_201109280735_0002_m_1909679810 given task:
attempt_201109280735_0002_m_000002_0
2011-09-28 07:38:41,099 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
ID: jvm_201109280735_0002_m_-622388657 given task:
attempt_201109280735_0002_m_000003_0
2011-09-28 07:38:43,264 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_m_000003_0 1.0%
2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker: Task
attempt_201109280735_0002_m_000003_0 is done.
2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker: reported
output size for attempt_201109280735_0002_m_000003_0  was 0
2011-09-28 07:38:43,266 INFO org.apache.hadoop.mapred.TaskTracker:
addFreeSlot : current free slots : 1
2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_m_000002_0 1.0%
2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker: Task
attempt_201109280735_0002_m_000002_0 is done.
2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker: reported
output size for attempt_201109280735_0002_m_000002_0  was 0
2011-09-28 07:38:43,322 INFO org.apache.hadoop.mapred.TaskTracker:
addFreeSlot : current free slots : 2
2011-09-28 07:38:43,362 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201109280735_0002_m_1909679810 exited. Number of tasks it ran: 1
2011-09-28 07:38:43,412 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201109280735_0002_m_-622388657 exited. Number of tasks it ran: 1
2011-09-28 07:36:10,343 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/10.0.0.1:54311. Already tried 0 time(s).
2011-09-28 07:36:31,346 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: master/10.0.0.1:54311. Already tried 1 time(s).
2011-09-28 07:36:31,581 INFO org.apache.hadoop.mapred.TaskTracker: Starting
thread: Map-events fetcher for all reduce tasks on
tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342
2011-09-28 07:36:31,583 INFO org.apache.hadoop.mapred.TaskTracker:  Using
MemoryCalculatorPlugin :
org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@ce16ad
2011-09-28 07:36:31,588 WARN org.apache.hadoop.mapred.TaskTracker:
TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
disabled.
2011-09-28 07:36:31,589 INFO org.apache.hadoop.mapred.IndexCache: IndexCache
created with max memory = 10485760
2011-09-28 07:37:35,032 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000006_0 task's
state:UNASSIGNED
2011-09-28 07:37:35,034 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
launch : attempt_201109280735_0002_m_000006_0
2011-09-28 07:37:35,034 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 2 and trying to launch
attempt_201109280735_0002_m_000006_0
2011-09-28 07:37:35,374 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201109280735_0002_m_774270481
2011-09-28 07:37:35,374 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201109280735_0002_m_774270481 spawned.
2011-09-28 07:37:35,773 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
ID: jvm_201109280735_0002_m_774270481 given task:
attempt_201109280735_0002_m_000006_0
2011-09-28 07:37:36,667 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_m_000006_0 0.0% setup
2011-09-28 07:37:36,668 INFO org.apache.hadoop.mapred.TaskTracker: Task
attempt_201109280735_0002_m_000006_0 is done.
2011-09-28 07:37:36,668 INFO org.apache.hadoop.mapred.TaskTracker: reported
output size for attempt_201109280735_0002_m_000006_0  was 0
2011-09-28 07:37:36,670 INFO org.apache.hadoop.mapred.TaskTracker:
addFreeSlot : current free slots : 2
2011-09-28 07:37:36,818 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201109280735_0002_m_774270481 exited. Number of tasks it ran: 1
2011-09-28 07:37:38,033 INFO org.apache.hadoop.mapred.TaskTracker:
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
taskTracker/jobcache/job_201109280735_0002/attempt_201109280735_0002_m_000006_0/output/file.out
in any of the configured local directories
2011-09-28 07:37:38,045 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction
(registerTask): attempt_201109280735_0002_m_000000_0 task's state:UNASSIGNED
2011-09-28 07:37:38,045 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000001_0 task's
state:UNASSIGNED
2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
launch : attempt_201109280735_0002_m_000000_0
2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 2 and trying to launch
attempt_201109280735_0002_m_000000_0
2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker: Received
KillTaskAction for task: attempt_201109280735_0002_m_000006_0
2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker: About to
purge task: attempt_201109280735_0002_m_000006_0
2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskRunner:
attempt_201109280735_0002_m_000006_0 done; removing files.
2011-09-28 07:37:38,047 INFO org.apache.hadoop.mapred.IndexCache: Map ID
attempt_201109280735_0002_m_000006_0 not found in cache
2011-09-28 07:37:38,080 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
launch : attempt_201109280735_0002_m_000001_0
2011-09-28 07:37:38,080 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 1 and trying to launch
attempt_201109280735_0002_m_000001_0
2011-09-28 07:37:38,082 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-1878969571
2011-09-28 07:37:38,086 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201109280735_0002_m_-1878969571 spawned.
2011-09-28 07:37:38,129 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-364970930
2011-09-28 07:37:38,129 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201109280735_0002_m_-364970930 spawned.
2011-09-28 07:37:38,590 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
ID: jvm_201109280735_0002_m_-364970930 given task:
attempt_201109280735_0002_m_000001_0
2011-09-28 07:37:38,604 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
ID: jvm_201109280735_0002_m_-1878969571 given task:
attempt_201109280735_0002_m_000000_0
2011-09-28 07:37:40,644 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_m_000000_0 1.0%
2011-09-28 07:37:40,645 INFO org.apache.hadoop.mapred.TaskTracker: Task
attempt_201109280735_0002_m_000000_0 is done.
2011-09-28 07:37:40,645 INFO org.apache.hadoop.mapred.TaskTracker: reported
output size for attempt_201109280735_0002_m_000000_0  was 0
2011-09-28 07:37:40,645 INFO org.apache.hadoop.mapred.TaskTracker:
addFreeSlot : current free slots : 1
2011-09-28 07:37:40,697 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201109280735_0002_m_-1878969571 exited. Number of tasks it ran: 1
2011-09-28 07:37:41,067 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000004_0 task's
state:UNASSIGNED
2011-09-28 07:37:41,067 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
launch : attempt_201109280735_0002_m_000004_0
2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 1 and trying to launch
attempt_201109280735_0002_m_000004_0
2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker:
LaunchTaskAction (registerTask): attempt_201109280735_0002_r_000000_0 task's
state:UNASSIGNED
2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker: Trying to
launch : attempt_201109280735_0002_r_000000_0
2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker: In
TaskLauncher, current free slots : 2 and trying to launch
attempt_201109280735_0002_r_000000_0
2011-09-28 07:37:41,118 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201109280735_0002_m_531367236
2011-09-28 07:37:41,118 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201109280735_0002_m_531367236 spawned.
2011-09-28 07:37:41,165 INFO org.apache.hadoop.mapred.JvmManager: In
JvmRunner constructed JVM ID: jvm_201109280735_0002_r_774270481
2011-09-28 07:37:41,166 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner
jvm_201109280735_0002_r_774270481 spawned.
2011-09-28 07:37:41,635 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
ID: jvm_201109280735_0002_r_774270481 given task:
attempt_201109280735_0002_r_000000_0
2011-09-28 07:37:41,660 INFO org.apache.hadoop.mapred.TaskTracker: JVM with
ID: jvm_201109280735_0002_m_531367236 given task:
attempt_201109280735_0002_m_000004_0
2011-09-28 07:37:43,417 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_m_000001_0 1.0%
2011-09-28 07:37:43,418 INFO org.apache.hadoop.mapred.TaskTracker: Task
attempt_201109280735_0002_m_000001_0 is done.
2011-09-28 07:37:43,418 INFO org.apache.hadoop.mapred.TaskTracker: reported
output size for attempt_201109280735_0002_m_000001_0  was 0
2011-09-28 07:37:43,419 INFO org.apache.hadoop.mapred.TaskTracker:
addFreeSlot : current free slots : 1
2011-09-28 07:37:43,456 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201109280735_0002_m_-364970930 exited. Number of tasks it ran: 1
2011-09-28 07:37:46,712 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_m_000004_0 1.0%
2011-09-28 07:37:46,713 INFO org.apache.hadoop.mapred.TaskTracker: Task
attempt_201109280735_0002_m_000004_0 is done.2011-09-28 07:37:46,713 INFO
org.apache.hadoop.mapred.TaskTracker: reported output size for
attempt_201109280735_0002_m_000004_0  was 0
2011-09-28 07:37:46,714 INFO org.apache.hadoop.mapred.TaskTracker:
addFreeSlot : current free slots : 2
2011-09-28 07:37:46,847 INFO org.apache.hadoop.mapred.JvmManager: JVM :
jvm_201109280735_0002_m_531367236 exited. Number of tasks it ran: 1
2011-09-28 07:37:47,276 INFO org.apache.hadoop.mapred.TaskTracker: Sent out
466814 bytes for reduce: 0 from map: attempt_201109280735_0002_m_000000_0
given 466814/466810
2011-09-28 07:37:47,276 INFO
org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.1.1:50060,
dest: 127.0.0.1:52662, bytes: 466814, op: MAPRED_SHUFFLE, cliID:
attempt_201109280735_0002_m_000000_0
2011-09-28 07:37:47,287 INFO org.apache.hadoop.mapred.TaskTracker: Sent out
267013 bytes for reduce: 0 from map: attempt_201109280735_0002_m_000001_0
given 267013/267009
2011-09-28 07:37:47,287 INFO
org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.1.1:50060,
dest: 127.0.0.1:52663, bytes: 267013, op: MAPRED_SHUFFLE, cliID:
attempt_201109280735_0002_m_000001_0
2011-09-28 07:37:48,128 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.13333334% reduce > copy (2 of 5 at
0.12 MB/s) >
2011-09-28 07:37:51,130 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.13333334% reduce > copy (2 of 5 at
0.12 MB/s) >
2011-09-28 07:37:51,176 INFO org.apache.hadoop.mapred.TaskTracker: Sent out
163388 bytes for reduce: 0 from map: attempt_201109280735_0002_m_000004_0
given 163388/163384
2011-09-28 07:37:51,176 INFO
org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.1.1:50060,
dest: 127.0.0.1:52664, bytes: 163388, op: MAPRED_SHUFFLE, cliID:
attempt_201109280735_0002_m_000004_0
2011-09-28 07:37:57,132 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:38:03,135 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:38:06,136 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:38:18,140 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:38:21,142 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:38:27,144 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:38:33,146 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:38:36,148 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >2011-09-28 07:38:42,150 INFO
org.apache.hadoop.mapred.TaskTracker: attempt_201109280735_0002_r_000000_0
0.20000002% reduce > copy (3 of 5 at 0.09 MB/s) >
2011-09-28 07:38:48,153 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:38:51,163 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:38:57,166 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:03,168 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:06,170 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:12,172 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:18,175 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:21,176 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:27,178 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:33,181 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:36,183 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:42,185 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:48,187 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:51,189 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:39:57,191 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:40:03,193 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:40:06,195 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:40:12,197 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:40:18,199 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:40:21,201 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0
0.20000002% reduce > copy (3 of 5 at 0.09 MB/s) >
2011-09-28 07:40:27,203 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:40:33,205 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:40:36,206 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:40:42,209 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:40:51,212 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:40:57,214 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:03,217 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:06,218 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:12,221 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:18,223 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:21,225 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:27,227 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:33,229 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:36,230 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:42,233 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:48,238 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:51,240 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:41:57,242 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:03,244 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:06,246 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:12,248 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:18,250 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:21,252 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:27,254 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:33,256 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:36,258 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:42,261 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:48,263 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:51,265 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:42:57,267 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:43:03,269 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:43:06,271 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:43:12,273 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:43:18,275 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:43:27,278 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:43:33,281 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:43:36,282 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:43:42,285 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:43:48,287 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:43:51,289 INFO org.apache.hadoop.mapred.TaskTracker:
attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
0.09 MB/s) >
2011-09-28 07:43:57,291 INFO org.apache.hadoop.mapred.TaskTracker:


*Great thanks hoping you could help me.*


On Mon, Sep 26, 2011 at 6:39 PM, Devaraj k <devaraj.k@huawei.com> wrote:

> Hi Bharath,
>
> There are few reasons to cause this problem. I have listed below some
> reasons with solutions. This might help you to solve this. If you post the
> logs, the problem can be figured out.
>
>
> Reason 1:
> It could be that the mapping in the /etc/hosts file is not present.
> The DNS server is down as a result of which the hostnames cannot be
> resolved.
> The DNS server is in-correctly configured.
>
> Solution: Setting the slave.host.name property can be one solution.
>  Appropriate changes need to be done based on the problem.
>
>
> Reason 2: If the map outputs are larger, we may get
> java.lang.OutOfMemoryError: Java heap space. Because of this there are too
> many fetch failures.
>
> Solution: The error, java.lang.OutOfMemoryError: Java heap space in task
> tracker logs can be solved by any of the following methods:
>
> By decreasing the value configured for
> mapred.job.shuffle.input.buffer.percent.
> By increasing the heap memory of child JVM options for the property
> mapred.child.java.opts.
>
>
>
>
> Thanks
> Devaraj
>
> ________________________________________
> From: bharath vissapragada [bharathvissapragada1990@gmail.com]
> Sent: Monday, September 26, 2011 8:54 PM
> To: common-user@hadoop.apache.org
> Subject: Re: Too many fetch failures. Help!
>
> Hey,
>
> Try configuring your cluster with hostnames instead of ips and add
> those entries to /etc/hosts and sync it across all the nodes in the
> cluster. You need to restart the cluster after making these changes.
>
> Hope this helps,
>
> On Mon, Sep 26, 2011 at 8:46 PM, Abdelrahman Kamel <abdoukamel@gmail.com>
> wrote:
> > Hi,
> > This is my first post here.
> > I'm new to Hadoop.
> > I've already installed Hadoop on 2 Ubuntu boxes (one is both master and
> > slave and the other is only slave).
> > When I run a Wordcount example on 5 small txt files, the process never
> > completes and I get a "Too many fetch failures" error on my terminal.
> > If you can help me, I cant post my terminal's output and any log files
> > needed.
> > Great thanks.
> >
> > --
> > Abdelrahman Kamel
> >
>
>
>
> --
> Regards,
> Bharath .V
> w:http://researchweb.iiit.ac.in/~bharath.v
>



-- 
Abdelrahman Kamel

--00151747bb4cbe14b404adfaed3d--