Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of siddharth.karandikar@gmail.com
 designates 209.85.214.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=IX/o04F79gRBUTzG5Ac4DItdrO+BU8QuM3CNjFoo/gZbZ+jvluZXpA7UUgHUx4Drp0
         /HrJhtrzU+isw0E4OcrzzIaFPJI/1pG8VyxwWoXxX5af3Xfnqq9psHd5uzvihHJVhypM
         E5BWv1nsX5NGmC+57cJ8g9DfOI4Ejey8QOh/E=
MIME-Version: 1.0
Date: Tue, 29 Jun 2010 20:41:47 +0530
Message-ID: <AANLkTinJIHOF2ZSPcCQlYTgXSZMfXuZkHNQFOJ6Ik8dW@mail.gmail.com>
Subject: newbie - job failing at reduce
From: Siddharth Karandikar <siddharth.karandikar@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8

Hi All,

I am new to Hadoop, but by reading online docs and other resource, I
have moved ahead and now trying to run a cluster of 3 nodes.
Before doing this, tried my program on standalone and pseudo systems
and thats working fine.

Now the issue that I am facing - mapping phase works correctly. While
doing reduce, I am seeing following error on one of the nodes -

2010-06-29 14:35:01,848 WARN org.apache.hadoop.mapred.TaskTracker:
getMapOutput(attempt_201006291958_0001_m_000008_0,0) failed :
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
taskTracker/jobcache/job_201006291958_0001/attempt_201006291958_0001_m_000008_0/output/file.out.index
in any of the configured local directories

Lets say this is @ Node1. But there is no such directory named
'taskTracker/jobcache/job_201006291958_0001/attempt_201006291958_0001_m_000008_0'
under /tmp/mapred/local/taskTracker/ on Node1. Interestingly, this
directory is available on Node2 (or Node3). Tried running the job
multiple times, but its always failing while reducing. Same error.

I have configured /tmp/mapred/local on each node from mapred-site.xml.

I really don't understand why mappers are misplacing these files? Or
am I missing something in configuration?

If someone wants to look @ configurations, I have pasted that below.

Thanks,
Siddharth


Configurations
==========

conf/core-site.xml
---------------------------

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://192.168.2.115/</value>
  </property>
</configuration>


conf/hdfs-site.xml
--------------------------
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://192.168.2.115</value>
  </property>
  <property>
    <name>dfs.data.dir</name>
    <value>/home/siddharth/hdfs/data</value>
  </property>
  <property>
    <name>dfs.name.dir</name>
    <value>/home/siddharth/hdfs/name</value>
  </property>
  <property>
    <name>dfs.replication</name>
    <value>3</value>
  </property>
</configuration>

conf/mapred-site.xml
------------------------------
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>192.168.2.115:8021</value>
  </property>
  <property>
    <name>mapred.local.dir</name>
    <value>/tmp/mapred/local</value>
    <final>true</final>
  </property>
  <property>
    <name>mapred.system.dir</name>
    <value>hdfs://192.168.2.115/maperdsystem</value>
    <final>true</final>
  </property>
  <property>
    <name>mapred.tasktracker.map.tasks.maximum</name>
    <value>4</value>
    <final>true</final>
  </property>
  <property>
    <name>mapred.tasktracker.reduce.tasks.maximum</name>
    <value>4</value>
    <final>true</final>
  </property>
  <property>
    <name>mapred.child.java.opts</name>
    <value>-Xmx512m</value>
    <!-- Not marked as final so jobs can include JVM debugging options -->
  </property>
</configuration>