Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of siddharth.karandikar@gmail.com
 designates 209.85.214.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=xGmlZsoQn3xKaPPKJUEKOY6cn3qVKkjy3TkackTdm23+rUAsaG3VDSN+qX6IynHWu5
         Gh0O/HhF8yCK/SJUwfVqzZ2cOL7u9sdcpdThQNTZkjZWYF2VwGsuQW+pJ7J926AyQg2o
         Wt0D1D+CVJE4XiFCHFxspjICLZygGb2C3yhiY=
MIME-Version: 1.0
In-Reply-To: <AANLkTinJIHOF2ZSPcCQlYTgXSZMfXuZkHNQFOJ6Ik8dW@mail.gmail.com>
References: <AANLkTinJIHOF2ZSPcCQlYTgXSZMfXuZkHNQFOJ6Ik8dW@mail.gmail.com>
Date: Wed, 30 Jun 2010 15:32:26 +0530
Message-ID: <AANLkTil1Su9DoYR_mYPlYLW34iAVtDq4PFe6aAmlphFS@mail.gmail.com>
Subject: Re: newbie - job failing at reduce
From: Siddharth Karandikar <siddharth.karandikar@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Anyone?


On Tue, Jun 29, 2010 at 8:41 PM, Siddharth Karandikar
<siddharth.karandikar@gmail.com> wrote:
> Hi All,
>
> I am new to Hadoop, but by reading online docs and other resource, I
> have moved ahead and now trying to run a cluster of 3 nodes.
> Before doing this, tried my program on standalone and pseudo systems
> and thats working fine.
>
> Now the issue that I am facing - mapping phase works correctly. While
> doing reduce, I am seeing following error on one of the nodes -
>
> 2010-06-29 14:35:01,848 WARN org.apache.hadoop.mapred.TaskTracker:
> getMapOutput(attempt_201006291958_0001_m_000008_0,0) failed :
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
> taskTracker/jobcache/job_201006291958_0001/attempt_201006291958_0001_m_00=
0008_0/output/file.out.index
> in any of the configured local directories
>
> Lets say this is @ Node1. But there is no such directory named
> 'taskTracker/jobcache/job_201006291958_0001/attempt_201006291958_0001_m_0=
00008_0'
> under /tmp/mapred/local/taskTracker/ on Node1. Interestingly, this
> directory is available on Node2 (or Node3). Tried running the job
> multiple times, but its always failing while reducing. Same error.
>
> I have configured /tmp/mapred/local on each node from mapred-site.xml.
>
> I really don't understand why mappers are misplacing these files? Or
> am I missing something in configuration?
>
> If someone wants to look @ configurations, I have pasted that below.
>
> Thanks,
> Siddharth
>
>
> Configurations
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
>
> conf/core-site.xml
> ---------------------------
>
> <?xml version=3D"1.0"?>
> <?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>
> <configuration>
> =C2=A0<property>
> =C2=A0 =C2=A0<name>fs.default.name</name>
> =C2=A0 =C2=A0<value>hdfs://192.168.2.115/</value>
> =C2=A0</property>
> </configuration>
>
>
> conf/hdfs-site.xml
> --------------------------
> <?xml version=3D"1.0"?>
> <?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>
> <configuration>
> =C2=A0<property>
> =C2=A0 =C2=A0<name>fs.default.name</name>
> =C2=A0 =C2=A0<value>hdfs://192.168.2.115</value>
> =C2=A0</property>
> =C2=A0<property>
> =C2=A0 =C2=A0<name>dfs.data.dir</name>
> =C2=A0 =C2=A0<value>/home/siddharth/hdfs/data</value>
> =C2=A0</property>
> =C2=A0<property>
> =C2=A0 =C2=A0<name>dfs.name.dir</name>
> =C2=A0 =C2=A0<value>/home/siddharth/hdfs/name</value>
> =C2=A0</property>
> =C2=A0<property>
> =C2=A0 =C2=A0<name>dfs.replication</name>
> =C2=A0 =C2=A0<value>3</value>
> =C2=A0</property>
> </configuration>
>
> conf/mapred-site.xml
> ------------------------------
> <?xml version=3D"1.0"?>
> <?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>
> <configuration>
> =C2=A0<property>
> =C2=A0 =C2=A0<name>mapred.job.tracker</name>
> =C2=A0 =C2=A0<value>192.168.2.115:8021</value>
> =C2=A0</property>
> =C2=A0<property>
> =C2=A0 =C2=A0<name>mapred.local.dir</name>
> =C2=A0 =C2=A0<value>/tmp/mapred/local</value>
> =C2=A0 =C2=A0<final>true</final>
> =C2=A0</property>
> =C2=A0<property>
> =C2=A0 =C2=A0<name>mapred.system.dir</name>
> =C2=A0 =C2=A0<value>hdfs://192.168.2.115/maperdsystem</value>
> =C2=A0 =C2=A0<final>true</final>
> =C2=A0</property>
> =C2=A0<property>
> =C2=A0 =C2=A0<name>mapred.tasktracker.map.tasks.maximum</name>
> =C2=A0 =C2=A0<value>4</value>
> =C2=A0 =C2=A0<final>true</final>
> =C2=A0</property>
> =C2=A0<property>
> =C2=A0 =C2=A0<name>mapred.tasktracker.reduce.tasks.maximum</name>
> =C2=A0 =C2=A0<value>4</value>
> =C2=A0 =C2=A0<final>true</final>
> =C2=A0</property>
> =C2=A0<property>
> =C2=A0 =C2=A0<name>mapred.child.java.opts</name>
> =C2=A0 =C2=A0<value>-Xmx512m</value>
> =C2=A0 =C2=A0<!-- Not marked as final so jobs can include JVM debugging o=
ptions -->
> =C2=A0</property>
> </configuration>
>