Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of sandy.ryza@cloudera.com
 designates 209.85.220.43 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <7B0D51053A50034199FF706B2513104F09C59287@SACEXCMBX01-PRD.hq.netapp.com>
References: 
 <7B0D51053A50034199FF706B2513104F09C59287@SACEXCMBX01-PRD.hq.netapp.com>
Date: Wed, 15 May 2013 13:49:41 -0700
Message-ID: 
 <CACBYxKJC84a=Ch2Q7hW4bQNsL6obN59ZkrGsg3y6Coryc+7hug@mail.gmail.com>
Subject: Re: Map Tasks do not obey data locality principle........
From: Sandy Ryza <sandy.ryza@cloudera.com>
To: user@hadoop.apache.org
Content-Type: multipart/alternative; boundary=047d7b10cbbd9366bf04dcc7e350

--047d7b10cbbd9366bf04dcc7e350
Content-Type: text/plain; charset=ISO-8859-1

Hi Nikhil,

Which scheduler are you using?

-Sandy


On Tue, May 14, 2013 at 3:55 AM, Agarwal, Nikhil
<Nikhil.Agarwal@netapp.com>wrote:

>  Hi,****
>
> ** **
>
> I  have a 3-node cluster, with JobTracker running on one machine and
> TaskTrackers on other two (say, slave1 and slave2). Instead of using HDFS,
> I have written my own FileSystem implementation. Since, unlike HDFS I am
> unable to provide a shared filesystem view to JobTrackers and TaskTracker
> thus, I mounted the root container of slave2 on a directory in slave1 (nfs
> mount). By doing this I am able to submit MR job to JobTracker, with input
> path as my_scheme://slave1_IP:Port/dir1, etc.  MR runs successfully but
> what happens is that data locality is not ensured i.e. if files A,B,C are
> kept on slave1 and D,E,F on slave2 then according to data locality, map
> tasks should be submitted such that map task of A,B,C are submitted to
> TaskTracker running on slave1 and D,E,F on slave2. Instead of this, it
> randomly schedules the map task to any of the tasktrackers. If map task of
> file A is submitted to TaskTracker running on slave2 then it implies that
> file A is being fetched over the network by slave2.****
>
> ** **
>
> How do I avoid this from happening?****
>
> ** **
>
> Thanks,****
>
> Nikhil****
>
> ** **
>
> ** **
>

--047d7b10cbbd9366bf04dcc7e350
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Nikhil,<div><br></div><div>Which scheduler are you usin=
g?</div><div><br></div><div style>-Sandy</div></div><div class=3D"gmail_ext=
ra"><br><br><div class=3D"gmail_quote">On Tue, May 14, 2013 at 3:55 AM, Aga=
rwal, Nikhil <span dir=3D"ltr">&lt;<a href=3D"mailto:Nikhil.Agarwal@netapp.=
com" target=3D"_blank">Nikhil.Agarwal@netapp.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">


<div lang=3D"EN-US" link=3D"blue" vlink=3D"purple">
<div>
<p class=3D"MsoNormal">Hi,<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=A0<u></u></p>
<p class=3D"MsoNormal">I=A0 have a 3-node cluster, with JobTracker running =
on one machine and TaskTrackers on other two (say, slave1 and slave2). Inst=
ead of using HDFS, I have written my own FileSystem implementation. Since, =
unlike HDFS I am unable to provide a
 shared filesystem view to JobTrackers and TaskTracker thus, I mounted the =
root container of slave2 on a directory in slave1 (nfs mount). By doing thi=
s I am able to submit MR job to JobTracker, with input path as my_scheme://=
slave1_IP:Port/dir1, etc. =A0MR runs
 successfully but what happens is that data locality is not ensured i.e. if=
 files A,B,C are kept on slave1 and D,E,F on slave2 then according to data =
locality, map tasks should be submitted such that map task of A,B,C are sub=
mitted to TaskTracker running on
 slave1 and D,E,F on slave2. Instead of this, it randomly schedules the map=
 task to any of the tasktrackers. If map task of file A is submitted to Tas=
kTracker running on slave2 then it implies that file A is being fetched ove=
r the network by slave2.<u></u><u></u></p>

<p class=3D"MsoNormal"><u></u>=A0<u></u></p>
<p class=3D"MsoNormal">How do I avoid this from happening?<u></u><u></u></p=
>
<p class=3D"MsoNormal"><u></u>=A0<u></u></p>
<p class=3D"MsoNormal">Thanks,<u></u><u></u></p>
<p class=3D"MsoNormal">Nikhil<u></u><u></u></p>
<p class=3D"MsoNormal"><u></u>=A0<u></u></p>
<p class=3D"MsoNormal"><u></u>=A0<u></u></p>
</div>
</div>

</blockquote></div><br></div>

--047d7b10cbbd9366bf04dcc7e350--