Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of Nikhil.Agarwal@netapp.com
 designates 216.240.18.38 as permitted sender)
From: "Agarwal, Nikhil" <Nikhil.Agarwal@netapp.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: RE: Map Tasks do not obey data locality principle........
Thread-Topic: Map Tasks do not obey data locality principle........
Thread-Index: Ac5QkWcV9ppaf2q7RI2I/01hA0HsdABWJPgAAARqIjA=
Date: Thu, 16 May 2013 06:08:43 +0000
Message-ID: 
 <7B0D51053A50034199FF706B2513104F09C5A605@SACEXCMBX01-PRD.hq.netapp.com>
References: 
 <7B0D51053A50034199FF706B2513104F09C59287@SACEXCMBX01-PRD.hq.netapp.com>
 <CAOcnVr1ZU+2Kz5dr+jFU2N8e7YSG+nbf-EP_3Nu7sw-3U6g=FQ@mail.gmail.com>
In-Reply-To: 
 <CAOcnVr1ZU+2Kz5dr+jFU2N8e7YSG+nbf-EP_3Nu7sw-3U6g=FQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

No, it does not.  I have kept the granularity at file level rather than a b=
lock. I do not think that should affect the mapping of tasks ?

Regards,
Nikhil=20

-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]=20
Sent: Thursday, May 16, 2013 2:31 AM
To: <user@hadoop.apache.org>
Subject: Re: Map Tasks do not obey data locality principle........

Also, does your custom FS report block locations in the exact same format a=
s how HDFS does?

On Tue, May 14, 2013 at 4:25 PM, Agarwal, Nikhil <Nikhil.Agarwal@netapp.com=
> wrote:
> Hi,
>
>
>
> I  have a 3-node cluster, with JobTracker running on one machine and=20
> TaskTrackers on other two (say, slave1 and slave2). Instead of using=20
> HDFS, I have written my own FileSystem implementation. Since, unlike=20
> HDFS I am unable to provide a shared filesystem view to JobTrackers=20
> and TaskTracker thus, I mounted the root container of slave2 on a=20
> directory in slave1 (nfs mount). By doing this I am able to submit MR=20
> job to JobTracker, with input path as my_scheme://slave1_IP:Port/dir1,=20
> etc.  MR runs successfully but what happens is that data locality is=20
> not ensured i.e. if files A,B,C are kept on
> slave1 and D,E,F on slave2 then according to data locality, map tasks=20
> should be submitted such that map task of A,B,C are submitted to=20
> TaskTracker running on slave1 and D,E,F on slave2. Instead of this, it=20
> randomly schedules the map task to any of the tasktrackers. If map=20
> task of file A is submitted to TaskTracker running on slave2 then it=20
> implies that file A is being fetched over the network by slave2.
>
>
>
> How do I avoid this from happening?
>
>
>
> Thanks,
>
> Nikhil
>
>
>
>


--
Harsh J