Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of Nikhil.Agarwal@netapp.com
 designates 216.240.18.77 as permitted sender)
From: "Agarwal, Nikhil" <Nikhil.Agarwal@netapp.com>
To: "user@hadoop.apache.org" <user@hadoop.apache.org>
Subject: RE: Map Tasks do not obey data locality principle........
Thread-Topic: Map Tasks do not obey data locality principle........
Thread-Index: Ac5QkWcV9ppaf2q7RI2I/01hA0HsdABWJPgAAARqIjAADwOTAAAOnTjA
Date: Thu, 16 May 2013 06:21:42 +0000
Message-ID: 
 <7B0D51053A50034199FF706B2513104F09C5A630@SACEXCMBX01-PRD.hq.netapp.com>
References: 
 <7B0D51053A50034199FF706B2513104F09C59287@SACEXCMBX01-PRD.hq.netapp.com>
 <CAOcnVr1ZU+2Kz5dr+jFU2N8e7YSG+nbf-EP_3Nu7sw-3U6g=FQ@mail.gmail.com>
 <7B0D51053A50034199FF706B2513104F09C5A605@SACEXCMBX01-PRD.hq.netapp.com>
 <CAOcnVr2Ps-2Gyp_sgAPVXp5bT5psYn=EnRj-BbB7hNP7xTm=SA@mail.gmail.com>
In-Reply-To: 
 <CAOcnVr2Ps-2Gyp_sgAPVXp5bT5psYn=EnRj-BbB7hNP7xTm=SA@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

Agreed. Thanks for replying. As hints what I have given is the ip address o=
f the node where the file is residing but still it does not follow data loc=
ality.

One clarification -  If map task for file A is being submitted to a TaskTra=
cker running on different node then does it necessarily mean that entire fi=
le A was transferred to the other node?

Regards,
Nikhil

-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]=20
Sent: Thursday, May 16, 2013 11:47 AM
To: <user@hadoop.apache.org>
Subject: Re: Map Tasks do not obey data locality principle........

The scheduling is done based on block locations filled in by the input spli=
ts. If there's no hints being provided by your FS, then the result you're s=
eeing is correct.

Note that if you don't use a block concept, you ought to consider a whole f=
ile as one block and return a location based on that.

Essentially, your
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/fs/FileSystem.h=
tml#getFileBlockLocations(org.apache.hadoop.fs.FileStatus,%20long,%20long)
form of API calls has to return valid values for scheduling to work.

On Thu, May 16, 2013 at 11:38 AM, Agarwal, Nikhil <Nikhil.Agarwal@netapp.co=
m> wrote:
> No, it does not.  I have kept the granularity at file level rather than a=
 block. I do not think that should affect the mapping of tasks ?
>
> Regards,
> Nikhil
>
> -----Original Message-----
> From: Harsh J [mailto:harsh@cloudera.com]
> Sent: Thursday, May 16, 2013 2:31 AM
> To: <user@hadoop.apache.org>
> Subject: Re: Map Tasks do not obey data locality principle........
>
> Also, does your custom FS report block locations in the exact same format=
 as how HDFS does?
>
> On Tue, May 14, 2013 at 4:25 PM, Agarwal, Nikhil <Nikhil.Agarwal@netapp.c=
om> wrote:
>> Hi,
>>
>>
>>
>> I  have a 3-node cluster, with JobTracker running on one machine and=20
>> TaskTrackers on other two (say, slave1 and slave2). Instead of using=20
>> HDFS, I have written my own FileSystem implementation. Since, unlike=20
>> HDFS I am unable to provide a shared filesystem view to JobTrackers=20
>> and TaskTracker thus, I mounted the root container of slave2 on a=20
>> directory in slave1 (nfs mount). By doing this I am able to submit MR=20
>> job to JobTracker, with input path as=20
>> my_scheme://slave1_IP:Port/dir1, etc.  MR runs successfully but what=20
>> happens is that data locality is not ensured i.e. if files A,B,C are=20
>> kept on
>> slave1 and D,E,F on slave2 then according to data locality, map tasks=20
>> should be submitted such that map task of A,B,C are submitted to=20
>> TaskTracker running on slave1 and D,E,F on slave2. Instead of this,=20
>> it randomly schedules the map task to any of the tasktrackers. If map=20
>> task of file A is submitted to TaskTracker running on slave2 then it=20
>> implies that file A is being fetched over the network by slave2.
>>
>>
>>
>> How do I avoid this from happening?
>>
>>
>>
>> Thanks,
>>
>> Nikhil
>>
>>
>>
>>
>
>
>
> --
> Harsh J


--
Harsh J