hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hiroyuki Yamada <mogwa...@gmail.com>
Subject Re: Question about the task assignment strategy
Date Wed, 12 Sep 2012 09:44:50 GMT
Hi,

Thank you for replaying the experiments.

I launched a job through hive with default TextInputFormat.
The job is TPC-H Q1 query, which is a simple selection query for lineitem table.
The each size of data (data01...data14) is about 300GB, so about
4.2TB(=300GB*14) in total.

I really appreciate your help.

Regards,
Hiroyuki


On Wed, Sep 12, 2012 at 2:47 PM, Hemanth Yamijala
<yhemanth@thoughtworks.com> wrote:
> Hi,
>
> I tried a similar experiment as yours but couldn't replicate the issue.
>
> I generated 64 MB files and added them to my DFS - one file from every
> machine, with a replication factor of 1,  like you did. My block size was
> 64MB. I verified the blocks were located on the same machine as where I
> added them from.
>
> Then, I launched a wordcount (without the min split size config). As
> expected, it created 8 maps, and I could verify that it ran all the tasks as
> data local - i.e. every task read off its own datanode. From the launch
> times of the tasks, I could roughly feel that this scheduling behaviour was
> independent of the order in which the tasks were launched. This behaviour
> was retained even with the min split size config.
>
> Could you share the size of input you generated (i.e the size of
> data01..data14) ? Also, what job are you running - specifically what is the
> input format ?
>
> BTW, this wiki entry: http://wiki.apache.org/hadoop/HowManyMapsAndReduces
> talks a little bit about how the maps are created.
>
> Thanks
> Hemanth
>
> On Wed, Sep 12, 2012 at 7:49 AM, Hiroyuki Yamada <mogwaing@gmail.com> wrote:
>>
>> I figured out the cause.
>> HDFS block size is 128MB, but
>> I specify mapred.min.split.size as 512MB,
>> and data local I/O processing goes wrong for some reason.
>> When I remove the mapred.min.split.size configuration,
>> tasktrackers pick data-local tasks.
>> Why does it happen ?
>>
>> It seems like a bug.
>> Split is a logical container of blocks,
>> so nothing is wrong logically.
>>
>> On Wed, Sep 12, 2012 at 1:20 AM, Hiroyuki Yamada <mogwaing@gmail.com>
>> wrote:
>> > Hi, thank you for the comment.
>> >
>> >> Task assignment takes data locality into account first and not block
>> >> sequence.
>> >
>> > Does it work like that when replica factor is set to 1 ?
>> >
>> > I just had a experiment to check the behavior.
>> > There are 14 nodes (node01 to node14) and there are 14 datanodes and
>> > 14 tasktrackers working.
>> > I first created a data to be processed in each node (say data01 to
>> > data14),
>> > and I put the each data to the hdfs from each node (at /data
>> > directory. /data/data01, ... /data/data14).
>> > Replica factor is set to 1, so according to the default block placement
>> > policy,
>> > each data is stored at local node. (data01 is stored at node01, data02
>> > is stored at node02 and so on)
>> > In that setting, I launched a job that processes the /data and
>> > what happened is that tasktrackers read from data01 to data14
>> > sequentially,
>> > which means tasktrackers first take all data from node01 and then
>> > node02 and then node03 and so on.
>> >
>> > If tasktracker takes data locality into account as you say,
>> > each tasktracker should take the local task(data). (tasktrackers at
>> > node02 should take data02 blocks if there is any)
>> > But, it didn't work like that.
>> > What this is happening ?
>> >
>> > Is there any documents about this ?
>> > What part of the source code is doing that ?
>> >
>> > Regards,
>> > Hiroyuki
>> >
>> > On Tue, Sep 11, 2012 at 11:27 PM, Hemanth Yamijala
>> > <yhemanth@thoughtworks.com> wrote:
>> >> Hi,
>> >>
>> >> Task assignment takes data locality into account first and not block
>> >> sequence. In hadoop, tasktrackers ask the jobtracker to be assigned
>> >> tasks.
>> >> When such a request comes to the jobtracker, it will try to look for an
>> >> unassigned task which needs data that is close to the tasktracker and
>> >> will
>> >> assign it.
>> >>
>> >> Thanks
>> >> Hemanth
>> >>
>> >>
>> >> On Tue, Sep 11, 2012 at 6:31 PM, Hiroyuki Yamada <mogwaing@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> I want to make sure my understanding about task assignment in hadoop
>> >>> is correct or not.
>> >>>
>> >>> When scanning a file with multiple tasktrackers,
>> >>> I am wondering how a task is assigned to each tasktracker .
>> >>> Is it based on the block sequence or data locality ?
>> >>>
>> >>> Let me explain my question by example.
>> >>> There is a file which composed of 10 blocks (block1 to block10), and
>> >>> block1 is the beginning of the file and block10 is the tail of the
>> >>> file.
>> >>> When scanning the file with 3 tasktrackers (tt1 to tt3),
>> >>> I am wondering if
>> >>> task assignment is based on the block sequence like
>> >>> first tt1 takes block1 and tt2 takes block2 and tt3 takes block3 and
>> >>> tt1 takes block4 and so on
>> >>> or
>> >>> task assignment is based on the task(data) locality like
>> >>> first tt1 takes block2(because it's located in the local) and tt2
>> >>> takes block1 (because it's located in the local) and
>> >>> tt3 takes block 4(because it's located in the local) and so on.
>> >>>
>> >>> As far as I experienced and the definitive guide book says,
>> >>> I think that the first case is the task assignment strategy.
>> >>> (and if there are many replicas, closest one is picked.)
>> >>>
>> >>> Is this right ?
>> >>>
>> >>> If this is right, is there any way to do like the second case
>> >>> with the current implementation ?
>> >>>
>> >>> Thanks,
>> >>>
>> >>> Hiroyuki
>> >>
>> >>
>
>

Mime
View raw message