hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From stan lee <lee.stan...@gmail.com>
Subject Re: what's the mechnism to determine the reducer number and reduce progress
Date Tue, 18 May 2010 15:33:18 GMT
Thanks PanFeng, do you have more detailed explanation on this? Is it
caculated by how many reduce files has completed each phase?

Also, what's the answer for my second question? Thanks!

On Mon, May 17, 2010 at 12:44 PM, 原攀峰 <ypf412@163.com> wrote:

> For a reduce task, the execution is divided into three phases, each of
> which accounts for 1/3 of the score:
> • The copy phase, when the task fetches map outputs.
> • The sort phase, when map outputs are sorted by key.
> • The reduce phase, when a user-defined function is applied to the list of
> map outputs with each key.
> --
>
> Yuan Panfeng(原攀峰) | BeiHang University
>
> TEL: +86-13426166934
>
> MSN: ypf412@hotmail.com
>
> EMAIL: ypf412@gmail.com
>
> QQ: 362889262
>
>
>
>
> 在2010-05-17 09:44:38,"stan lee" <lee.stan065@gmail.com> 写道:
>  >When I run the sort job, I found when there are 70 reduce tasks running
> and
> >no one completed, the progress bar shows that it has finished about 80%,
> so
> >how the mapreduce mechnism to caculate this?
> >
> >Also,  when I run a job, as we know, we can determine the number of total
> >reduce tasks through setNumReduceTasks() function, but how to determine
> the
> >reducer number(I mean the tasktracker number which run the reduce task)
> >being used?
> >
> >Thanks!
> >Stan. Lee
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message