hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bibek Paudel <eternalyo...@gmail.com>
Subject Re: disable pipelining in Hadoop
Date Tue, 01 Mar 2011 14:42:13 GMT
On Tue, Mar 1, 2011 at 3:27 PM, Benjamin Gufler <benjamin.gufler@tum.de> wrote:
> Hi Bikash,
>
> On 2011-03-01 15:13, bikash sharma wrote:
>>
>> Is there a way to disable the use of pipelining , i.e., the reduce phase
>> is
>> started only after the map phase is completed?
>
> you need to configure the mapred.reduce.slowstart.completed.maps property in
> mapred-site.xml. It gives the percentage of mappers which must be complete
> before the first reducers are launched. By setting it to 1, you should
> obtain the wanted behaviour.
>

I think this only schedules the reducers, and the scheduled reducers
start "copy" (followed by sort) stages. The actual "reduce" functions
are called only after all the intermediate data from all mappers have
been copied over.

-b

> Cheers,
>        Benjamin
>

Mime
View raw message