hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amar Kamat <ama...@yahoo-inc.com>
Subject Re: How can reducers start before the mappers have finished?
Date Tue, 04 Mar 2008 03:59:10 GMT
On Mon, 3 Mar 2008, Paco NATHAN wrote:

> Just a side note:
> If a combiner is used, that usually would be considered as part of the
> cost of the mapper phase, right?
>
Yes.
>
> On Mon, Mar 3, 2008 at 12:17 PM, Amar Kamat <amarrk@yahoo-inc.com> wrote:
>> Currently reducers go through 2 phases
>>  1) Shuffle phase : copying the map outputs
>>  2) Reduce phase : Actual reducing
>>  So by starting the reducers we actually start the shuffle phase. Ideally
>>  the shuffle phase should interleaved with the map phase.
>>  Amar
>>
>>
>> On Mon, 3 Mar 2008, Marc Harris wrote:
>>
>> > I noticed when reading http://wiki.apache.org/hadoop/HardwareBenchmarks
>> > the following comment:
>> >
>> > "I ran into some odd behavior on Herd2 where if i [ . . . ] the reducers
>> > don't start until the mappers finish, slowing the job significantly."
>> >
>

Mime
View raw message