hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rakesh Davanum <rakesh...@gmail.com>
Subject Re: Reduce Job Start
Date Wed, 23 Apr 2014 12:48:34 GMT
You can control when the reduce task even starts in the first place. The
parameter *mapred.reduce.slowstart.completed.maps *
specifies the fraction of the number of maps in the job which should be
complete before reduces are scheduled for the job. So for example if you
set this to .70 then reduce task will start after 70% of mappers are
completed.  Remember that the actual 'reduce' phase of the reducer will not
start until all the mappers are competed. Sometimes when this value is set
too low, reduce tasks starts even when there are a lot of mappers yet to
complete. This results in a lot of killed reduce task attempts as they are
waiting for the outputs from all the mappers to be available.

Thanks & Regards,
Rakesh


On Wed, Apr 23, 2014 at 7:43 AM, Chi Huynh <huynh@initions.com> wrote:

> The MapReduce-Job contains a shuffle phase, where the intermediary map
> outputs are copied to the reducer nodes. This phase of the job is assumed
> to be part of the reduce-phase, therefore. the counter already starts
> before the map-phase has finished. The actual reduce task will be started,
> just as you have heard, when all the map tasks are finished.
>
>
> On Wednesday, April 23, 2014 1:18:40 PM UTC+2, Kishore kumar wrote:
>>
>> Hi All,
>>
>> I heard about the reduce job, it will be started after all map tasks
>> finished 100%, but in my hive query the reduce job started at below stage,
>> please explain why is this.(I copied below line when the job is running).
>>
>> 2014-04-22 21:15:12,803 Stage-1 map = 83%, reduce = 1%, Cumulative CPU
>> 4194.4 sec
>>
>> --
>>
>>
>> *Kishore *
>>
>

Mime
View raw message