hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@yahoo-inc.com>
Subject Re: Lazy initialization of Reducers
Date Wed, 21 Jul 2010 22:22:59 GMT
Moving to mapreduce-user@, bcc general@. Please do not use the  
general@ list for project specific discussions.


On Jul 21, 2010, at 10:15 AM, Syed Wasti wrote:
> It says “:In M/R job Reducers are initialized with Mappers at the  
> job initialization, but the reduce method is called in reduce phase  
> when all the maps had been finished. So in large jobs where Reducer  
> loads data (>100 MB for business logic) in-memory on initialization,  
> the performance can be increased by lazily initializing Reducers  
> i.e. loading data in reduce method controlled by an initialize flag  
> variable which assures that it is loaded only once. By lazily  
> initializing Reducers which require memory (for business logic) on  
> initialization, number of maps can be increased.”

The part about 'loading data in reduce method controlled by an  
initialize flag variable which assures that it is loaded only once'  
makes no sense to me.

However, you can 'slowstart' reduces by ensuring sufficient maps are  
complete before _any_ reduces are launched... from mapred-default.xml:

<property>
   <name>mapred.reduce.slowstart.completed.maps</name>
   <value>0.05</value>
   <description>Fraction of the number of maps in the job which should  
be
   complete before reduces are scheduled for the job.
   </description>
</property>

Arun



Mime
View raw message