hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kalbande, Manish" <mkalba...@shopping.com>
Subject Task type priorities during scheduling ?
Date Thu, 20 Jul 2006 18:32:13 GMT
Hi,
 
I am running a cluster of 21 nodes.
while running any task I observed that reduce tasks are getting
scheduled much before all the map tasks are finished.
As a result, reduce tasks are waiting for map tasks to finish and total
time for map tasks is more because they are not getting scheduled
quickly.
 
It will be better if reduce tasks are scheduled only after there are no
map tasks to be performed.
 
For example, during generate job, we had total 544 map tasks and 41
reduce tasks.
All 41 reduce tasks got scheduled and only 42 map tasks could be
schedules at a time.
 
My current configuration
 
mapred.map.tasks = 83
mapred.reduce.tasks=41
mapred.tasktracker.tasks.maximum=2
 
Also, does "mapred.tasktracker.tasks.maximum" applies to per task type?
or is it for all tasks? From my observation is appears to be per task
type.
 
thanks
Manish
 

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message