hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhang, Guibin" <gzh...@freewheel.tv>
Subject How to configure the hadoop to distribute less tasks to weak nodes?
Date Thu, 28 Feb 2008 03:29:53 GMT
Hi,all

         I find that when the sub-nodes' hardware configurations are
different, some nodes are strong(more cpus and more memory), others are
weak (leas cpus and less memory), when I run the job, the task are
almost evenly distributed to all the sub-nodes. This makes the weak
nodes pretty slow and a lot of tasks on the weak nodes are killed. This
may lead the whole job processing becoming slow, I am sure, because a
lot of tasks(more than 10 tasks) are processed twice.

         Question: How can I configure the hadoop to distribute less
tasks to weak nodes and distribute more tasks to strong nodes?

 

I configure the strong nodes with 

"mapred.tasktracker.map.tasks.maximum=75", 

"mapred.map.tasks=60", 

"mapred.tasktracker.reduce.tasks.maximum=18", 

"mapred.reduce.tasks=15" 

 

and the weak nodes with 

"mapred.tasktracker.map.tasks.maximum=60", 

"mapred.map.tasks=45", 

"mapred.tasktracker.reduce.tasks.maximum=15", 

"mapred.reduce.tasks=12"

 

I have 4 nodes totally. One for name node and job tracker, the others
are for sub-nodes.

 

Thanks.

 

Guibin zhang


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message