hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@apache.org>
Subject Re: Multicore nodes
Date Mon, 01 Oct 2007 16:37:32 GMT
Toby DiPasquale wrote:
> In short, yes. Hadoop's code takes advantage of multiple native
> threads and you can tune the level of concurrency in the system by
> setting mapred.map.tasks and mapred.reduce.tasks to take advantage of
> multiple cores on the nodes which have them.

More importantly, you should set mapred.tasktracker.tasks.maximum 
according to the number of cores per node.  That parameter determines 
how many tasks will be run simultaneously per node.  Note that, at this 
point, this parameter is global for the cluster, and not independently 
configurable per node.  Someone with a heterogeneous cluster might be 
interested in fixing that someday...

Doug

Mime
View raw message