hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <praveen.pe...@nokia.com>
Subject controlling no. of mapper tasks
Date Mon, 20 Jun 2011 19:24:59 GMT
Hi there,
I know client can send "mapred.reduce.tasks" to specify no. of reduce tasks and hadoop honours
it but "mapred.map.tasks" is not honoured by Hadoop. Is there any way to control number of
map tasks? What I noticed is that Hadoop is choosing too many mappers and there is an extra
overhead being added due to this. For example, when I have only 10 map tasks, my job finishes
faster than when Hadoop chooses 191 map tasks. I have 5 slave cluster and 10 tasks can run
in parallel. I want to set both map and reduce tasks to be 10 for max efficiency.


View raw message