hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <...@hortonworks.com>
Subject Re: Variable mapreduce.tasktracker.*.tasks.maximum per job
Date Tue, 20 Dec 2011 00:31:17 GMT

The CapacityScheduler in 0.20.205 (in fact since 0.20.203) supports the notion of 'high memory
jobs' with which you can specify, for each job, the number of 'slots' for each map/reduce.
For e.g. you can say for job1 that each map needs 2 slots and so on.

Unfortunately, I don't know how well this works in 0.22 - I might be wrong, but I heavily
doubt it's been tested in 0.22. YMMV.

Hope that helps.


On Dec 19, 2011, at 3:02 PM, Markus Jelsma wrote:

> Hi,
> We have many different jobs running on a 0.22.0 cluster, each with its own memory consumption.
Some jobs can easily be run with a large amount of *.tasks per job and others require much
more memory and can only be run with a minimum number of tasks per node.
> Is there any way to reconfigure a running cluster on a per job basis so we can set the
heap size and number of mapper and reduce tasks per node? If not, we have to force all settings
to a level that is right for the toughest jobs which will have a negative impact on simpler
> Thoughts?
> Thanks

View raw message