hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <ar...@yahoo-inc.com>
Subject Re: Override mapred.tasktracker.tasks.maximum?
Date Thu, 07 Feb 2008 22:50:05 GMT

On Feb 7, 2008, at 7:23 AM, Steve Schlosser wrote:

> Hello all
>
> Is it possible for a Hadoop program to override
> mapred.tasktracker.tasks.maximum at runtime?  I've found that my job
> overloads our nodes when running our default 8 tasks per node, but if
> I decrease mapred.tasktracker.tasks.maximum in the config file and
> restart the cluster it seems to go fine.  However, when I tried to set
> this parameter in my program it seemed to be ignored.

No, mapred.tasktracker.tasks.maximum (mapred.tasktracker.{map|reduce} 
tasks.maximum in 0.16.0) is a property of the cluster and not a job- 
specific property. Hence you cannot override it.

As Jason mentioned, you could use Haddop-On-Demand (a resource  
utilization framework coming in 0.16.0) to launch a new cluster per  
job and then configure each job (and hence cluster) differently...

Arun

> I'd like to set
> this at runtime rather than having to restart the cluster.  Is this
> only read when tasktrackers are started or can I override it?
>
> Thanks!
>
> -steve


Mime
View raw message