hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jimmy Wan" <ji...@indeed.com>
Subject Re: Limiting Total # of TaskTracker threads
Date Thu, 20 Mar 2008 22:49:29 GMT
On Tue, 18 Mar 2008 19:53:04 -0500, Ted Dunning <tdunning@veoh.com> wrote:

> I think the original request was to limit the sum of maps and reduces  
> rather than limiting the two parameters independently.

Ted, yes this is exactly what I'm looking for. I just found an issue that  
seems to state that the old deprecated property is there, but it is not  
documented:

https://issues.apache.org/jira/browse/HADOOP-2300

I tried using the max tasks in combination with setting the new values,  
but that didn't seem to work. =( My machine labelled as "LIMITED MACHINE"  
had 2 maps and 1 reduce running at the same time.

The scenario I have is that I want to run multiple concurrent jobs through  
my cluster and have the CPU usage for that node be bound. Should I file a  
new issue?

This was all with Hadoop 0.16.0

LIMITED MACHINE:
	<property>
	  <name>mapred.tasktracker.tasks.maximum</name>
	  <value>2</value>
	  <description>The maximum number of total tasks that will be run
	  simultaneously by a task tracker.
	  </description>
	</property>
	<property>
	  <name>mapred.tasktracker.map.tasks.maximum</name>
	  <value>1</value>
	  <description>The maximum number of map tasks that will be run
	  simultaneously by a task tracker.
	  </description>
	</property>
	<property>
	  <name>mapred.tasktracker.reduce.tasks.maximum</name>
	  <value>1</value>
	  <description>The maximum number of reduce tasks that will be run
	  simultaneously by a task tracker.
	  </description>
	</property>

OTHER CLUSTER MACHINES:
	<property>
	  <name>mapred.tasktracker.tasks.maximum</name>
	  <value>8</value>
	  <description>The maximum number of total tasks that will be run
	  simultaneously by a task tracker.
	  </description>
	</property>
	<property>
	  <name>mapred.tasktracker.map.tasks.maximum</name>
	  <value>4</value>
	  <description>The maximum number of map tasks that will be run
	  simultaneously by a task tracker.
	  </description>
	</property>
	<property>
	  <name>mapred.tasktracker.reduce.tasks.maximum</name>
	  <value>4</value>
	  <description>The maximum number of reduce tasks that will be run
	  simultaneously by a task tracker.
	  </description>
	</property>

> On 3/18/08 5:26 PM, "Arun C Murthy" <arunc@yahoo-inc.com> wrote:
>
>> The map/reduce tasks are not threads, they are run in separate JVMs
>> which are forked by the tasktracker.

Arun, yes, I did mean tasks, not threads.


-- 
Jimmy

Mime
View raw message