hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: Hadoop in a Heterogeneous Environment - taking advantage of different processor specs
Date Tue, 28 Jul 2009 15:23:49 GMT
Saptarshi Guha wrote:
> Hello,
> Not sure if this has been asked or answered.
> Suppose I have tasktrackers A1,A2,A3 each with 4 cores and 16GB ram.
> 
> mapred.tasktracker.map.tasks.maximum = 6
> mapred.tasktracker.reduce.tasks.maximum = 4
> 
> Now suppose I have one more machine(X) with 8 cores and 32GB ram.
> 
> Since (if i'm not mistaken) tasktrackers talk to the jobtracker, can I take
> advantage of X by
> a) A1,A2,A3 each have the same hadoop-site.xml with the above values for
> mapred.*
> b.0) on B, i have a hadoop-site.xml with these valus
> 
> mapred.tasktracker.map.tasks.maximum = 5
> mapred.tasktracker.reduce.tasks.maximum = 3
> 
> and start * one * tasktracker.
> b.1) Then edit hadoop-site.xml, change the tasktracker port (if there is
> such a thing) and
> 
> b.2) start *another* tasktracker.
> 
> We can skip b.1) if there is no such thing as a tasktracker port.
> Hence I wil have /two/ tasktrackers running on X, one on Ai.  and thus take
> advantage of X.
> 
> Is this at all possible? Or am i talking nonsense?

Why not just start the TT's on machine X with a different set of max 
values from the others?

Mime
View raw message