hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley" <o...@yahoo-inc.com>
Subject Re: Estimating number of worker nodes
Date Tue, 14 Feb 2006 15:56:14 GMT

On Feb 13, 2006, at 9:48 PM, Eric Baldeschwieler wrote:

> Some of our discussed future work may make this impractical.  The 
> number of available workers may become a variable that depends on 
> priority, other parallel work etc.
>
> Perhaps it is best to express your requirements in terms of the input 
> or the output, for example size of input per job?

I think that in the short term, it is worthwhile letting the 
application see the current size of the cluster (and the current number 
of maps and reduces). I created a patch that does this and submitted it 
at:

http://issues.apache.org/jira/browse/HADOOP-37

Even if there is eventually a meta scheduler, it still makes sense to 
ask the question, you just can't assume that the size of the cluster is 
constant.

-- Owen


Mime
View raw message