hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Enis Soztutar <enis.soz.nu...@gmail.com>
Subject Re: [Lucene-hadoop Wiki] Update of "FAQ" by DevarajDas
Date Fri, 07 Sep 2007 14:41:54 GMT
Hadoop has been used in quite varying cluster sizes (in the range 
1-2000), so am strongly in favor of as much automatic configuration as 
possible.

Doug Cutting wrote:
> Raghu Angadi wrote:
>> Right now Namenode does not know about the cluster size before 
>> starting IPC server.
>
> Sounds like perhaps we should make the handler count, queue size, etc. 
> dynamically adjustable, e.g., by adding Server methods for 
> setHandlerCount(), setQueueSize(), etc.  There's been talk of trying 
> to automatically adjust these within Server.java, based on load, and 
> that would be better yet, but short of that, we might adjust them 
> heuristically based on cluster size.
>
> The urgent thing, since we expect the best settings for large clusters 
> to change, is to make it so that folks don't need to adjust these 
> manually, even if the automation is an ill-understood heuristic.  I 
> think we can easily get some workable heuristics into 0.15, but we 
> might not get be able to implement async responses or figure out how 
> to adjust it automatically in Server.java or whatever in that 
> timeframe.  Perhaps we should just change the defaults to be big 
> enough for 2000 nodes, but that seems like too big of a hammer.
>
> Doug
>

Mime
View raw message