hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marco Nicosia <ma...@yahoo-inc.com>
Subject Re: Starting up a larger cluster
Date Tue, 12 Feb 2008 15:08:29 GMT
DFS should place one replica per rack:
http://issues.apache.org/jira/browse/HADOOP-2559

On 2/9/08 22:53, "Owen O'Malley" <oom@yahoo-inc.com> wrote:

> 
> On Feb 8, 2008, at 9:32 AM, Jeff Eastman wrote:
> 
>> I noticed that phenomena right off the bat. Is that a designed
>> "feature"
>> or just an unhappy consequence of how blocks are allocated?
> 
> It was driven by a desire to maximize HDFS write throughput, which
> has unfortunate effects in the case of a small set of nodes uploading
> data.
> 
> We are going to be exploring different block placements and looking
> at what happens to performance as we do so. The current block
> allocations look like:
> 
> local node -> rack local -> off rack
> 
> My personal inclination is for something like:
> 
> rack local -> off rack -> other node on second rack
> 
> That will dramatically cut down on both the node and rack hotspots
> without killing your write performance, because you still only have
> one write through the network core.
> 
> -- Owen
> 

-- 
   Marco Nicosia - Grid Services Ops
   Systems, Tools, and Services Group



Mime
View raw message