hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Owen O'Malley <...@yahoo-inc.com>
Subject Re: Starting up a larger cluster
Date Sun, 10 Feb 2008 06:53:32 GMT

On Feb 8, 2008, at 9:32 AM, Jeff Eastman wrote:

> I noticed that phenomena right off the bat. Is that a designed  
> "feature"
> or just an unhappy consequence of how blocks are allocated?

It was driven by a desire to maximize HDFS write throughput, which  
has unfortunate effects in the case of a small set of nodes uploading  
data.

We are going to be exploring different block placements and looking  
at what happens to performance as we do so. The current block  
allocations look like:

local node -> rack local -> off rack

My personal inclination is for something like:

rack local -> off rack -> other node on second rack

That will dramatically cut down on both the node and rack hotspots  
without killing your write performance, because you still only have  
one write through the network core.

-- Owen

Mime
View raw message