hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Steve Loughran <ste...@apache.org>
Subject Re: Load balancing requests in HDFS
Date Tue, 18 Oct 2011 16:37:16 GMT
On 16/10/11 02:53, Bharath Ravi wrote:
> Hi all,
> I have a question about how HDFS load balances requests for files/blocks:
> HDFS currently distributes data blocks randomly, for balance.
> However, if certain files/blocks are more popular than others, some nodes
> might get an "unfair" number of requests.
> Adding more replicas for these popular files might not help, unless HDFS
> explicitly distributes requests fairly among the replicas.

Have a look at the ReplicationTargetChooser class; it does take datanode 
load into account, though it's concern is distribution for data 
availability, not performance.

The standard technique for popular files -including MR job JAR files- is 
to over-replicate. One problem: how to determine what is popular without 
adding more load on the namenode

View raw message