hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Load balancing requests in HDFS
Date Sun, 16 Oct 2011 12:15:47 GMT
Bharath,

You may want to take a look at
http://issues.apache.org/jira/browse/HDFS-782 and continue the
discussion on that.

On Sun, Oct 16, 2011 at 7:23 AM, Bharath Ravi <bharathravi1@gmail.com> wrote:
> Hi all,
>
> I have a question about how HDFS load balances requests for files/blocks:
>
> HDFS currently distributes data blocks randomly, for balance.
> However, if certain files/blocks are more popular than others, some nodes
> might get an "unfair" number of requests.
> Adding more replicas for these popular files might not help, unless HDFS
> explicitly distributes requests fairly among the replicas.
>
> Looking around, I saw a bit of discussion on this dating to 2009,
> Is something like this is done by HDFS today?
>
> http://www.mail-archive.com/core-user@hadoop.apache.org/msg04357.html
> http://grokbase.com/p/hadoop.apache.org/common-dev/2009/02/re-hdfs-load-traffic-balancing/18en5o4z2j4b6eaqnemkp4eppd7q
>
> Thanks a lot in advance!
> --
> Bharath Ravi
>



-- 
Harsh J

Mime
View raw message