hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bharath Ravi <bharathra...@gmail.com>
Subject Load balancing requests in HDFS
Date Sun, 16 Oct 2011 01:53:36 GMT
Hi all,

I have a question about how HDFS load balances requests for files/blocks:

HDFS currently distributes data blocks randomly, for balance.
However, if certain files/blocks are more popular than others, some nodes
might get an "unfair" number of requests.
Adding more replicas for these popular files might not help, unless HDFS
explicitly distributes requests fairly among the replicas.

Looking around, I saw a bit of discussion on this dating to 2009,
Is something like this is done by HDFS today?

http://www.mail-archive.com/core-user@hadoop.apache.org/msg04357.html
http://grokbase.com/p/hadoop.apache.org/common-dev/2009/02/re-hdfs-load-traffic-balancing/18en5o4z2j4b6eaqnemkp4eppd7q

Thanks a lot in advance!
-- 
Bharath Ravi

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message