hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer <awittena...@linkedin.com>
Subject Re: slow transfer rates w/ HDFS balancer
Date Thu, 08 Oct 2009 19:06:43 GMT

On 10/8/09 12:03 PM, "David Howell" <dehowell@gmail.com> wrote:

> I just added a couple of new node to my cluster today, and ran the
> balancer to get some data onto them. With scp, the transfer rate
> between these machines is in the neighborhood of 50 MB/s... but the
> transfers during the balancing cap out at 1 MB/s.
> Is this to be expected, and if not, does anyone have any ideas of
> where to start looking for a problem?

Balancer was intended to run as a background job so that the grid isn't
limping while the balance is going on.  But this is a tunable: change the
dfs.balance.bandwidthPerSec setting. 

View raw message