hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ferdy Galema <ferdy.gal...@kalooga.com>
Subject Re: distcp performing much better for rebalancing than dedicated balancer
Date Thu, 05 May 2011 18:55:10 GMT
I actually tried 1% right after I ran the balancer with the default 
threshold. The data moved was the same as default. So in short this is 
what I tried:

default: fast (lots of data moved; 30 to 40GB every iteration)
1%: fast (same as above)
0.01%: slow (it moves only 1.26GB in a iteration, for hours long the 
exact same amount)

At the moment the cluster is already fully balanced.

On 05/05/2011 05:46 PM, Eric Fiala wrote:
> Ferdy - that is interesting.
> I would expect lower threshold = more data to move around (or equal to
> default 10%)
>
> Try with a whole integer, we regularly run balancer, -threshold 1 (to
> balance to 1%), maybe the decimal is throwing a wrench at hadoop.
>
> EF
>
> On 5 May 2011 09:27, Ferdy Galema<ferdy.galema@kalooga.com>  wrote:
>
>> I figured out what caused the slow balancing. Starting the balancer with a
>> too small threshold will decrease the speed dramatically:
>>
>> ./start-balancer.sh -threshold 0.01
>> 2011-05-05 17:17:04,132 INFO
>> org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 1.26 GBbytes in
>> this iteration
>> 2011-05-05 17:17:36,684 INFO
>> org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 1.26 GBbytes in
>> this iteration
>> 2011-05-05 17:18:09,737 INFO
>> org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 1.26 GBbytes in
>> this iteration
>> 2011-05-05 17:18:41,977 INFO
>> org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 1.26 GBbytes in
>> this iteration
>>
>> as opposed to:
>>
>> ./start-balancer.sh
>> 2011-05-05 17:19:01,676 INFO
>> org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 40 GBbytes in
>> this iteration
>> 2011-05-05 17:21:36,800 INFO
>> org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 30 GBbytes in
>> this iteration
>> 2011-05-05 17:24:13,191 INFO
>> org.apache.hadoop.hdfs.server.balancer.Balancer: Will move 30 GBbytes in
>> this iteration
>>
>> I'd expect setting the granularity would not affect speed, just the
>> stopping threshold. Perhaps a bug?
>>
>>
>> On 05/05/2011 03:43 PM, Ferdy Galema wrote:
>>
>>> The decommissioning was performed with solely refreshNodes, but that's
>>> somewhat irrelevant because the balancing tests were performed after I
>>> re-added the 11 empty nodes. (FYI the drives were formatted with another
>>> unix fs). Though I did notice that the decommissioning shows about the same
>>> metrics as that of the balancer test afterwards,  not very fast that is.
>>>
>>> On 05/05/2011 02:57 PM, Mathias Herberts wrote:
>>>
>>>> Did you explicitely start a balancer or did you decommission the nodes
>>>> using dfs.hosts.exclude and a dfsadmin -refreshNodes?
>>>>
>>>> On Thu, May 5, 2011 at 14:30, Ferdy Galema<ferdy.galema@kalooga.com>
>>>>   wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> On our 15node cluster (1GB ethernet and 4x1TB disk per node) I noticed
>>>>> that
>>>>> distcp does a much better job at rebalancing than the dedicated balancer
>>>>> does. We needed to decommision 11 nodes, so that prior to rebalancing
we
>>>>> had
>>>>> 4 used and 11 empty nodes. The 4 used nodes had about 25% usage each.
>>>>> Most
>>>>> of our files are of average size: We have about 500K files in 280K
>>>>> blocks
>>>>> and 800K blocks total (blocksize is 64MB).
>>>>>
>>>>> So I changed dfs.balance.bandwidthPerSec to 800100100 and restarted the
>>>>> cluster. Started the balancer tool and I noticed that the it moved about
>>>>> 200GB in 1 hour. (I grepped the balancer log for "Need to move").
>>>>>
>>>>> After stopping the balancer I started a distcp.  This tool copied 900GB
>>>>> in
>>>>> just 45 minutes, with an average replication of 2 so it's total
>>>>> throughput
>>>>> was around 2.4 TB/hour. Fair enough, it is not purely rebalancing
>>>>> because
>>>>> the 4 overused nodes also get new blocks, still it performs much better.
>>>>> Munin confirms the much higher disk/ethernet throughputs of the distcp.
>>>>>
>>>>> Are these characteristics to be expected? Either way, can the balancer
>>>>> be
>>>>> boosted even more? (Aside the dfs.balance.bandwidthPerSec property).
>>>>>
>>>>> Ferdy.
>>>>>
>>>>>

Mime
View raw message