hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Usman Waheed <usm...@opera.com>
Subject Re: Balancing datanodes - Running hadoop 0.18.3
Date Mon, 27 Apr 2009 12:24:27 GMT
Hi Tamir,

Thanks for the info, makes sense now :).

Cheers,
Usman
> Hi,
>
> The balancer works with the average utilization of all the nodes in the
> cluster - in your case it's about 13%. Only nodes that are +/- 10% off the
> average will be rebalanced. Node 4 isn't under-utilized because 13-10=3
> which is less than 4%. You can use a different threshold than the default
> 10% (hadoop balancer -threshold 5). Read more here:
> http://hadoop.apache.org/core/docs/current/hdfs_user_guide.html#Rebalancer
>
> Tamir
>
>
> On Mon, Apr 27, 2009 at 11:36 AM, Usman Waheed <usmanw@opera.com> wrote:
>
>   
>> Hi,
>> I had sent out an email yesterday asking about how to balance the cluster
>> after setting the replication level to 2. I have 4 datanodes and one
>> namenode in my setup.
>> Using the -R switch with -setrep did the trick but one of my nodes became
>> under utilized. I then ran hadoop balancer and it did help but upto a
>> certain extent.
>>
>> Datanode 4 noted below is now up to almost 5% but when i try to balance the
>> datanode again using the "hadoop balance" command it says that the cluster
>> is already balanced which isnt.
>> I wonder if there is an alternate way(s) or maybe overtime Datanode-4 will
>> pick up more blocks?
>>
>> Any clues?
>>
>> Thanks,
>> Usman
>>
>> Name: 1
>> State          : In Service
>> Total raw bytes: 293778976768 (273.6 GB)
>> Remaining raw bytes: 222235858599(206.97 GB)
>> Used raw bytes: 48140136448 (44.83 GB)
>> % used: 16.39%
>> Last contact: Mon Apr 27 08:34:46 UTC 2009
>>
>>
>> Name: 2
>> State          : In Service
>> Total raw bytes: 293778976768 (273.6 GB)
>> Remaining raw bytes: 231235100994(215.35 GB)
>> Used raw bytes: 40704245760 (37.91 GB)
>> % used: 13.86%
>> Last contact: Mon Apr 27 08:34:45 UTC 2009
>>
>>
>> Name: 3
>> State          : In Service
>> Total raw bytes: 293778976768 (273.6 GB)
>> Remaining raw bytes: 211936026161(197.38 GB)
>> Used raw bytes: 59591700480 (55.5 GB)
>> % used: 20.28%
>> Last contact: Mon Apr 27 08:34:45 UTC 2009
>>
>>
>> *Name: 4
>> *State          : In Service
>> Total raw bytes: 293778976768 (273.6 GB)
>> Remaining raw bytes: 258876991693(241.1 GB)
>> Used raw bytes: 12142653440 (11.31 GB)
>> % used: 4.13%
>> Last contact: Mon Apr 27 08:34:46 UTC 2009
>>
>>
>>     
>
>   


Mime
View raw message