hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Balaji Narayanan (பாலாஜி நாராயணன்) <li...@balajin.net>
Subject Re: disk used percentage is not symmetric on datanodes (balancer)
Date Sun, 24 Mar 2013 18:23:18 GMT
Are you running balancer? If balancer is running and if it is slow, try
increasing the balancer bandwidth


On 24 March 2013 09:21, Tapas Sarangi <tapas.sarangi@gmail.com> wrote:

> Thanks for the follow up. I don't know whether attachment will pass
> through this mailing list, but I am attaching a pdf that contains the usage
> of all live nodes.
>
> All nodes starting with letter "g" are the ones with smaller storage space
> where as nodes starting with letter "s" have larger storage space. As you
> will see, most of the "gXX" nodes are completely full whereas "sXX" nodes
> have a lot of unused space.
>
> Recently, we are facing crisis frequently as 'hdfs' goes into a mode where
> it is not able to write any further even though the total space available
> in the cluster is about 500 TB. We believe this has something to do with
> the way it is balancing the nodes, but don't understand the problem yet.
> May be the attached PDF will help some of you (experts) to see what is
> going wrong here...
>
> Thanks
> ------
>
>
>
>
>
>
>
> Balancer know about topology,but when calculate balancing it operates only
> with nodes not with racks.
> You can see how it work in Balancer.java in  BalancerDatanode about string
> 509.
>
> I was wrong about 350Tb,35Tb it calculates in such way :
>
> For example:
> cluster_capacity=3.5Pb
> cluster_dfsused=2Pb
>
> avgutil=cluster_dfsused/cluster_capacity*100=57.14% used cluster capacity
> Then we know avg node utilization (node_dfsused/node_capacity*100)
> .Balancer think that all good if  avgutil
> +10>node_utilizazation>=avgutil-10.
>
> Ideal case that all node used avgutl of capacity.but for 12TB node its
> only 6.5Tb and for 72Tb its about 40Tb.
>
> Balancer cant help you.
>
> Show me http://namenode.rambler.ru:50070/dfsnodelist.jsp?whatNodes=LIVEif you can.
>
>
>
>>
>>
>>  In ideal case with replication factor 2 ,with two nodes 12Tb and 72Tb
>> you will be able to have only 12Tb replication data.
>>
>>
>> Yes, this is true for exactly two nodes in the cluster with 12 TB and 72
>> TB, but not true for more than two nodes in the cluster.
>>
>>
>> Best way,on my opinion,it is using multiple racks.Nodes in rack must be
>> with identical capacity.Racks must be identical capacity.
>> For example:
>>
>> rack1: 1 node with 72Tb
>> rack2: 6 nodes with 12Tb
>> rack3: 3 nodes with 24Tb
>>
>> It helps with balancing,because dublicated  block must be another rack.
>>
>>
>> The same question I asked earlier in this message, does multiple racks
>> with default threshold for the balancer minimizes the difference between
>> racks ?
>>
>> Why did you select hdfs?May be lustre,cephfs and other is better choise.
>>
>>
>> It wasn't my decision, and I probably can't change it now. I am new to
>> this cluster and trying to understand few issues. I will explore other
>> options as you mentioned.
>>
>> --
>> http://balajin.net/blog
>> http://flic.kr/balajijegan
>>

Mime
View raw message