hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From schubert zhang <zson...@gmail.com>
Subject Re: HDFS unbalance issue. (HBase over HDFS)
Date Wed, 25 Mar 2009 10:37:46 GMT
>From another point of view, I think HBase cannot control to delete blocks on
which node, it would just delete files, and HDFS delete blocks where the
blocks locating.

Schubert

On Wed, Mar 25, 2009 at 6:28 PM, schubert zhang <zsongbo@gmail.com> wrote:

> Thanks Ryan. Balancer may take a long time.
>
> The number of block are too different. But maybe it is caused by HBase not
> deleting garbage blocks on regionserver1 and regionserver2 and maybe others.
>
> We grep the logs of hadoop and find there is no any "deleting block" in
> node1 and node2.
>
> Following is the grep (grep -c "ask 10.24.1.1?:50010 to delete") result of
> hasoop logs:
>
> namenode:
>
> -----grep -c "ask 10.24.1.12:50010 to delete"-----node1
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
> 4754
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
> 1062
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.12:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 0
>
> -----grep -c "ask 10.24.1.14:50010 to delete"-----node2
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 1494
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
> 3305
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
> 3385
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.14:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 1494
>
> -----grep -c "ask 10.24.1.16:50010 to delete"-----node3
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
> 8022
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
> 8238
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.16:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 4302
>
> -----grep -c "ask 10.24.1.18:50010 to delete"-----node4
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
> 8591
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
> 9111
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.18:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 5038
>
> -----grep -c "ask 10.24.1.20:50010 to delete"-----node5
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-23
> 3794
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log.2009-03-24
> 3946
> [schubert@nd0-rack0-cloud logs]$ grep -c "ask 10.24.1.20:50010 to delete"
> hadoop-schubert-namenode-nd0-rack0-cloud.log
> 2989
>
> So, I think it may caused by HBase.
> I just grep the log of the zero "delete block" node. and find:
> [schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
> hadoop-schubert-datanode-nd1-rack0-cloud.log.2009-03-24
> 104739
> [schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
> hadoop-schubert-datanode-nd1-rack0-cloud.log.2009-03-23
> 465927
> [schubert@nd1-rack0-cloud logs]$ grep -c "Deleting block"
> hadoop-schubert-datanode-nd1-rack0-cloud.log
> 0
>
>
>
>
> On Wed, Mar 25, 2009 at 5:14 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>
>> Try
>> hadoop/bin/start-balancer.sh
>>
>> HDFS doesnt auto-balance.  Balancing in HDFS requires moving data around,
>> whereas balancing in HBase just means opening a file on a different
>> machine.
>>
>> On Wed, Mar 25, 2009 at 2:12 AM, schubert zhang <zsongbo@gmail.com>
>> wrote:
>>
>> > Hi all,
>> > I am using hbase-0.19.1 and hadoop-0.19.
>> > My cluster have 5+1 nodes, and there are about 512 regions in HBase
>> (256MB
>> > per region).
>> >
>> > But I found the blocks in HDFS is very unbalanced. Following is the
>> status
>> > from HDFS web GUI.
>> >
>> > (Node: I don't know if this mailing list can display html!)
>> >
>> > HDFS blocks:
>> > node1   509036 blocks
>> > node2   157937 blocks
>> > node3   15783   blocks
>> > node4   15117   blocks
>> > node5   20158   blocks
>> >
>> > But my HBase regions are very balanced.
>> > node1   88   regions
>> > node2   108 regions
>> > node3   111 regions
>> > node4   102 regions
>> > node5   105 regions
>> >
>> >
>> >
>> > NodeLast
>> > ContactAdmin StateConfigured
>> > Capacity (GB)Used
>> > (GB)Non DFS
>> > Used (GB)Remaining
>> > (GB)Used
>> > (%)Used
>> > (%)Remaining
>> > (%)Blocksnd1-rack0-cloud<
>> >
>> http://nd1-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>> > >
>> > 0In Service822.8578.6743.28200.8670.3324.41509036nd2-rack0-cloud<
>> >
>> http://nd2-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>> > >
>> > 0In Service822.8190.0242.96589.8223.0971.68157937nd3-rack0-cloud<
>> >
>> http://nd3-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>> > >
>> > 0In Service822.851.9542.61728.246.3188.5115783nd4-rack0-cloud<
>> >
>> http://nd4-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>> > >
>> > 6In Service822.846.1942.84733.775.6189.1815117nd5-rack0-cloud<
>> >
>> http://nd5-rack0-cloud:50075/browseDirectory.jsp?namenodeInfoPort=50070&dir=%2F
>> > >
>> > 1In Service1215.6152.3762.911100.324.3190.5220158
>> >
>> >
>> > But my HBase regions are very balanced.
>> >
>> > AddressStart CodeLoadnd1-rack0-cloud:60020 <
>> http://nd1-rack0-cloud:60030/>
>> > 1237967027050requests=383, regions=88, usedHeap=978, maxHeap=1991
>> > nd2-rack0-cloud:60020 <http://nd2-rack0-cloud:60030/
>> > >1237788871362requests=422,
>> > regions=108, usedHeap=1433,
>> > maxHeap=1991nd3-rack0-cloud:60020<http://nd3-rack0-cloud:60030/>
>> > 1237788881667requests=962, regions=111, usedHeap=1534, maxHeap=1991
>> > nd4-rack0-cloud:60020 <http://nd4-rack0-cloud:60030/
>> > >1237788859541requests=369,
>> > regions=102, usedHeap=1059,
>> > maxHeap=1991nd5-rack0-cloud:60020<http://nd5-rack0-cloud:60030/>
>> > 1237788899331requests=384, regions=105, usedHeap=1535,
>> > maxHeap=1983Total:servers:
>> > 5 requests=2520, regions=514
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message