hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mingjian Deng <koven2...@gmail.com>
Subject Re: hadoop / hbase /zookeeper architecture for best performance
Date Tue, 21 Jun 2011 12:13:36 GMT
    Big memory is good for read. Because blockcache use 25% of heapsize
default. You can change "hfile.block.cache.size" to 0.4 or more.

    I don't use HBASE_HEAPSIZE, I use the follow:
" export HBASE_MASTER_OPTS="-Xmx8000m"
 export HBASE_ZOOKEEPER_OPTS="-Xmx1000m"
 export HBASE_REGIONSERVER_OPTS="-Xmx16000m"
"
    They will rewrite HEAPSIZE for different processes.

2011/6/21 Andre Reiter <a.reiter@web.de>

> Hi Mingjian,
>
> so if i understand it right, the region servers should get as much memory
> as possible, correct?
> at the moment our situation: the default amount of 1000 MB is used for the
> heap size (HBASE_HEAPSIZE)  on region servers
>
> so if we ran namenode, tasktracker and regionserver on the same hardware,
> how should the memory distribution look like?
> lets say for example 60% of memory for regionserver, 20% for namenode and
> 20% for tasktracker? is that a proper configuration?
>
> you use 16GB for the region server, is that an amound we should go for?
>
> thanks iin advance
> andre
>
>
>
> schrieb Mingjian Deng:
>
>  Hi andre:
>>     I think datanode and regionserver is in the same node is better.
>> Because
>> it will useful for locality.
>>     zookeeper is lightly, so you can share them in your cluster with other
>> processes. But in my opinion, you can running them in an extra cluster if
>> there are more than 1 hbase cluster use the same zookeeper.
>>
>>     In our cluster, we have 43 nodes with 3 clusters. Two of them have 10
>> regionservers and datanodes nodes and another has 20 regionservers and
>> datanodes. And all the 3 clusters use 1 zookeeper cluster with 5 nodes
>> running in virtual machines.
>>
>>     Each machine in the 43 nodes have 12 disks, one is 1TB.
>>     Memory: Datanode: 2GB HMaster:8GB Regionserver: 16GB. Zookeeper: 1GB.
>>
>>     The performance is pretty good.
>>
>>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message