hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Rawson <ryano...@gmail.com>
Subject Re: setting of HBase, how much memory should be reserved for a Hbase regionserver.
Date Mon, 01 Mar 2010 09:29:46 GMT
The block cache is the memory limit used to keep blocks in the
regionserver memory to allow you to have faster operations. It's
literally a ram cache.

And yes, adding more regionservers helps :-)  Hooray for near-linear
scalability.

On Mon, Mar 1, 2010 at 12:26 AM, steven zhuang <zhuangxin8448@gmail.com> wrote:
> thanks, Ryan,
>            I think I can enlarge the heap size a little, and meanwhile
> shrink the write buffer sounds good too, but I am not sure we can disable
> the block cache, since the table is already built, anyway I can do that?
>
>            I still have the question that if I add more regionservers, will
> the load on current nodes be reduced to make the cluster run OK?
>
>
>
> On Mon, Mar 1, 2010 at 4:08 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>
>> I would consider doing the following things:
>>
>> - disable the block cache - you dont have ram to spare. 4gb is really
>> minimum (my laptop is 4gb ram and i decry how sucky I am occasionally)
>> - run between 1000-2000m, with the cache disabled you are in better shape
>> - consider adjusting the config values:
>>    <name>hbase.regionserver.global.memstore.upperLimit</name>
>>    <value>0.4</value>
>>
>> This implies that memstore (the write buffer) can occupy 40% of heap.
>> This might be too aggressive with a small heap and CMS.  Try changing
>> that to 0.2 and:
>>
>>    <name>hbase.regionserver.global.memstore.lowerLimit</name>
>>    <value>0.35</value>
>>
>> to 0.10 or so.
>>
>> Good luck!
>> -ryan
>>
>> On Sun, Feb 28, 2010 at 11:59 PM, steven zhuang <zhuangxin8448@gmail.com>
>> wrote:
>> > hi, all,
>> >          We have some PCs(a dozens of them) with  4 cores CPU and 4GB
ram
>> > each, a hadoop instance is already running on these machines.
>> >
>> >          Currently there is around 1.2GB ram left on every node.
>> >
>> >          We want to setup a Hbase instance on these machines, with 3
>> quorum
>> > servers and 5 region servers, but we kept getting "out of memory" error
>> when
>> > the table grows big on some nodes.
>> >
>> >          I know HBase is really a memory eater, my question is can we set
>> > up a Hbase cluster on these PC if we add more region servers? how much
>> ram
>> > should we reserve to make hbase run happily?
>> >
>> >          An option is we put hbase regionservers and Hadoop
>> > datanode/tasktracker on different node respectively(Not tried yet), but I
>> > don't know how much this would influence the performance, as we can not
>> > benefit from the data locality.
>> >
>> > --
>> >       best wishes.
>> >                            steven
>> >
>>
>
>
>
> --
>       best wishes.
>                            steven
>

Mime
View raw message