Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hbase-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of ryanobjc@gmail.com designates
 209.85.223.172 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=kzPAic8wyuAX2S4E5LwHCExMlE+x31Pzmgj5Mn+j67uEq8HIolam9QhchZYAq4mos1
         HqlsVQAPu6wc7CK7Px0c0pqQD7Ak2KU0fwy/9WRCSaAXCkhX+ubwHHroUqhDgZDaV8zf
         6GoZMSE6+xPZ8tbeMCh5Ooc98boWNe787eCcE=
MIME-Version: 1.0
In-Reply-To: <a549c83b1003010026r1a3265f4hea6d8193c1d7cefc@mail.gmail.com>
References: <a549c83b1002282359y66b9b19bx32ce093112f78315@mail.gmail.com>
	 <78568af11003010008u125c2f1dga148efacb390eeee@mail.gmail.com>
	 <a549c83b1003010026r1a3265f4hea6d8193c1d7cefc@mail.gmail.com>
Date: Mon, 1 Mar 2010 01:29:46 -0800
Message-ID: <78568af11003010129k29ab248fle0adb1ee5e09203f@mail.gmail.com>
Subject: Re: setting of HBase, how much memory should be reserved for a Hbase
	regionserver.
From: Ryan Rawson <ryanobjc@gmail.com>
To: hbase-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

The block cache is the memory limit used to keep blocks in the
regionserver memory to allow you to have faster operations. It's
literally a ram cache.

And yes, adding more regionservers helps :-)  Hooray for near-linear
scalability.

On Mon, Mar 1, 2010 at 12:26 AM, steven zhuang <zhuangxin8448@gmail.com> wr=
ote:
> thanks, Ryan,
> =A0 =A0 =A0 =A0 =A0 =A0I think I can enlarge the heap size a little, and =
meanwhile
> shrink the write buffer sounds good too, but I am not sure we can disable
> the block cache, since the table is already built, anyway I can do that?
>
> =A0 =A0 =A0 =A0 =A0 =A0I still have the question that if I add more regio=
nservers, will
> the load on current nodes be reduced to make the cluster run OK?
>
>
>
> On Mon, Mar 1, 2010 at 4:08 PM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>
>> I would consider doing the following things:
>>
>> - disable the block cache - you dont have ram to spare. 4gb is really
>> minimum (my laptop is 4gb ram and i decry how sucky I am occasionally)
>> - run between 1000-2000m, with the cache disabled you are in better shap=
e
>> - consider adjusting the config values:
>> =A0 =A0<name>hbase.regionserver.global.memstore.upperLimit</name>
>> =A0 =A0<value>0.4</value>
>>
>> This implies that memstore (the write buffer) can occupy 40% of heap.
>> This might be too aggressive with a small heap and CMS. =A0Try changing
>> that to 0.2 and:
>>
>> =A0 =A0<name>hbase.regionserver.global.memstore.lowerLimit</name>
>> =A0 =A0<value>0.35</value>
>>
>> to 0.10 or so.
>>
>> Good luck!
>> -ryan
>>
>> On Sun, Feb 28, 2010 at 11:59 PM, steven zhuang <zhuangxin8448@gmail.com=
>
>> wrote:
>> > hi, all,
>> > =A0 =A0 =A0 =A0 =A0We have some PCs(a dozens of them) with =A04 cores =
CPU and 4GB ram
>> > each, a hadoop instance is already running on these machines.
>> >
>> > =A0 =A0 =A0 =A0 =A0Currently there is around 1.2GB ram left on every n=
ode.
>> >
>> > =A0 =A0 =A0 =A0 =A0We want to setup a Hbase instance on these machines=
, with 3
>> quorum
>> > servers and 5 region servers, but we kept getting "out of memory" erro=
r
>> when
>> > the table grows big on some nodes.
>> >
>> > =A0 =A0 =A0 =A0 =A0I know HBase is really a memory eater, my question =
is can we set
>> > up a Hbase cluster on these PC if we add more region servers? how much
>> ram
>> > should we reserve to make hbase run happily?
>> >
>> > =A0 =A0 =A0 =A0 =A0An option is we put hbase regionservers and Hadoop
>> > datanode/tasktracker on different node respectively(Not tried yet), bu=
t I
>> > don't know how much this would influence the performance, as we can no=
t
>> > benefit from the data locality.
>> >
>> > --
>> > =A0 =A0 =A0 best wishes.
>> > =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0steven
>> >
>>
>
>
>
> --
> =A0 =A0 =A0 best wishes.
> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0steven
>