hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean-Daniel Cryans <jdcry...@apache.org>
Subject Re: regionserver loads but never unload?
Date Fri, 05 Mar 2010 18:15:56 GMT
On Thu, Mar 4, 2010 at 7:19 PM, steven zhuang
<steven.zhuang.1984@gmail.com> wrote:
> thanks, J.D.
>
>              I am still not sure about the second question, from the log I
> can see lines like:
> *org.apache.hadoop.hbase.regionserver.Store: loaded
> /user/ccenterq/hbase/XXX/1702600912/queries/1289015788537930719,
> isReference=false, sequence id=1389720128, length=**175533391**,
> majorCompaction=true (this is the region data, not the index, right?)*
>              I do have some region really big, with millions of columns in
> one column family, but isn't this length a little too big.

The index and the metadata of the files of that Store in that region
was loaded here.

>
>
>        About the third one, I am actually not very clear of how memory is
> used in Hbase, if it's only the few KBs by holding region info, it won't
> release right?

I don't understand your question. Try an example?

>
>        Since we adding more regionserver into our cluster, the chance of
> OOME is much less, but still there are some(we have only 1GB heap for the
> regoinserver), I wonder which part of Hbase eats most of the heap
> memory? The region data takes most, right?

We recommend using at least 4GB on a production system. So what
occupies data is:

- The memstores (we have a global limit per region server).
- The block cache (global limit too)
- All the objects used to manage the regions

The block cache is an LRU, so once blocks of data are evicted they
occupy memory until garbage collected.

>        I think there should be some rule to cache/write-back the region
> data, but now it seems that the Hbase doesn't detect the Memory usage status
> thus cannot release the memory in time.

We do in fact. If you grow your block cache too much, it will start
evicting even more. If you get to the limit of memstore size, hbase
will force flush them to free space... then that's all garbage
collected by java.

>
>
> On Fri, Mar 5, 2010 at 10:00 AM, Jean-Daniel Cryans <jdcryans@apache.org>wrote:
>
>> Inline.
>>
>> J-D
>>
>> On Thu, Mar 4, 2010 at 5:53 PM, steven zhuang
>> <steven.zhuang.1984@gmail.com> wrote:
>> > greetings,
>> >
>> >             I have a cluster which contains a dozens of PCs with 4cores
>> and
>> > 4GB ram each, at first there were only 5 regionservers in the cluster,
>> now
>> > we have added 18 regionservers into the cluseter.
>> >             But after reading the log, I have some questions, hope you
>> guys
>> > can help me out.
>> >
>> >            1. the cluster will reassign the regions around old
>> > regionservers and newly added regionservers because of auto-balancing,
>> > right?
>>
>> Yep. It's a bit intense and we plan on making it less disruptive.
>>
>> >
>> >            2. I saw a lot of regions OPEN messages, does this means the
>> > regionserver will load the whole regoin into memory or just portion of
>> it,
>> > or just the index(?I don't know what is in the index data)?
>>
>> Just the region information and files indexes.
>>
>> >
>> >            3. will the memory be released after the loaded region opened
>> is
>> > not needed anymore, Coz I don't see anything from the log.
>>
>> Which memory? The few KBs used to store the region's information? Or
>> are you talking about the memory taken by regions that moved to other
>> region servers? In this case it's really just the java garbage
>> collector that takes care of it.
>>
>

Mime
View raw message