hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Rawson <ryano...@gmail.com>
Subject Re: memory usage too high in regionserver
Date Wed, 25 Aug 2010 22:13:18 GMT
Hey,

It's all over the place.  HeapSize is one interface.  There is also
stuff in HRegion, Store and Memstore that tracks memory used.  On top
of it the LruBlockCache does more stuff as well.

-ryan

On Wed, Aug 25, 2010 at 10:59 AM, Alex Baranau <alex.baranov.v@gmail.com> wrote:
> Hello guys,
>
>> HBase is a self memory tuning app
>
> Is there any documentation on that or if you could point to some sources I
> will be very thankful (even if sources are particular classes, please name
> at least few to look at).
>
> Thank you in advance,
> Alex Baranau.
>
> On Tue, Aug 10, 2010 at 1:25 AM, Ryan Rawson <ryanobjc@gmail.com> wrote:
>
>> Hey,
>>
>> HBase is a self memory tuning app, unlike most Java apps you may have
>> experience with.  We use as much memory as possible for memstore and
>> block cache, but we also flush memory to keep overall usage below
>> maximums.
>>
>> Enjoy,
>> -ryan
>>
>> On Mon, Aug 9, 2010 at 3:22 PM, Jinsong Hu <jinsong_hu@hotmail.com> wrote:
>> > you are right. now it shows up as
>> >
>> > request=180.5, regions=60, stores=61, storefiles=67,
>> storefileIndexSize=143,
>> > memstoreSize=20, compactionQueueSize=0, usedHeap=1417, maxHeap=6127,
>> > blockCacheSize=1061944288, blockCacheFree=223059360,
>> blockCacheCount=16239,
>> > blockCacheHitRatio=14
>> >
>> > still 60 regions, and the usedHeap is only 1.4G. I already put all the
>> > memory tunning suggested in the hbase-env.sh to do incremental GC.
>> > looks the jvm still have drastic GC even with the incremental GC turned
>> on.
>> > now the GC log shows:
>> >
>> > 1462948.533: [GC 1462948.533: [ParNew: 138376K->1441K(153344K), 0.0037940
>> > secs]
>> > 1446591K->1309805K(6274432K) icms_dc=0 , 0.0038990 secs] [Times:
>> user=0.03
>> > sys=0
>> > .00, real=0.00 secs]
>> > 1463005.844: [GC 1463005.845: [ParNew: 137716K->17024K(153344K),
>> 0.0139630
>> > secs]
>> > 1446081K->1327172K(6274432K) icms_dc=0 , 0.0140740 secs] [Times:
>> user=0.11
>> > sys=
>> > 0.00, real=0.02 secs]
>> >
>> > which is reasonable.
>> >
>> > Jimmy.
>> >
>> > --------------------------------------------------
>> > From: "Ryan Rawson" <ryanobjc@gmail.com>
>> > Sent: Monday, August 09, 2010 3:07 PM
>> > To: <user@hbase.apache.org>
>> > Subject: Re: memory usage too high in regionserver
>> >
>> >> Unfortunately 'usedHeap' includes the unfound garbage...
>> >>
>> >> Perhaps you can post more of your GC log.
>> >>
>> >> On Mon, Aug 9, 2010 at 3:02 PM, Jinsong Hu <jinsong_hu@hotmail.com>
>> wrote:
>> >>>
>> >>> Hi, There:
>> >>>
>> >>>  I am using cloudera cdh3 regionserver and today, I noticed that one
of
>> >>> the
>> >>> regionserver memory usage is very high:
>> >>>
>> >>> request=8.8, regions=60, stores=61, storefiles=67,
>> >>> storefileIndexSize=142,
>> >>> memstoreSize=58, compactionQueueSize=0, usedHeap=5869, maxHeap=6127,
>> >>> blockCacheSize=1061944288, blockCacheFree=223059360,
>> >>> blockCacheCount=16239,
>> >>> blockCacheHitRatio=14
>> >>>
>> >>> this shows that only 60 regions calused close to 6G of memory usage.
>> This
>> >>> is
>> >>> very high.
>> >>>
>> >>> the gc-hbase.log shows:
>> >>>
>> >>> 1461575.642: [GC 1461575.643: [ParNew: 143538K->17024K(153344K),
>> >>> 0.0487050
>> >>> secs]
>> >>> 5948479K->5846957K(6274432K) icms_dc=0 , 0.0488210 secs] [Times:
>> >>> user=0.33
>> >>> sys=
>> >>> 0.00, real=0.05 secs]
>> >>> 1461693.726: [GC 1461693.726: [ParNew: 153344K->17024K(153344K),
>> >>> 0.0454440
>> >>> secs]
>> >>> 5983277K->5875331K(6274432K) icms_dc=0 , 0.0455680 secs] [Times:
>> >>> user=0.31
>> >>> sys=
>> >>> 0.01, real=0.05 secs]
>> >>>
>> >>>
>> >>> This shows that the regionserver is indeed using 6G of memory. very
>> close
>> >>> to
>> >>> the setting of -Xmx6G limit.
>> >>> and very soon the regionserver will be OOME.
>> >>>
>> >>>
>> >>> if the above holds true, that means a regionserver can only support
>> >>> around
>> >>> 60 regions. I configured the region size to be default 256M.
>> >>>
>> >>> I checked the load on the server and the load is very light. cpu and
>> disk
>> >>> usage is very low. What can I do to drastically increase the number
of
>> >>> regions
>> >>> supported per regionserver in hbase ?
>> >>>
>> >>>
>> >>> Jimmy
>> >>>
>> >>
>> >
>>
>

Mime
View raw message