hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matt Corgan <mcor...@hotpads.com>
Subject Re: heap memory allocation
Date Fri, 15 Apr 2011 15:23:38 GMT
Some of our servers have 5.2gb hbase heaps with the standard 40% (2gb)
memstore and 20% (1gb) blockcache.  I'm wondering where the indexes and
bloom filters get counted.  Is it in that remaining 2gb, and are the bloom
filters counted in the storefileIndexSize=566 metric?

The overall data in each server isn't too large on disk (~50gb) after gzip
compression of 25x, but it's made of long keys with short values so there is
a lot of metadata.  I'm thinking of upping the block size to 256k but
thought i'd ask how it worked first.

some metrics:

regions=401, stores=401, storefiles=758, storefileIndexSize=566,
memstoreSize=1793, usedHeap=3658, maxHeap=5183, blockCacheSize=816306600

Thanks for the help,
Matt

On Fri, Apr 15, 2011 at 10:39 AM, Doug Meil
<doug.meil@explorysmedical.com>wrote:

> The HBase book has come through some updates recently on these metrics.
>
> http://hbase.apache.org/book.html#hbase_metrics
>
>
> -----Original Message-----
> From: saint.ack@gmail.com [mailto:saint.ack@gmail.com] On Behalf Of Stack
> Sent: Friday, April 15, 2011 5:04 AM
> To: user@hbase.apache.org
> Cc: 陈加俊; hbase-user@hadoop.apache.org
> Subject: Re: heap memory allocation
>
> On Thu, Apr 14, 2011 at 9:50 PM, 陈加俊 <cjjvictory@gmail.com> wrote:
> > How does heap memory allocated in regionserver ?
> >
> > Metrics:
> >
> > request=0.0, regions=1283, stores=2304, storefiles=1968,
> > storefileIndexSize=246, memstoreSize=791, compactionQueueSize=0,
> > usedHeap=5669, maxHeap=11991, blockCacheSize=2095157424,
> > blockCacheFree=419681872, blockCacheCount=24410,
> > blockCacheHitRatio=92, fsReadLatency=0, fsWriteLatency=0,
> > fsSyncLatency=0
> >
> >
> >
> > maxHeap=11991
> > usedHeap=5669
> >
>
> This just the state of the JVM memory bean at the last update time.
> Looks like you have given hbase 12G.  We're using about 5.5 Gig.   See
> here for where we read it from the bean:
>
> http://hbase.apache.org/xref/org/apache/hadoop/hbase/regionserver/HRegionServer.html#751
>  These look like they report in MB.
>
> > blockCacheSize=2095157424
> > memstoreSize=791
> > ????
>
> These look like they report in bytes and MB.  Should make them all same
> units.  HBase has a cache of hdfs blocks.  And memstore is where we write
> edits too before they get flushed to the filesystem.
>
> >
> > Free=maxHeap-usedHeap?
> >
>
> Read the javadoc on the bean returned by
> ManagementFactory.getMemoryMXBean() call.
>
> St.Ack
>
> >
> > --
> > Thanks & Best regards
> > jiajun
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message