hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dan Washusen <...@reactive.org>
Subject Re: Optimizations for random read performance
Date Tue, 16 Feb 2010 01:22:04 GMT
So roughly 72% of reads use the blocks held in the block cache...

It would be interesting to see the difference between when it was working OK
and now.  Could you try increasing the memory allocated to one of the
regions and also increasing the "hfile.block.cache.size" to say '0.4' on the
same region?

On 16 February 2010 11:54, James Baldassari <james@dataxu.com> wrote:

> Hi Dan.  Thanks for your suggestions.  I am doing writes at the same
> time as reads, but there are usually many more reads than writes.  Here
> are the stats for all three region servers:
>
> Region Server 1:
> request=0.0, regions=15, stores=16, storefiles=34, storefileIndexSize=3,
> memstoreSize=308, compactionQueueSize=0, usedHeap=3096, maxHeap=4079,
> blockCacheSize=705474544, blockCacheFree=150032400, blockCacheCount=10606,
> blockCacheHitRatio=76, fsReadLatency=0, fsWriteLatency=0, fsSyncLatency=0
>
> Region Server 2:
> request=0.0, regions=16, stores=16, storefiles=39, storefileIndexSize=4,
> memstoreSize=225, compactionQueueSize=0, usedHeap=3380, maxHeap=4079,
> blockCacheSize=643172800, blockCacheFree=212334144, blockCacheCount=9660,
> blockCacheHitRatio=69, fsReadLatency=0, fsWriteLatency=0, fsSyncLatency=0
>
> Region Server 3:
> request=0.0, regions=13, stores=13, storefiles=31, storefileIndexSize=4,
> memstoreSize=177, compactionQueueSize=0, usedHeap=1905, maxHeap=4079,
> blockCacheSize=682848608, blockCacheFree=172658336, blockCacheCount=10262,
> blockCacheHitRatio=72, fsReadLatency=0, fsWriteLatency=0, fsSyncLatency=0
>
> The average blockCacheHitRatio is about 72.  Is this too low?  Anything
> else I can check?
>
> -James
>
>
> On Mon, 2010-02-15 at 18:16 -0600, Dan Washusen wrote:
> > Maybe the block cache is thrashing?
> >
> > If you are regularly writing data to your tables then it's possible that
> the
> > block cache is no longer being effective.  On the region server web UI
> check
> > the blockCacheHitRatio value.  You want this value to be high (0 - 100).
>  If
> > this value is low it means that HBase has to go to disk to fetch blocks
> of
> > data.  You can control the amount of VM memory that HBase allocates to
> the
> > block cache using the "hfile.block.cache.size" property (default is 0.2
> > (20%)).
> >
> > Cheers,
> > Dan
> >
> > On 16 February 2010 10:45, James Baldassari <james@dataxu.com> wrote:
> >
> > > Hi,
> > >
> > > Does anyone have any tips to share regarding optimization for random
> > > read performance?  For writes I've found that setting a large write
> > > buffer and setting auto-flush to false on the client side significantly
> > > improved put performance.  Are there any similar easy tweaks to improve
> > > random read performance?
> > >
> > > I'm using HBase 0.20.3 in a very read-heavy real-time system with 1
> > > master and 3 region servers.  It was working ok for a while, but today
> > > there was a severe degradation in read performance.  Restarting Hadoop
> > > and HBase didn't help, are there are no errors in the logs.  Read
> > > performance starts off around 1,000-2,000 gets/second but quickly
> > > (within minutes) drops to around 100 gets/second.
> > >
> > > I've already looked at the performance tuning wiki page.  On the server
> > > side I've increased hbase.regionserver.handler.count from 10 to 100,
> but
> > > it didn't help.  Maybe this is expected because I'm only using a single
> > > client to do reads.  I'm working on implementing a client pool now, but
> > > I'm wondering if there are any other settings on the server or client
> > > side that might improve things.
> > >
> > > Thanks,
> > > James
> > >
> > >
> > >
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message