hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Gray <jg...@facebook.com>
Subject RE: The hfile.block.cache.size = 0 performance is better than default(0.2) in random read? Is it possible?
Date Thu, 21 Oct 2010 16:48:47 GMT
By using the block cache, read blocks are referenced within the block cache data structures
and referenced for a longer amount of time than if not put into the block cache.

This will definitely add additional stress to the GC.

If you expect a very low hit ratio, it can be advantageous to not use the block cache.

You can also turn the block cache off on a per-query basis with setCacheBlocks(), though that's
only supported on Scan right now.  Still makes sense for Gets so we should add it there too.

JG

> -----Original Message-----
> From: ChingShen [mailto:chingshenchen@gmail.com]
> Sent: Thursday, October 21, 2010 4:27 AM
> To: user@hbase.apache.org
> Subject: Re: The hfile.block.cache.size = 0 performance is better than
> default(0.2) in random read? Is it possible?
> 
> Hi Ryan,
> 
> *hfile.block.cache.size = 0, GC log:*
> 2010-10-21T15:53:27.486+0800: 1428.317: [GC [PSYoungGen:
> 18270K->320K(17728K)] 62008K->44066K(61696K), *0.0043520 secs*] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
> 2010-10-21T15:53:27.933+0800: 1428.764: [GC [PSYoungGen:
> 17641K->256K(17024K)] 61386K->44025K(60992K), *0.0036030 secs*] [Times:
> user=0.00 sys=0.00, real=0.00 secs]
> 2010-10-21T15:53:28.380+0800: 1429.212: [GC [PSYoungGen:
> 17024K->288K(19648K)] 60793K->44130K(63616K), *0.0044410 secs*] [Times:
> user=0.00 sys=0.00, real=0.01 secs]
> 2010-10-21T15:53:28.385+0800: 1429.216: [Full GC [PSYoungGen:
> 288K->0K(19648K)] [PSOldGen: 43841K->32920K(41536K)] 44130K-
> >32920K(61184K)
> [PSPermGen: 15684K->15683K(24640K)], *0.0480800 secs*] [Times:
> user=0.05
> sys=0.00, real=0.05 secs]
> 
> *hfile.block.cache.size = 0.2, GC log:*
> 2010-10-21T16:18:31.884+0800: 1234.166: [GC [PSYoungGen:
> 469577K->182750K(534208K)] 1183254K->1013795K(1663424K), *0.1265180
> secs*]
> [Times: user=0.49 sys=0.00, real=0.13 secs]
> 2010-10-21T16:18:56.837+0800: 1259.119: [GC [PSYoungGen:
> 460382K->179115K(451392K)] 1291427K->1116923K(1580608K), *0.1231190
> secs*]
> [Times: user=0.48 sys=0.00, real=0.13 secs]
> 2010-10-21T16:19:20.121+0800: 1282.403: [GC [PSYoungGen:
> 451371K->175649K(510016K)] 1389179K->1206321K(1639232K), *0.1153410
> secs*]
> [Times: user=0.31 sys=0.01, real=0.11 secs]
> 2010-10-21T16:19:20.236+0800: 1282.518: [Full GC [PSYoungGen:
> 175649K->0K(510016K)] [PSOldGen: 1030672K->582437K(1179200K)]
> 1206321K->582437K(1689216K) [PSPermGen: 16041K->16041K(21248K)],
> *0.2538730
> secs*] [Times: user=0.26 sys=0.00, real=0.26 secs]
> 
> hfile.block.cache.size = 0:
> avg. per minor gc ~ *4ms*
> avg. per full gc ~ *50ms*
> 
> hfile.block.cache.size = 0.2:
> avg. per minor gc ~* 120ms*
> avg. per full gc ~ *250ms *
> 
> Does it mean that because I got a low hit ratio in random read, so the
> LruBlockCache object creates too many CachedBlock objects?
> 
> Thanks.
> 
> Shen
> 
> On Thu, Oct 21, 2010 at 2:20 PM, Ryan Rawson <ryanobjc@gmail.com>
> wrote:
> 
> > block cache invalidation is done async and isnt done inline, so that
> > shouldn't be an issue.
> >
> > could be related to GC... if you could produce and compare GC logs
> > that'd be helpful.
> >
> >

Mime
View raw message