hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Khaled Elmeleegy <kd...@hotmail.com>
Subject RE: HBase read performance
Date Sat, 11 Oct 2014 02:59:39 GMT
Nick,

I am still prototyping, hence the small instance. I thought XX:MaxDirectMemorySize is subset
of Xmx, so I thought the total is 6GB. You are saying they are added. If so, that's not good
for performance as the JVM will swap. However, this can't be the reason for the bug. 

For now, I guess the only option is to use LruBlockCache. However, I expect it to suffer
from significant GC, as I churn memory fast because I want to cache blocks on writes and I
do significant writing.

Khaled

----------------------------------------
> From: ndimiduk@gmail.com
> Date: Fri, 10 Oct 2014 15:00:45 -0700
> Subject: Re: HBase read performance
> To: user@hbase.apache.org
>
> Hang on, how are you using 11G total memory? m1.large only has 7.5G total
> RAM.
>
> On Fri, Oct 10, 2014 at 2:56 PM, Nick Dimiduk <ndimiduk@gmail.com> wrote:
>
>> ByteBuffer position math errors makes me suspect #1 cacheonwrite and #2
>> bucketcache (and #3 their use in combination ;) )
>>
>> 11G memory is a small enough footprint that I'd not bother with
>> BucketCache; just stay on heap with default LruBlockCache.
>>
>> On Fri, Oct 10, 2014 at 2:11 PM, Stack <stack@duboce.net> wrote:
>>
>>> On Fri, Oct 10, 2014 at 10:59 AM, Khaled Elmeleegy <kdiaa@hotmail.com>
>>> wrote:
>>>
>>>> Yes, I can reproduce it with some work.
>>>> The workload is basically as follows:
>>>> There are writers streaming writes to a table. Then, there is a reader
>>>> (invoked via a web interface). The reader does a 1000 parallel reverse
>>>> scans, all end up hitting the same region in my case. The scans are
>>>> effectively "gets" as I just need to get one record off of each of
>>> them. I
>>>> just need to do a "reverse" get, which is not supported (would be great
>>> to
>>>> have :)), so I do it via reverse scan. After few tries, the reader
>>>> consistently hits this bug.
>>>>
>>>> This happens with these config changes:
>>>> hbase-env:HBASE_REGIONSERVER_OPTS=-Xmx6G -XX:MaxDirectMemorySize=5G
>>>> -XX:CMSInitiatingOccupancyFraction=88 -XX:+AggressiveOpts -verbose:gc
>>>> -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xlog
>>>> gc:/tmp/hbase-regionserver-gc.loghbase-site:
>>>> hbase.bucketcache.ioengine=offheap
>>>> hbase.bucketcache.size=4196
>>>> hbase.rs.cacheblocksonwrite=true
>>>> hfile.block.index.cacheonwrite=true
>>>> hfile.block.bloom.cacheonwrite=true
>>>>
>>>> Interestingly, without these config changes, I can't reproduce the
>>> problem.
>>>
>>>
>>>
>>> How hard to play w/ combinations? Could you eliminate the cacheonwrites
>>> on
>>> one server and see if that cures the issue? Could trun off block cache on
>>> another to see if that the problem? Anything in your .out files related?
>>>
>>> St.Ack
>>>
>>
>>
 		 	   		  
Mime
View raw message