hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From OpenSource Dev <dev.opensou...@gmail.com>
Subject Re: High cpu usage on a region server
Date Wed, 11 Sep 2013 21:02:44 GMT
No, dfs.client.read.shortcircuit is set to false by default in our cluster.

Looks like this is a good performance improvement parameter, are there
any side effects of turning it on ?

Thx

On Wed, Sep 11, 2013 at 1:57 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> Have you turned on short-circuit read ?
>
> Cheers
>
>
> On Wed, Sep 11, 2013 at 1:49 PM, OpenSource Dev <dev.opensource@gmail.com>wrote:
>
>> Hi,
>>
>> I'm using HBase 0.94.6 (CDH 4.3) for Opentsdb. So far I have had no
>> issues with writes/puts. System is handles upto 800k puts per seconds
>> without issue. On average we do 250k puts per second.
>>
>> I am having the problem with Reads, I've also isolated where the
>> problem is but not been able to find the root cause.
>>
>> I have 16 machines running hbase-region server, each has ~35 regions.
>> Once in a while cpu goes flatout 80% in 1 region server. These are the
>> things i've noticed in ganglia:
>>
>> hbase.regionserver.request - evenly distributed. Not seeing any spikes
>> on the busy server
>> hbase.regionserver.blockCacheSize - between 500MB and 1000MB
>> hbase.regionserver.compactionQueueSize - avg 2 or less
>> hbase.regionserver.blockCacheHitRatio - 30% on busy node, >60% on other
>> nodes
>>
>>
>> JVM Heap size is set to 16GB and I'm using -XX:+UseParNewGC
>> -XX:+UseConcMarkSweepGC
>>
>> I've noticed the system load moves to a different region, sometimes
>> within a minute, if the busy region is restarted.
>>
>> Any suggestion what could be causing the load and/or what other
>> metrics should I check ?
>>
>>
>> Thank you!
>>

Mime
View raw message