hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Антон Лыска <ant...@wildec.com>
Subject hbase performance issue
Date Sun, 11 Mar 2012 19:39:30 GMT
Hi guys!

I have a little instance of hbase cluster with only 2 machines (8core cpu,
12G mem, 3*1GB hdd on each machine).
I use cloudera's cdh3u1 distro.
Cluster serves two tables and total data size is about 300 GB with 300
regions.
The average Get time is usually 20-50ms, but sometimes it rises up to
500-800ms which is unacceptable.

Gets per day: 13*10^6
Puts per day: 11*10^6
Deletes per day: 2*10^6

My conf is:
<configuration>
 ....
<property>
<name>dfs.replication</name>
<value>2</value>
</property>

<property>
<name>hbase.regionserver.handler.count</name>
 <value>50</value>
</property>

<property>
<name>hbase.hregion.majorcompaction</name>
 <value>86400000000</value>
</property>
  </configuration>

My scheme is:
{NAME => 'table1', MAX_FILESIZE => '536870912', FAMILIES => [{NAME => 'c',
BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE',
VERSIONS => '1', TTL => '2147483647', BLOCKSIZE => '16384', IN_MEMORY =>
'false', BLOCKCACHE => 'true'}, {NAME => 'p', BLOOMFILTER => 'NONE',
REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '1', TTL =>
'2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
'true'}, {NAME => 's', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0',
COMPRESSION => 'NONE', VERSIONS => '1', TTL => '2147483647', BLOCKSIZE =>
'65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}

{NAME => 'table2', FAMILIES => [{NAME => 'n', BLOOMFILTER => 'NONE',
REPLICATION_SCOPE => '0', VERSIONS => '1', COMPRESSION => 'NONE', TTL =>
'2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
'true'}]}

I disabled major compaction by setting a big value, and run it manually
each day at 3:00am (server is least loaded at that time).
Get time usually starts increasing at around 23:00-24:00.
Once hbase is restarted, Get time returns to 20ms.
What it can be? what options should I set to avoid this issue?

Also I have installed ganglia, but I haven't seen anything strange there.

Thank you in advance!

Best regards, Anton.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message