hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Louis Hust <louis.h...@gmail.com>
Subject Re: Frequent flush and small compactions, region server load is very high
Date Wed, 20 May 2015 03:07:34 GMT
Hi, @ted,

It seems not only affect the table ttsi_order:ttsi_order_process, some
other table's region also flushed and compact

2015-05-20 11:04 GMT+08:00 Louis Hust <louis.hust@gmail.com>:

> Hi, ted
>
> hbase(main):001:0> describe 'ttsi_order:ttsi_order_process'
> DESCRIPTION
>          ENABLED
>  'ttsi_order:ttsi_order_process', {NAME => 'orderinfo',
> DATA_BLOCK_ENCODING => 'NO true
>  NE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1',
> COMPRESSION
>   => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS =>
> 'FALSE',
>   BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
> 1 row(s) in 1.0260 seconds
>
>
> 2015-05-20 10:59 GMT+08:00 Ted Yu <yuzhihong@gmail.com>:
>
>> How many column families does your table have ?
>>
>> Can you show us the output of " describe 'ttsi_order:ttsi_order_process'
>> " ?
>>
>> Thanks
>>
>> On Tue, May 19, 2015 at 4:56 AM, Louis Hust <louis.hust@gmail.com> wrote:
>>
>> > Hi, all
>> >
>> > My cluster is cdh5.2.0 with hbase 0.98.6, and recently some region
>> server
>> > load is very high,
>> > about 50, and the  CPU usage is about 2400%(with 24 core).
>> >
>> > I check the log in region server, and found frequent memstore flush and
>> > compactions, is this the reason for the high load?
>> >
>> > {log}
>> > 2015-05-19 08:40:21,749 INFO  [MemStoreFlusher.0] regionserver.HRegion:
>> > Started memstore flush for
>> >
>> >
>> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.,
>> > current region memstore size 1.1 M
>> > 2015-05-19 08:40:21,917 INFO  [MemStoreFlusher.0]
>> > regionserver.DefaultStoreFlusher: Flushed, sequenceid=11905428,
>> memsize=1.1
>> > M, hasBloomFilter=true, into tmp file
>> >
>> >
>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
>> > 2015-05-19 08:40:21,934 DEBUG [MemStoreFlusher.0]
>> > regionserver.HRegionFileSystem: Committing store file
>> >
>> >
>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
>> > as
>> >
>> >
>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82
>> > 2015-05-19 08:40:21,941 INFO  [MemStoreFlusher.0] regionserver.HStore:
>> > Added
>> >
>> >
>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
>> > entries=1441, sequenceid=11905428, filesize=847.8 K
>> > 2015-05-19 08:40:21,941 INFO  [MemStoreFlusher.0] regionserver.HRegion:
>> > Finished memstore flush of ~1.1 M/1126904, currentsize=0/0 for region
>> >
>> >
>> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
>> > in 192ms, sequenceid=11905428, compaction requested=true
>> > 2015-05-19 08:40:21,941 DEBUG [MemStoreFlusher.0]
>> > regionserver.CompactSplitThread: Small Compaction requested: system;
>> > Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0,
>> > merge_queue=0
>> > 2015-05-19 08:40:21,941 DEBUG
>> > [regionserver60020-smallCompactions-1431261234450]
>> > compactions.RatioBasedCompactionPolicy: Selecting compaction from 4
>> store
>> > files, 0 compacting, 4 eligible, 30 blocking
>> > 2015-05-19 08:40:21,941 DEBUG
>> > [regionserver60020-smallCompactions-1431261234450]
>> > compactions.ExploringCompactionPolicy: Exploring compaction algorithm
>> has
>> > selected 3 files of size 36749798 starting at candidate #1 after
>> > considering 3 permutations with 1 in ratio
>> > 2015-05-19 08:40:21,941 DEBUG
>> > [regionserver60020-smallCompactions-1431261234450] regionserver.HStore:
>> > 378c4f2ebc93770ef6cbf956de60d093 - orderinfo: Initiating minor
>> compaction
>> > 2015-05-19 08:40:21,941 INFO
>> >  [regionserver60020-smallCompactions-1431261234450]
>> regionserver.HRegion:
>> > Starting compaction on orderinfo in region
>> >
>> >
>> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
>> > 2015-05-19 08:40:21,942 INFO
>> >  [regionserver60020-smallCompactions-1431261234450] regionserver.HStore:
>> > Starting compaction of 3 file(s) in orderinfo of
>> >
>> >
>> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
>> > into
>> >
>> >
>> tmpdir=hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp,
>> > totalSize=35.0 M
>> > 2015-05-19 08:40:21,942 DEBUG
>> > [regionserver60020-smallCompactions-1431261234450]
>> compactions.Compactor:
>> > Compacting
>> >
>> >
>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/a89297d7bbc34db7aa9f23e42789e2bd,
>> > keycount=22628, bloomtype=ROW, size=33.8 M, encoding=NONE,
>> seqNum=11902843
>> > 2015-05-19 08:40:21,942 DEBUG
>> > [regionserver60020-smallCompactions-1431261234450]
>> compactions.Compactor:
>> > Compacting
>> >
>> >
>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/d1f8531b00fa4f77ad901001ebe39644,
>> > keycount=512, bloomtype=ROW, size=468.8 K, encoding=NONE,
>> seqNum=11903863
>> > 2015-05-19 08:40:21,942 DEBUG
>> > [regionserver60020-smallCompactions-1431261234450]
>> compactions.Compactor:
>> > Compacting
>> >
>> >
>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
>> > keycount=597, bloomtype=ROW, size=847.8 K, encoding=NONE,
>> seqNum=11905428
>> > {/log}
>> >
>> > Why the flush so small memstore which just  1.1 M, and many other flush
>> > even small than 1M。
>> >
>> >
>> > My configuration:
>> >
>> > RegionServer: MaxHeap: 64G
>> > hbase.hregion.memstore.flush.size = 128M
>> > hbase.regionserver.global.memstore.upperLimit=0.4
>> > hbase.regionserver.global.memstore.lowerLimit=0.38
>> > hbase.hregion.memstore.block.multiplier=4
>> > hbase.hregion.memstore.mslab.enabled=true
>> > hbase.hregion.max.filesize = 10G
>> > hbase.hstore.compaction.max=10
>> > hbase.hstore.compactionThreshold=3
>> > hbase.hstore.compaction.kv.max=10
>> >
>> > And the region server has 223 regions.
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message