hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Louis Hust <louis.h...@gmail.com>
Subject Re: Frequent flush and small compactions, region server load is very high
Date Wed, 20 May 2015 06:50:23 GMT
Hi, ted,

today i counter the problem again, and I find the metrics of GCCOUNT G1
YOUNG GENERATION is related to the
CPU LOAD, so this means it's GC problem?

The options for regionserver like below:

/home/q/java/default/bin/java -Dproc_regionserver
-XX:OnOutOfMemoryError=kill -9 %p -Xmx8192m -Xms8g -Xmx8g -XX:NewSize=4g
-XX:MaxNewSize=4g -XX:PermSize=1g -XX:MaxPermSize=1g -server
-XX:+DisableExplicitGC -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC
-XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=75 -XX:SoftRefLRUPolicyMSPerMB=0
-XX:+PrintClassHistogram -XX:+PrintGCDetails -XX:+PrintGCDateStamps
-XX:+PrintHeapAtGC
-Xloggc:/home/q/hbase/hbase-0.98.6-cdh5.2.0/logs/gc-l-hbase1.dba.cn1-hbase.log
-XX:ParallelGCThreads=10 -Xms64g -Xmx64g -XX:NewSize=4g -XX:MaxNewSize=4g
-XX:PermSize=1g -XX:MaxPermSize=1g -server -XX:+DisableExplicitGC
-XX:+UnlockExperimentalVMOptions -XX:+UseG1GC -XX:+UseFastAccessorMethods
-XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=75
-XX:SoftRefLRUPolicyMSPerMB=0 -XX:+PrintClassHistogram -XX:+PrintGCDetails
-XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
-Xloggc:/home/q/hbase/hbase-0.98.6-cdh5.2.0/logs/gc-l-hbase1.dba.cn1-hbase.log
-XX:ParallelGCThreads=10 -Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.port=10102
-Dhbase.log.dir=/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/../logs
-Dhbase.log.file=hbase-hadoop-regionserver-l-hbase1.dba.cn1.log
-Dhbase.home.dir=/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/..
-Dhbase.id.str=hadoop -Dhbase.root.logger=INFO,RFA
-Djava.library.path=/home/q/hadoop/hadoop-2.3.0-cdh5.1.0/lib/native:/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/../lib/native/Linux-amd64-64
-Dhbase.security.logger=INFO,RFAS
org.apache.hadoop.hbase.regionserver.HRegionServer start


Any idea?



2015-05-20 11:07 GMT+08:00 Louis Hust <louis.hust@gmail.com>:

> Hi, @ted,
>
> It seems not only affect the table ttsi_order:ttsi_order_process, some
> other table's region also flushed and compact
>
> 2015-05-20 11:04 GMT+08:00 Louis Hust <louis.hust@gmail.com>:
>
>> Hi, ted
>>
>> hbase(main):001:0> describe 'ttsi_order:ttsi_order_process'
>> DESCRIPTION
>>          ENABLED
>>  'ttsi_order:ttsi_order_process', {NAME => 'orderinfo',
>> DATA_BLOCK_ENCODING => 'NO true
>>  NE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS => '1',
>> COMPRESSION
>>   => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER', KEEP_DELETED_CELLS =>
>> 'FALSE',
>>   BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}
>> 1 row(s) in 1.0260 seconds
>>
>>
>> 2015-05-20 10:59 GMT+08:00 Ted Yu <yuzhihong@gmail.com>:
>>
>>> How many column families does your table have ?
>>>
>>> Can you show us the output of " describe 'ttsi_order:ttsi_order_process'
>>> " ?
>>>
>>> Thanks
>>>
>>> On Tue, May 19, 2015 at 4:56 AM, Louis Hust <louis.hust@gmail.com>
>>> wrote:
>>>
>>> > Hi, all
>>> >
>>> > My cluster is cdh5.2.0 with hbase 0.98.6, and recently some region
>>> server
>>> > load is very high,
>>> > about 50, and the  CPU usage is about 2400%(with 24 core).
>>> >
>>> > I check the log in region server, and found frequent memstore flush and
>>> > compactions, is this the reason for the high load?
>>> >
>>> > {log}
>>> > 2015-05-19 08:40:21,749 INFO  [MemStoreFlusher.0] regionserver.HRegion:
>>> > Started memstore flush for
>>> >
>>> >
>>> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.,
>>> > current region memstore size 1.1 M
>>> > 2015-05-19 08:40:21,917 INFO  [MemStoreFlusher.0]
>>> > regionserver.DefaultStoreFlusher: Flushed, sequenceid=11905428,
>>> memsize=1.1
>>> > M, hasBloomFilter=true, into tmp file
>>> >
>>> >
>>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
>>> > 2015-05-19 08:40:21,934 DEBUG [MemStoreFlusher.0]
>>> > regionserver.HRegionFileSystem: Committing store file
>>> >
>>> >
>>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
>>> > as
>>> >
>>> >
>>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82
>>> > 2015-05-19 08:40:21,941 INFO  [MemStoreFlusher.0] regionserver.HStore:
>>> > Added
>>> >
>>> >
>>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
>>> > entries=1441, sequenceid=11905428, filesize=847.8 K
>>> > 2015-05-19 08:40:21,941 INFO  [MemStoreFlusher.0] regionserver.HRegion:
>>> > Finished memstore flush of ~1.1 M/1126904, currentsize=0/0 for region
>>> >
>>> >
>>> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
>>> > in 192ms, sequenceid=11905428, compaction requested=true
>>> > 2015-05-19 08:40:21,941 DEBUG [MemStoreFlusher.0]
>>> > regionserver.CompactSplitThread: Small Compaction requested: system;
>>> > Because: MemStoreFlusher.0; compaction_queue=(0:0), split_queue=0,
>>> > merge_queue=0
>>> > 2015-05-19 08:40:21,941 DEBUG
>>> > [regionserver60020-smallCompactions-1431261234450]
>>> > compactions.RatioBasedCompactionPolicy: Selecting compaction from 4
>>> store
>>> > files, 0 compacting, 4 eligible, 30 blocking
>>> > 2015-05-19 08:40:21,941 DEBUG
>>> > [regionserver60020-smallCompactions-1431261234450]
>>> > compactions.ExploringCompactionPolicy: Exploring compaction algorithm
>>> has
>>> > selected 3 files of size 36749798 starting at candidate #1 after
>>> > considering 3 permutations with 1 in ratio
>>> > 2015-05-19 08:40:21,941 DEBUG
>>> > [regionserver60020-smallCompactions-1431261234450] regionserver.HStore:
>>> > 378c4f2ebc93770ef6cbf956de60d093 - orderinfo: Initiating minor
>>> compaction
>>> > 2015-05-19 08:40:21,941 INFO
>>> >  [regionserver60020-smallCompactions-1431261234450]
>>> regionserver.HRegion:
>>> > Starting compaction on orderinfo in region
>>> >
>>> >
>>> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
>>> > 2015-05-19 08:40:21,942 INFO
>>> >  [regionserver60020-smallCompactions-1431261234450]
>>> regionserver.HStore:
>>> > Starting compaction of 3 file(s) in orderinfo of
>>> >
>>> >
>>> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
>>> > into
>>> >
>>> >
>>> tmpdir=hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp,
>>> > totalSize=35.0 M
>>> > 2015-05-19 08:40:21,942 DEBUG
>>> > [regionserver60020-smallCompactions-1431261234450]
>>> compactions.Compactor:
>>> > Compacting
>>> >
>>> >
>>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/a89297d7bbc34db7aa9f23e42789e2bd,
>>> > keycount=22628, bloomtype=ROW, size=33.8 M, encoding=NONE,
>>> seqNum=11902843
>>> > 2015-05-19 08:40:21,942 DEBUG
>>> > [regionserver60020-smallCompactions-1431261234450]
>>> compactions.Compactor:
>>> > Compacting
>>> >
>>> >
>>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/d1f8531b00fa4f77ad901001ebe39644,
>>> > keycount=512, bloomtype=ROW, size=468.8 K, encoding=NONE,
>>> seqNum=11903863
>>> > 2015-05-19 08:40:21,942 DEBUG
>>> > [regionserver60020-smallCompactions-1431261234450]
>>> compactions.Compactor:
>>> > Compacting
>>> >
>>> >
>>> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
>>> > keycount=597, bloomtype=ROW, size=847.8 K, encoding=NONE,
>>> seqNum=11905428
>>> > {/log}
>>> >
>>> > Why the flush so small memstore which just  1.1 M, and many other flush
>>> > even small than 1M。
>>> >
>>> >
>>> > My configuration:
>>> >
>>> > RegionServer: MaxHeap: 64G
>>> > hbase.hregion.memstore.flush.size = 128M
>>> > hbase.regionserver.global.memstore.upperLimit=0.4
>>> > hbase.regionserver.global.memstore.lowerLimit=0.38
>>> > hbase.hregion.memstore.block.multiplier=4
>>> > hbase.hregion.memstore.mslab.enabled=true
>>> > hbase.hregion.max.filesize = 10G
>>> > hbase.hstore.compaction.max=10
>>> > hbase.hstore.compactionThreshold=3
>>> > hbase.hstore.compaction.kv.max=10
>>> >
>>> > And the region server has 223 regions.
>>> >
>>>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message