hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Frequent flush and small compactions, region server load is very high
Date Thu, 21 May 2015 12:39:04 GMT
Kudo goes to Bryan Beaudreault
<http://search-hadoop.com/?q=&fc_author=Bryan%20Beaudreault>.

Cheers

On Wed, May 20, 2015 at 11:35 PM, Louis Hust <louis.hust@gmail.com> wrote:

> I will try the -XX:G1HeapRegionSize option and test again, thanks ted!
>
> 2015-05-20 21:29 GMT+08:00 Ted Yu <yuzhihong@gmail.com>:
>
> > Louis:
> > I don't see -XX:G1HeapRegionSize option in the command line.
> >
> > Please refer to Bryan's comment :
> > http://search-hadoop.com/m/YGbbupEDoKTrDo
> >
> > Cheers
> >
> > On Tue, May 19, 2015 at 11:50 PM, Louis Hust <louis.hust@gmail.com>
> wrote:
> >
> > > Hi, ted,
> > >
> > > today i counter the problem again, and I find the metrics of GCCOUNT G1
> > > YOUNG GENERATION is related to the
> > > CPU LOAD, so this means it's GC problem?
> > >
> > > The options for regionserver like below:
> > >
> > > /home/q/java/default/bin/java -Dproc_regionserver
> > > -XX:OnOutOfMemoryError=kill -9 %p -Xmx8192m -Xms8g -Xmx8g
> -XX:NewSize=4g
> > > -XX:MaxNewSize=4g -XX:PermSize=1g -XX:MaxPermSize=1g -server
> > > -XX:+DisableExplicitGC -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC
> > > -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly
> > > -XX:CMSInitiatingOccupancyFraction=75 -XX:SoftRefLRUPolicyMSPerMB=0
> > > -XX:+PrintClassHistogram -XX:+PrintGCDetails -XX:+PrintGCDateStamps
> > > -XX:+PrintHeapAtGC
> > >
> > >
> >
> -Xloggc:/home/q/hbase/hbase-0.98.6-cdh5.2.0/logs/gc-l-hbase1.dba.cn1-hbase.log
> > > -XX:ParallelGCThreads=10 -Xms64g -Xmx64g -XX:NewSize=4g
> -XX:MaxNewSize=4g
> > > -XX:PermSize=1g -XX:MaxPermSize=1g -server -XX:+DisableExplicitGC
> > > -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC
> -XX:+UseFastAccessorMethods
> > > -XX:+UseCMSInitiatingOccupancyOnly
> -XX:CMSInitiatingOccupancyFraction=75
> > > -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+PrintClassHistogram
> > -XX:+PrintGCDetails
> > > -XX:+PrintGCDateStamps -XX:+PrintHeapAtGC
> > >
> > >
> >
> -Xloggc:/home/q/hbase/hbase-0.98.6-cdh5.2.0/logs/gc-l-hbase1.dba.cn1-hbase.log
> > > -XX:ParallelGCThreads=10 -Dcom.sun.management.jmxremote.ssl=false
> > > -Dcom.sun.management.jmxremote.authenticate=false
> > > -Dcom.sun.management.jmxremote.port=10102
> > > -Dhbase.log.dir=/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/../logs
> > > -Dhbase.log.file=hbase-hadoop-regionserver-l-hbase1.dba.cn1.log
> > > -Dhbase.home.dir=/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/..
> > > -Dhbase.id.str=hadoop -Dhbase.root.logger=INFO,RFA
> > >
> > >
> >
> -Djava.library.path=/home/q/hadoop/hadoop-2.3.0-cdh5.1.0/lib/native:/home/q/hbase/hbase-0.98.6-cdh5.2.0/bin/../lib/native/Linux-amd64-64
> > > -Dhbase.security.logger=INFO,RFAS
> > > org.apache.hadoop.hbase.regionserver.HRegionServer start
> > >
> > >
> > > Any idea?
> > >
> > >
> > >
> > > 2015-05-20 11:07 GMT+08:00 Louis Hust <louis.hust@gmail.com>:
> > >
> > > > Hi, @ted,
> > > >
> > > > It seems not only affect the table ttsi_order:ttsi_order_process,
> some
> > > > other table's region also flushed and compact
> > > >
> > > > 2015-05-20 11:04 GMT+08:00 Louis Hust <louis.hust@gmail.com>:
> > > >
> > > >> Hi, ted
> > > >>
> > > >> hbase(main):001:0> describe 'ttsi_order:ttsi_order_process'
> > > >> DESCRIPTION
> > > >>          ENABLED
> > > >>  'ttsi_order:ttsi_order_process', {NAME => 'orderinfo',
> > > >> DATA_BLOCK_ENCODING => 'NO true
> > > >>  NE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSIONS
=>
> '1',
> > > >> COMPRESSION
> > > >>   => 'NONE', MIN_VERSIONS => '0', TTL => 'FOREVER',
> KEEP_DELETED_CELLS
> > > =>
> > > >> 'FALSE',
> > > >>   BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE =>
'true'}
> > > >> 1 row(s) in 1.0260 seconds
> > > >>
> > > >>
> > > >> 2015-05-20 10:59 GMT+08:00 Ted Yu <yuzhihong@gmail.com>:
> > > >>
> > > >>> How many column families does your table have ?
> > > >>>
> > > >>> Can you show us the output of " describe
> > > 'ttsi_order:ttsi_order_process'
> > > >>> " ?
> > > >>>
> > > >>> Thanks
> > > >>>
> > > >>> On Tue, May 19, 2015 at 4:56 AM, Louis Hust <louis.hust@gmail.com>
> > > >>> wrote:
> > > >>>
> > > >>> > Hi, all
> > > >>> >
> > > >>> > My cluster is cdh5.2.0 with hbase 0.98.6, and recently some
> region
> > > >>> server
> > > >>> > load is very high,
> > > >>> > about 50, and the  CPU usage is about 2400%(with 24 core).
> > > >>> >
> > > >>> > I check the log in region server, and found frequent memstore
> flush
> > > and
> > > >>> > compactions, is this the reason for the high load?
> > > >>> >
> > > >>> > {log}
> > > >>> > 2015-05-19 08:40:21,749 INFO  [MemStoreFlusher.0]
> > > regionserver.HRegion:
> > > >>> > Started memstore flush for
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.,
> > > >>> > current region memstore size 1.1 M
> > > >>> > 2015-05-19 08:40:21,917 INFO  [MemStoreFlusher.0]
> > > >>> > regionserver.DefaultStoreFlusher: Flushed, sequenceid=11905428,
> > > >>> memsize=1.1
> > > >>> > M, hasBloomFilter=true, into tmp file
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
> > > >>> > 2015-05-19 08:40:21,934 DEBUG [MemStoreFlusher.0]
> > > >>> > regionserver.HRegionFileSystem: Committing store file
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp/6fab6457782048898b1547c422fc0c82
> > > >>> > as
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82
> > > >>> > 2015-05-19 08:40:21,941 INFO  [MemStoreFlusher.0]
> > > regionserver.HStore:
> > > >>> > Added
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
> > > >>> > entries=1441, sequenceid=11905428, filesize=847.8 K
> > > >>> > 2015-05-19 08:40:21,941 INFO  [MemStoreFlusher.0]
> > > regionserver.HRegion:
> > > >>> > Finished memstore flush of ~1.1 M/1126904, currentsize=0/0
for
> > region
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> > > >>> > in 192ms, sequenceid=11905428, compaction requested=true
> > > >>> > 2015-05-19 08:40:21,941 DEBUG [MemStoreFlusher.0]
> > > >>> > regionserver.CompactSplitThread: Small Compaction requested:
> > system;
> > > >>> > Because: MemStoreFlusher.0; compaction_queue=(0:0),
> split_queue=0,
> > > >>> > merge_queue=0
> > > >>> > 2015-05-19 08:40:21,941 DEBUG
> > > >>> > [regionserver60020-smallCompactions-1431261234450]
> > > >>> > compactions.RatioBasedCompactionPolicy: Selecting compaction
> from 4
> > > >>> store
> > > >>> > files, 0 compacting, 4 eligible, 30 blocking
> > > >>> > 2015-05-19 08:40:21,941 DEBUG
> > > >>> > [regionserver60020-smallCompactions-1431261234450]
> > > >>> > compactions.ExploringCompactionPolicy: Exploring compaction
> > algorithm
> > > >>> has
> > > >>> > selected 3 files of size 36749798 starting at candidate #1
after
> > > >>> > considering 3 permutations with 1 in ratio
> > > >>> > 2015-05-19 08:40:21,941 DEBUG
> > > >>> > [regionserver60020-smallCompactions-1431261234450]
> > > regionserver.HStore:
> > > >>> > 378c4f2ebc93770ef6cbf956de60d093 - orderinfo: Initiating
minor
> > > >>> compaction
> > > >>> > 2015-05-19 08:40:21,941 INFO
> > > >>> >  [regionserver60020-smallCompactions-1431261234450]
> > > >>> regionserver.HRegion:
> > > >>> > Starting compaction on orderinfo in region
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> > > >>> > 2015-05-19 08:40:21,942 INFO
> > > >>> >  [regionserver60020-smallCompactions-1431261234450]
> > > >>> regionserver.HStore:
> > > >>> > Starting compaction of 3 file(s) in orderinfo of
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> ttsi_order:ttsi_order_process,fceaa9d2804f005B,1431599229048.378c4f2ebc93770ef6cbf956de60d093.
> > > >>> > into
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> tmpdir=hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/.tmp,
> > > >>> > totalSize=35.0 M
> > > >>> > 2015-05-19 08:40:21,942 DEBUG
> > > >>> > [regionserver60020-smallCompactions-1431261234450]
> > > >>> compactions.Compactor:
> > > >>> > Compacting
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/a89297d7bbc34db7aa9f23e42789e2bd,
> > > >>> > keycount=22628, bloomtype=ROW, size=33.8 M, encoding=NONE,
> > > >>> seqNum=11902843
> > > >>> > 2015-05-19 08:40:21,942 DEBUG
> > > >>> > [regionserver60020-smallCompactions-1431261234450]
> > > >>> compactions.Compactor:
> > > >>> > Compacting
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/d1f8531b00fa4f77ad901001ebe39644,
> > > >>> > keycount=512, bloomtype=ROW, size=468.8 K, encoding=NONE,
> > > >>> seqNum=11903863
> > > >>> > 2015-05-19 08:40:21,942 DEBUG
> > > >>> > [regionserver60020-smallCompactions-1431261234450]
> > > >>> compactions.Compactor:
> > > >>> > Compacting
> > > >>> >
> > > >>> >
> > > >>>
> > >
> >
> hdfs://mycluster:8020/hbase/airfare/data/ttsi_order/ttsi_order_process/378c4f2ebc93770ef6cbf956de60d093/orderinfo/6fab6457782048898b1547c422fc0c82,
> > > >>> > keycount=597, bloomtype=ROW, size=847.8 K, encoding=NONE,
> > > >>> seqNum=11905428
> > > >>> > {/log}
> > > >>> >
> > > >>> > Why the flush so small memstore which just  1.1 M, and many
other
> > > flush
> > > >>> > even small than 1M。
> > > >>> >
> > > >>> >
> > > >>> > My configuration:
> > > >>> >
> > > >>> > RegionServer: MaxHeap: 64G
> > > >>> > hbase.hregion.memstore.flush.size = 128M
> > > >>> > hbase.regionserver.global.memstore.upperLimit=0.4
> > > >>> > hbase.regionserver.global.memstore.lowerLimit=0.38
> > > >>> > hbase.hregion.memstore.block.multiplier=4
> > > >>> > hbase.hregion.memstore.mslab.enabled=true
> > > >>> > hbase.hregion.max.filesize = 10G
> > > >>> > hbase.hstore.compaction.max=10
> > > >>> > hbase.hstore.compactionThreshold=3
> > > >>> > hbase.hstore.compaction.kv.max=10
> > > >>> >
> > > >>> > And the region server has 223 regions.
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message