hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Serega Sheypak <serega.shey...@gmail.com>
Subject Re: Spikes when writing data to HBase
Date Tue, 11 Aug 2015 21:43:14 GMT
>How about GC activity? ApplicationStopTime? Do you track that?
yes, jviusalm says it's ok, newrelic also doesn't show something strange.
HBase also says it's OK.

Profiler says most time thread is waiting for response from hbase side. My
assumption is:
1. I have weird bug in HBase configuration
2. I have undiscovered problems with networking (BUT the same tomcats write
data to flume with higher rate, no data loss at all)
3. I have weird problem with HConnection HConnectionManager is
multithreaded env, when same servlet instance shared across many threads
4. some mystic process somewhere in the cluster....

>Is the issue reproducible? or you got it first time?
always. Spikes disappear during night, but RPM doesn't change too much.

I will run my controller code out of tomcat and see how it goes. I'm going
to isolate components...


2015-08-11 23:36 GMT+02:00 Vladimir Rodionov <vladrodionov@gmail.com>:

> How about GC activity? ApplicationStopTime? Do you track that?
>
> Is the issue reproducible? or you got it first time?
>
> Start with RS logs and try to find anything suspicious in a period of a
> very high latency. 1.5 sec HBase write latency does not look right.
>
> -Vlad
>
> On Tue, Aug 11, 2015 at 2:08 PM, Serega Sheypak <serega.sheypak@gmail.com>
> wrote:
>
> > Hi Vladimir!
> >
> > Here are graphs. Servlet (3 tomcats on 3 different hosts write to HBase)
> > http://www.bigdatapath.com/wp-content/uploads/2015/08/01_apps1.png
> > See how response time jump. I can't explain it. Write load is
> really-really
> > low.
> >
> > all RS have even load. I see request-metrics in HBase master web UI.
> > Tables are pre-splitted. I have 10 RS and pre-splitted tables on 50
> > regions.
> >
> >   >1. How large is your single write?
> > 1-2KB
> >
> >    >2. Do you see any RegionTooBusyException in a client log files
> > no HBase related exceptions. Response
> >
> >  >  3. How large is your table ( # of regions, # of column families)
> > 1 column familiy, table_01 150GB, table_02 130 GB
> >
> > I have two "major tables", here are stats for them:
> > http://www.bigdatapath.com/wp-content/uploads/2015/08/table_02.png
> > http://www.bigdatapath.com/wp-content/uploads/2015/08/table_01.png
> >    >4. RS memory related config: Max heap
> >
> >    5. memstore size (if not default - 0.4)
> > hbase.regionserver.global.memstore.upperLimit=0.4
> > hbase.regionserver.global.memstore.lowerLimit=0.38
> > RS heapsize=8GB
> >
> > >*Do you see any region splits?  *
> > no, never happened since tables are pre-splitted
> >
> > 2015-08-11 18:54 GMT+02:00 Vladimir Rodionov <vladrodionov@gmail.com>:
> >
> > > *Common questions:*
> > >
> > >
> > >    1. How large is your single write?
> > >    2. Do you see any RegionTooBusyException in a client log files
> > >    3. How large is your table ( # of regions, # of column families)
> > >    4. RS memory related config: Max heap
> > >    5. memstore size (if not default - 0.4)
> > >
> > >
> > > Memstore flush
> > >
> > > hbase.hregion.memstore.flush.size = 256M
> > > hbase.hregion.memstore.block.multiplier = N (do not block writes) N *
> > 256M
> > > MUST be greater than overall memstore size (HBASE_HEAPSIZE *
> > > hbase.regionserver.global.memstore.size)
> > >
> > > WAL files.
> > >
> > > Set HDFS block size to 256MB. hbase.regionserver.hlog.blocksize = 0.95
> > HDFS
> > > block size (256MB * 0.95). Keep hbase.regionserver.hlog.blocksize *
> > > hbase.regionserver.maxlogs just a bit above
> > > hbase.regionserver.global.memstore.lowerLimit
> > > (0.35-0.45) * HBASE_HEAPSIZE to avoid premature memstore flushing.
> > >
> > > *Do you see any region splits?  *
> > >
> > > Region split blocks writes. Try to presplit table and avoid splitting
> > after
> > > that. Disable splitting completely
> > >
> > > hbase.regionserver.region.split.policy
> > > =org.apache.hadoop.hbase.regionserver.DisabledRegionSplitPolicy
> > >
> > > -Vlad
> > >
> > >
> > >
> > >
> > > On Tue, Aug 11, 2015 at 3:22 AM, Serega Sheypak <
> > serega.sheypak@gmail.com>
> > > wrote:
> > >
> > > > Hi, we are using version 1.0.0+cdh5.4.4+160
> > > > We have heavy write load, ~ 10K per econd
> > > > We have 10 nodes 7 disks each. I read some perf notes, they state
> that
> > > > HBase can handle 1K per second writes per node without any problems.
> > > >
> > > >
> > > > I see some spikes on "writers". Write operation timing "jumps" from
> > > 40-50ms
> > > > to 200-500ms Probably I hit memstore limit. RegionServer starts to
> > flush
> > > > memstore and stop to accept updates.
> > > >
> > > > I have several questions:
> > > > 1. Does 4/(8 in hyperthreading) CPU + 7HDD node could absorb 1K
> writes
> > > per
> > > > second?
> > > > 2. What is the right way to fight with blocked writes?
> > > > 2.1. What I did:
> > > > hbase.hregion.memstore.flush.size to 256M to produce larger HFiles
> when
> > > > flushing memstore
> > > > base.hregion.memstore.block.multiplier to 4, since I have only one
> > > > intensive-write table. Let it grow
> > > > hbase.regionserver.optionallogflushinterval to 10s, i CAN loose some
> > > data,
> > > > NP here. The idea that I reduce I/O pressure on disks.
> > > > ===
> > > > Not sure if I can correctly play with these parameters.
> > > > hbase.hstore.blockingStoreFiles=10
> > > > hbase.hstore.compactionThreshold=3
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message