hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: 答复: one table flushes at much smaller sizes than other?
Date Fri, 27 Dec 2013 21:08:59 GMT
You're welcome, Norbert.


On Fri, Dec 27, 2013 at 9:44 AM, Norbert Burger <norbert.burger@gmail.com>wrote:

> Thanks Ted - this config change appears to have reduced quite a bit of the
> memstore flushes.
>
> Norbert
>
>
> On Fri, Dec 27, 2013 at 12:03 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > Raising maxlogs to 64 would give you improvement.
> >
> > Cheers
> >
> > On Dec 26, 2013, at 8:35 PM, Norbert Burger <norbert.burger@gmail.com>
> > wrote:
> >
> > > Thanks Ted and Liang.  I do see the "Too may hlogs" line in our logs.
> > >
> > > We're using the default values for hbase.regionserver.maxlogs = 32 and
> > > hbase.regionserver.hlog.blocksize = 64MB, so
> > >
> > > So given a heapsize = 12GB and lowerLimit = 0.35, does it make sense
> > that I
> > > should raise maxlogs to be something like 64 or 72?
> > >
> > > Norbert
> > >
> > >
> > >
> > > On Thu, Dec 26, 2013 at 11:03 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > >
> > >> Norbert:
> > >> What's your setting for hbase.regionserver.maxlogs ?
> > >>
> > >> Cheers
> > >>
> > >> On Dec 26, 2013, at 7:51 PM, 谢良 <xieliang@xiaomi.com> wrote:
> > >>
> > >>> Do you observe some log like "Too many hlogs" before flushing a few
> > >>> dozen KB log ?  Just want to double-check it's weather a forcing
> flush
> > >> issue
> > >>> or not:)
> > >>>
> > >>> Thanks,
> > >>> Liang
> > >>> ________________________________________
> > >>> 发件人: Norbert Burger [norbert.burger@gmail.com]
> > >>> 发送时间: 2013年12月27日 5:52
> > >>> 收件人: user@hbase.apache.org
> > >>> 主题: one table flushes at much smaller sizes than other?
> > >>>
> > >>> Hi folks, we're currently running CDH4.1.2 (HBase 0.92 + patches).
> > >>>
> > >>> Looking through the logs, I'm seeing that some tables are flushing
at
> > >> much
> > >>> lower size than others (a few dozen KB as opposed to the expected 128
> > >> MB).
> > >>>
> > >>> Looking through some of great case studies like [1], [2], and [3],
> it's
> > >> not
> > >>> clear to me why this is happening.  We're using the defaults for
> upper,
> > >>> lower limits (0.35 and 0.40, respectively), and the RS status pages
> > show
> > >>> that we're only using 25% of our memstore allocation.
> > >>>
> > >>> Our regions/RS ratio is currently at 60-70.  We do have some other
> > tables
> > >>> online where column families are skewed heavily sizewise -- could
> this
> > be
> > >>> causing other tables in our cluster to flush more frequently than
> > others?
> > >>>
> > >>> Norbert
> > >>>
> > >>> [1]
> > >>
> > http://blog.sematext.com/2012/07/16/hbase-memstore-what-you-should-know/
> > >>> [2] http://www.ngdata.com/visualizing-hbase-flushes-and-compactions/
> > >>> [3] http://gbif.blogspot.com/2012/07/optimizing-writes-in-hbase.html
> > >>
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message