hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mehdi Ben Haj Abbes <mehdi.ab...@gmail.com>
Subject Re: When compactions become major ones
Date Tue, 05 Jan 2016 20:36:28 GMT
Thanks guys for your feedbacks.  I will check the WAL property and make
other tests and let u know.
Best regards,
Mehdi
Le 5 janv. 2016 9:09 PM, "Vladimir Rodionov" <vladrodionov@gmail.com> a
écrit :

> >>And I still dont understand how the store files resulting after memstore
> >>flushes are having a size of 40MB. Does it hove smth to do with memstore
> >>upper limit and these 42MB are the result of forcing the memstore to be
> >>flushed? The problem is that all the newly store files added to HDFS are
> >>starting with this size (42MB) I did not mention that my CF is in-memory.
>
> Its due to Java object overhead, so 3x is normal (128MB  in memory -> 42MB
> on disk)
> Another aspect to take into account: flush can happen not only when we
> reach memstore size limit,
> there are other triggers as well:
>
> 1. maximum WAL files reached (hbase.regionserver.maxlogs)
> 2. periodic memstore flusher (once an 1h) can trigger flushes a s well
>
> -Vlad
>
>
>
> On Tue, Jan 5, 2016 at 9:37 AM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > For #1,
> > bq. would this minor which becomes major take care of deleted rows
> >
> > Yes.
> >
> > For #2, please consider the following guide:
> >
> > dfs.blocksize (value: ${propdata["dfs.blocksize"]}) * 0.95 *
> > hbase.regionserver.maxlogs (value:
> > ${propdata["hbase.regionserver.maxlogs"]}) should be greater than
> > hbase.regionserver.global.memstore.upperLimit * HBASE_HEAPSIZE (the value
> > for -Xmx)
> >
> > Cheers
> >
> > On Tue, Jan 5, 2016 at 8:39 AM, Mehdi Ben Haj Abbes <
> mehdi.abbes@gmail.com
> > >
> > wrote:
> >
> > > Thanks Ted for the clarification about the major compactions. So if I
> did
> > > understand well when a minor compaction is triggered and the policy
> > selects
> > > all the store files, this compaction becomes a major one. But would
> this
> > > minor which becomes major take care of deleted rows as a major one
> would
> > do
> > > or at the end it is just a minor that happened and selected all the
> store
> > > files ?
> > >
> > > About disabling splitting I have already hbase.hregion.max.filesize set
> > to
> > > 10GB besides I pre splitted my table.
> > >
> > > And I still dont understand how the store files resulting after
> memstore
> > > flushes are having a size of 40MB. Does it hove smth to do with
> memstore
> > > upper limit and these 42MB are the result of forcing the memstore to be
> > > flushed? The problem is that all the newly store files added to HDFS
> are
> > > starting with this size (42MB) I did not mention that my CF is
> in-memory.
> > >
> > > Best regards,
> > >
> > > On Tue, Jan 5, 2016 at 4:04 PM, Ted Yu <yuzhihong@gmail.com> wrote:
> > >
> > > > For #1, when all store files are selected for compaction, the
> > compaction
> > > > becomes major
> > > >
> > > > see 'Determine the Optimal Number of Pre-Split Regions' under:
> > > > http://hbase.apache.org/book.html#disable.splitting
> > > >
> > > > See also http://hbase.apache.org/book.html#managed.compactions
> > > >
> > > > Cheers
> > > >
> > > > On Tue, Jan 5, 2016 at 6:52 AM, Mehdi Ben Haj Abbes <
> > > mehdi.abbes@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > I'm using hbase 0.98. I have heavy writes workload. I'm writing to
> > one
> > > > > table with one CF compressed with GZ. My table is pre splitted to
> 27
> > > > > regions. As I start writing to this table I start seeing HFiles of
> > the
> > > > size
> > > > > of 2-4 MB across the regions. I have the default hbase
> configuration
> > > for
> > > > > compaction properties. The compactions start as soon as I start
> > writing
> > > > to
> > > > > HBase but many of these compaction are major ones. I can see this
> > > through
> > > > > HBase master UI on the table details view. So I wanted to
> understand
> > > > when a
> > > > > compaction becomes major.
> > > > >
> > > > > Another question, If I'm not wrong we have a memstore per region,
> so
> > > > when a
> > > > > memstore is flushed I will have a HFile with 128MB but I only see
> > files
> > > > > with 42MB (without compression and 2.5MB when compressed with GZ).
> > > > >
> > > > > Any explanation ?
> > > > >
> > > > > Thanks in advance.
> > > > > --
> > > > > Mehdi BEN HAJ ABBES
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Mehdi BEN HAJ ABBES
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message