hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Rodionov <vladrodio...@gmail.com>
Subject Re: Major compaction
Date Mon, 04 Apr 2016 17:14:54 GMT
>> Why I am trying to understand this is because Hbase also sets it to 24
hour default (for time based compaction) and I am looking to lower it to
say >> 20 mins to reduce stress by spreading the load.

The more frequently you run major compaction the more IO (disk/network) you
consume.

Usually, in production environment, periodic major compactions are disabled
and run manually  to avoid major compaction storms.

To control major compaction completely you will also need to disable
promotion minor compaction to major ones. You can do this, by setting
maximum compaction size for minor compaction:
*hbase.hstore.compaction.max.size*

-Vlad


On Mon, Apr 4, 2016 at 8:55 AM, Esteban Gutierrez <esteban@cloudera.com>
wrote:

> Hello Sumit,
>
> Ideally you shouldn't be triggering major compactions that frequently since
> minor compactions should be taking care of reducing the number of store
> files. The caveat of doing it more frequently is the additional
> disk/network I/O.
>
> Can you please elaborate more on "reduce stress by spreading the load." Is
> there anything else you are seeing in your cluster that is suggesting to
> you to lower the period for major compactions?
>
> esteban.
>
> --
> Cloudera, Inc.
>
>
> On Mon, Apr 4, 2016 at 8:35 AM, Sumit Nigam <sumit_only@yahoo.com.invalid>
> wrote:
>
> > Hi,
> > Are there major overheads to running major compaction frequently? As much
> > as I know, it produces one Hfile for a region and processes delete
> markers
> > and version related drops. So, if this process has happened once say. a
> few
> > mins back then another major compaction should ideally not cause much
> harm.
> > Why I am trying to understand this is because Hbase also sets it to 24
> > hour default (for time based compaction) and I am looking to lower it to
> > say 20 mins to reduce stress by spreading the load.
> > Or am I completely off-track?
> > Thanks,Sumit
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message