hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Esteban Gutierrez <este...@cloudera.com>
Subject Re: Major compaction
Date Tue, 05 Apr 2016 16:48:53 GMT
Why is RS is going down? are you hitting an OOME or just the RS goes into
heavy GC and you get a YouAreDeadException? my guess is that if you have
thousands of store files and you are major compacting probably you are low
in resources and the compaction queue goes out of control and triggers an
OOME but thats just an educated guess.

Also if you are generating 100s or 1000s of store files per day probably
the memstore settings are not the optimal for your workload and you are
triggering too many premature flushes. If you could upload the RS logs to
pastebin or another similar site we can take a look into and see if thats
the case.

thanks!
esteban.

--
Cloudera, Inc.


On Mon, Apr 4, 2016 at 9:29 PM, Sumit Nigam <sumit_only@yahoo.com.invalid>
wrote:

> Hello all,
> Thanks a lot for your replies.
> @Frank - I will try the compactor you wrote and let you know how it goes.
> @Esteban - I am trying to understand how to reduce major compaction load.
> In my cluster whenever it happens, it takes down a region server or two. My
> setting are - disable time based compaction, make blocking files = 1000,
> min files to compact = 5, max files to compact = 10. Other settings are
> defaults. So, a question I had is if a major compaction of certain table is
> performed a few mins back, then would another major compaction of the same
> take a lot of time? If not, why not process delete markers/ version related
> expulsions more often by kicking in major compaction more often? Assuming,
> splits are not happening after every major compaction cycle, we should be
> able to manage major compaction better by spreading this load more evenly
> rather than doing once a day? Of course, I am missing something?
> @Vladimir - By same understanding that I mentioned above, would every
> major compaction really lead to more disk I/O or n/w or hbase process load?
> I assumed that the main things done by major compaction are deletes,
> creating a huge file per store and region splits. If this process has
> already been done say, a few mins back (or could be an hour back), then the
> next compaction should ideally have less to do. Not sure, if I am
> misunderstanding major compaction altogether.
> Please let me know your inputs.
> Best regards,Sumit
>
>       From: Frank Luo <jluo@merkleinc.com>
>  To: "user@hbase.apache.org" <user@hbase.apache.org>
> Cc: Sumit Nigam <sumit_only@yahoo.com>
>  Sent: Monday, April 4, 2016 11:07 PM
>  Subject: RE: Major compaction
>
> I wrote a small program to do MC in a "smart" way here:
> https://github.com/jinyeluo/smarthbasecompactor/
>
> Instead of blindly running MC on a table level, the program find a non-hot
> regions that has most store-files on a per region-server base, and run MC
> on them. Once done, it finds the next candidates... It just keeps on going
> until time is up.
>
> I am sure it has a lot area for improvement if something wants to go crazy
> on. But the code has been running for about half a year and it seems
> working well.
>
> -----Original Message-----
> From: Vladimir Rodionov [mailto:vladrodionov@gmail.com]
> Sent: Monday, April 04, 2016 12:15 PM
> To: user@hbase.apache.org
> Cc: Sumit Nigam <sumit_only@yahoo.com>
> Subject: Re: Major compaction
>
> >> Why I am trying to understand this is because Hbase also sets it to
> >> 24
> hour default (for time based compaction) and I am looking to lower it to
> say >> 20 mins to reduce stress by spreading the load.
>
> The more frequently you run major compaction the more IO (disk/network)
> you consume.
>
> Usually, in production environment, periodic major compactions are
> disabled and run manually  to avoid major compaction storms.
>
> To control major compaction completely you will also need to disable
> promotion minor compaction to major ones. You can do this, by setting
> maximum compaction size for minor compaction:
> *hbase.hstore.compaction.max.size*
>
> -Vlad
>
>
> On Mon, Apr 4, 2016 at 8:55 AM, Esteban Gutierrez <esteban@cloudera.com>
> wrote:
>
> > Hello Sumit,
> >
> > Ideally you shouldn't be triggering major compactions that frequently
> > since minor compactions should be taking care of reducing the number
> > of store files. The caveat of doing it more frequently is the
> > additional disk/network I/O.
> >
> > Can you please elaborate more on "reduce stress by spreading the
> > load." Is there anything else you are seeing in your cluster that is
> > suggesting to you to lower the period for major compactions?
> >
> > esteban.
> >
> > --
> > Cloudera, Inc.
> >
> >
> > On Mon, Apr 4, 2016 at 8:35 AM, Sumit Nigam
> > <sumit_only@yahoo.com.invalid>
> > wrote:
> >
> > > Hi,
> > > Are there major overheads to running major compaction frequently? As
> > > much as I know, it produces one Hfile for a region and processes
> > > delete
> > markers
> > > and version related drops. So, if this process has happened once
> > > say. a
> > few
> > > mins back then another major compaction should ideally not cause
> > > much
> > harm.
> > > Why I am trying to understand this is because Hbase also sets it to
> > > 24 hour default (for time based compaction) and I am looking to
> > > lower it to say 20 mins to reduce stress by spreading the load.
> > > Or am I completely off-track?
> > > Thanks,Sumit
> >
> Merkle was named a leader in Customer Insights Services Providers by
> Forrester Research
> <
> http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agency/awards-recognition/merkle-named-leader-forrester?utm_source=emailfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> >
>
> Forrester Research report names 500friends, a Merkle Company, a leader in
> customer Loyalty Solutions for Midsize Organizations<
> http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agency/awards-recognition/500friends-merkle-company-named?utm_source=emailfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> >
> This email and any attachments transmitted with it are intended for use by
> the intended recipient(s) only. If you have received this email in error,
> please notify the sender immediately and then delete it. If you are not the
> intended recipient, you must not keep, use, disclose, copy or distribute
> this email without the author’s prior permission. We take precautions to
> minimize the risk of transmitting software viruses, but we advise you to
> perform your own virus checks on any attachment to this message. We cannot
> accept liability for any loss or damage caused by software viruses. The
> information contained in this communication may be confidential and may be
> subject to the attorney-client privilege.
>
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message