hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: is it a good idea to disable tables not currently hot?
Date Fri, 18 Mar 2016 17:10:16 GMT
Frank:
Can you take a look at the following to see if it may help with your use
case(s) ?

HBASE-15181 A simple implementation of date based tiered compaction

Cheers

On Fri, Mar 18, 2016 at 9:58 AM, Frank Luo <jluo@merkleinc.com> wrote:

> There are two reasons I am hesitating going that route.
>
> One is that most of tables are fairly small. Going to 10GB will force
> tables to shrink to some nodes but not evenly distributed around the
> cluster, hence discouraging parallelism. But I think I can manage this
> issue if the second is resolved.
>
> The second issue, which I have battled with for two years now, is that I
> am doing online puts, which occasionally triggers compacts when a region is
> heavily inserted, and whenever it happens, all subsequent read/write are
> all on hold and I can see time out error on the client side. A typical
> compact runs for 4 minutes now and I have to increase timeout on a number
> of places to accommodate that. So if I increase the size to 10 GB, will
> compact time double?
>
> -----Original Message-----
> From: Jean-Marc Spaggiari [mailto:jean-marc@spaggiari.org]
> Sent: Friday, March 18, 2016 11:34 AM
> To: user
> Subject: Re: is it a good idea to disable tables not currently hot?
>
> So you can safely increase your maximum region size to 10GB, which will
> divide the number of regions by 2. When you will be on 1.1.2 you can also
> do online merge to reduce this number of regions. The might help too.
>
> JMS
>
> 2016-03-18 12:32 GMT-04:00 Frank Luo <jluo@merkleinc.com>:
>
> > 0.98 on hdp 2.2 currently.
> >
> > Soon will be on hdp2.3.4, which has HBase 1.1.2.
> >
> > -----Original Message-----
> > From: Jean-Marc Spaggiari [mailto:jean-marc@spaggiari.org]
> > Sent: Friday, March 18, 2016 11:29 AM
> > To: user
> > Subject: Re: is it a good idea to disable tables not currently hot?
> >
> > Hi Frank,
> >
> > It might be doable.
> >
> > What HBase version are you running?
> >
> > JMS
> >
> > 2016-03-18 12:25 GMT-04:00 Frank Luo <jluo@merkleinc.com>:
> >
> > > No one has experience disabling tables?
> > >
> > > -----Original Message-----
> > > From: Frank Luo [mailto:jluo@merkleinc.com]
> > > Sent: Thursday, March 17, 2016 4:51 PM
> > > To: user@hbase.apache.org
> > > Subject: is it a good idea to disable tables not currently hot?
> > >
> > > We have a multi tenants environment and each client occupies x
> > > number of hbase regions. We currently have about 500 regions per
> > > region server and I understand the guideline is less than 200. So we
> > > need to reduce the region counts. Increasing region file size is no
> > > more an option because we are already at 5G and I don’t want to go
> higher.
> > >
> > > Due to our unique use cases, all clients are running for a few hours
> > > in a day, then being quiet for the rest of time. So I am thinking
> > > whether it is a good idea to disable all quiet tables and only
> > > enable them when they are ready to run. Does anyone have experience on
> that?
> > >
> > > One thing I worry about is the Balancer. I am pretty sure the
> > > balancer will be confused when regions come and go. And I cannot
> > > afford not to have it running in case of region server crashes and
> > > come back. So doesn’t anyone have good ideas how to handle it?
> > >
> > > I already doing compact myself so that is not an issue.
> > >
> > > Another related question, if a region is enabled but not active
> > > read/write, how much resources it takes in terms of region server?
> > >
> > > Thanks!
> > >
> > > Frank Luo
> > >
> >
> > Merkle was named a leader in Customer Insights Services Providers by
> > Forrester Research <
> > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-ag
> > ency/awards-recognition/merkle-named-leader-forrester?utm_source=email
> > footer&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> > >
> >
> > Forrester Research report names 500friends, a Merkle Company, a leader
> > in customer Loyalty Solutions for Midsize Organizations<
> > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-ag
> > ency/awards-recognition/500friends-merkle-company-named?utm_source=ema
> > ilfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> > >
> > This email and any attachments transmitted with it are intended for
> > use by the intended recipient(s) only. If you have received this email
> > in error, please notify the sender immediately and then delete it. If
> > you are not the intended recipient, you must not keep, use, disclose,
> > copy or distribute this email without the author’s prior permission.
> > We take precautions to minimize the risk of transmitting software
> > viruses, but we advise you to perform your own virus checks on any
> > attachment to this message. We cannot accept liability for any loss or
> > damage caused by software viruses. The information contained in this
> > communication may be confidential and may be subject to the
> attorney-client privilege.
> >
> Merkle was named a leader in Customer Insights Services Providers by
> Forrester Research
> <
> http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agency/awards-recognition/merkle-named-leader-forrester?utm_source=emailfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> >
>
> Forrester Research report names 500friends, a Merkle Company, a leader in
> customer Loyalty Solutions for Midsize Organizations<
> http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agency/awards-recognition/500friends-merkle-company-named?utm_source=emailfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> >
> This email and any attachments transmitted with it are intended for use by
> the intended recipient(s) only. If you have received this email in error,
> please notify the sender immediately and then delete it. If you are not the
> intended recipient, you must not keep, use, disclose, copy or distribute
> this email without the author’s prior permission. We take precautions to
> minimize the risk of transmitting software viruses, but we advise you to
> perform your own virus checks on any attachment to this message. We cannot
> accept liability for any loss or damage caused by software viruses. The
> information contained in this communication may be confidential and may be
> subject to the attorney-client privilege.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message