hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From anil gupta <anilgupt...@gmail.com>
Subject Re: is it a good idea to disable tables not currently hot?
Date Fri, 18 Mar 2016 17:36:32 GMT
@Frank, regarding write amplification:
1. What is your flush size? default is 128 MB. You should increase your
"hbase.hregion.memstore.flush.size" so that you dont run over the limit of
store files.
2. Have a look at "hbase.regionserver.global.memstore.lowerLimit".
3. Your heap size is also too big. Maybe you also run into GC issues. Have
you checked your GC logs?
4. IMO, writes getting blocks at 9 files might be very less for a big
Region Server. So, you can also consider increasing that.

On Fri, Mar 18, 2016 at 10:22 AM, Frank Luo <jluo@merkleinc.com> wrote:

> Ted,
>
> Thanks for sharing. I learned something today.
>
> But I guess it doesn't apply to my case.  It is true that I only run a
> client for a few hours in a day, but the data is not date based.
>
> -----Original Message-----
> From: Ted Yu [mailto:yuzhihong@gmail.com]
> Sent: Friday, March 18, 2016 12:10 PM
> To: user@hbase.apache.org
> Subject: Re: is it a good idea to disable tables not currently hot?
>
> Frank:
> Can you take a look at the following to see if it may help with your use
> case(s) ?
>
> HBASE-15181 A simple implementation of date based tiered compaction
>
> Cheers
>
> On Fri, Mar 18, 2016 at 9:58 AM, Frank Luo <jluo@merkleinc.com> wrote:
>
> > There are two reasons I am hesitating going that route.
> >
> > One is that most of tables are fairly small. Going to 10GB will force
> > tables to shrink to some nodes but not evenly distributed around the
> > cluster, hence discouraging parallelism. But I think I can manage this
> > issue if the second is resolved.
> >
> > The second issue, which I have battled with for two years now, is that
> > I am doing online puts, which occasionally triggers compacts when a
> > region is heavily inserted, and whenever it happens, all subsequent
> > read/write are all on hold and I can see time out error on the client
> > side. A typical compact runs for 4 minutes now and I have to increase
> > timeout on a number of places to accommodate that. So if I increase
> > the size to 10 GB, will compact time double?
> >
> > -----Original Message-----
> > From: Jean-Marc Spaggiari [mailto:jean-marc@spaggiari.org]
> > Sent: Friday, March 18, 2016 11:34 AM
> > To: user
> > Subject: Re: is it a good idea to disable tables not currently hot?
> >
> > So you can safely increase your maximum region size to 10GB, which
> > will divide the number of regions by 2. When you will be on 1.1.2 you
> > can also do online merge to reduce this number of regions. The might
> help too.
> >
> > JMS
> >
> > 2016-03-18 12:32 GMT-04:00 Frank Luo <jluo@merkleinc.com>:
> >
> > > 0.98 on hdp 2.2 currently.
> > >
> > > Soon will be on hdp2.3.4, which has HBase 1.1.2.
> > >
> > > -----Original Message-----
> > > From: Jean-Marc Spaggiari [mailto:jean-marc@spaggiari.org]
> > > Sent: Friday, March 18, 2016 11:29 AM
> > > To: user
> > > Subject: Re: is it a good idea to disable tables not currently hot?
> > >
> > > Hi Frank,
> > >
> > > It might be doable.
> > >
> > > What HBase version are you running?
> > >
> > > JMS
> > >
> > > 2016-03-18 12:25 GMT-04:00 Frank Luo <jluo@merkleinc.com>:
> > >
> > > > No one has experience disabling tables?
> > > >
> > > > -----Original Message-----
> > > > From: Frank Luo [mailto:jluo@merkleinc.com]
> > > > Sent: Thursday, March 17, 2016 4:51 PM
> > > > To: user@hbase.apache.org
> > > > Subject: is it a good idea to disable tables not currently hot?
> > > >
> > > > We have a multi tenants environment and each client occupies x
> > > > number of hbase regions. We currently have about 500 regions per
> > > > region server and I understand the guideline is less than 200. So
> > > > we need to reduce the region counts. Increasing region file size
> > > > is no more an option because we are already at 5G and I don’t want
> > > > to go
> > higher.
> > > >
> > > > Due to our unique use cases, all clients are running for a few
> > > > hours in a day, then being quiet for the rest of time. So I am
> > > > thinking whether it is a good idea to disable all quiet tables and
> > > > only enable them when they are ready to run. Does anyone have
> > > > experience on
> > that?
> > > >
> > > > One thing I worry about is the Balancer. I am pretty sure the
> > > > balancer will be confused when regions come and go. And I cannot
> > > > afford not to have it running in case of region server crashes and
> > > > come back. So doesn’t anyone have good ideas how to handle it?
> > > >
> > > > I already doing compact myself so that is not an issue.
> > > >
> > > > Another related question, if a region is enabled but not active
> > > > read/write, how much resources it takes in terms of region server?
> > > >
> > > > Thanks!
> > > >
> > > > Frank Luo
> > > >
> > >
> > > Merkle was named a leader in Customer Insights Services Providers by
> > > Forrester Research <
> > > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-
> > > ag
> > > ency/awards-recognition/merkle-named-leader-forrester?utm_source=ema
> > > il footer&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> > > >
> > >
> > > Forrester Research report names 500friends, a Merkle Company, a
> > > leader in customer Loyalty Solutions for Midsize Organizations<
> > > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-
> > > ag
> > > ency/awards-recognition/500friends-merkle-company-named?utm_source=e
> > > ma ilfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> > > >
> > > This email and any attachments transmitted with it are intended for
> > > use by the intended recipient(s) only. If you have received this
> > > email in error, please notify the sender immediately and then delete
> > > it. If you are not the intended recipient, you must not keep, use,
> > > disclose, copy or distribute this email without the author’s prior
> permission.
> > > We take precautions to minimize the risk of transmitting software
> > > viruses, but we advise you to perform your own virus checks on any
> > > attachment to this message. We cannot accept liability for any loss
> > > or damage caused by software viruses. The information contained in
> > > this communication may be confidential and may be subject to the
> > attorney-client privilege.
> > >
> > Merkle was named a leader in Customer Insights Services Providers by
> > Forrester Research <
> > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-ag
> > ency/awards-recognition/merkle-named-leader-forrester?utm_source=email
> > footer&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> > >
> >
> > Forrester Research report names 500friends, a Merkle Company, a leader
> > in customer Loyalty Solutions for Midsize Organizations<
> > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-ag
> > ency/awards-recognition/500friends-merkle-company-named?utm_source=ema
> > ilfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> > >
> > This email and any attachments transmitted with it are intended for
> > use by the intended recipient(s) only. If you have received this email
> > in error, please notify the sender immediately and then delete it. If
> > you are not the intended recipient, you must not keep, use, disclose,
> > copy or distribute this email without the author’s prior permission.
> > We take precautions to minimize the risk of transmitting software
> > viruses, but we advise you to perform your own virus checks on any
> > attachment to this message. We cannot accept liability for any loss or
> > damage caused by software viruses. The information contained in this
> > communication may be confidential and may be subject to the
> attorney-client privilege.
> >
> Merkle was named a leader in Customer Insights Services Providers by
> Forrester Research
> <
> http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agency/awards-recognition/merkle-named-leader-forrester?utm_source=emailfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> >
>
> Forrester Research report names 500friends, a Merkle Company, a leader in
> customer Loyalty Solutions for Midsize Organizations<
> http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agency/awards-recognition/500friends-merkle-company-named?utm_source=emailfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> >
> This email and any attachments transmitted with it are intended for use by
> the intended recipient(s) only. If you have received this email in error,
> please notify the sender immediately and then delete it. If you are not the
> intended recipient, you must not keep, use, disclose, copy or distribute
> this email without the author’s prior permission. We take precautions to
> minimize the risk of transmitting software viruses, but we advise you to
> perform your own virus checks on any attachment to this message. We cannot
> accept liability for any loss or damage caused by software viruses. The
> information contained in this communication may be confidential and may be
> subject to the attorney-client privilege.
>



-- 
Thanks & Regards,
Anil Gupta

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message