hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: is it a good idea to disable tables not currently hot?
Date Fri, 18 Mar 2016 20:30:31 GMT
bq. By default memsotre is 40%. Here it's 24%

bq. Memstore.lowerLimit=0.24

J-M:
Looks like you misread the config Frank listed.

On Fri, Mar 18, 2016 at 12:36 PM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:

> By default memsotre is 40%. Here it's 24%. There is a lot you might want to
> look at on your cluster and usecase :(
>
> 1) You might have long pause GCs causing issues. Think about offheap cache
> and reduce heap to less than 20GB
> 2) Way to many regions. Think about your usecases and tables design to
> reduce that. Increase region size to 10GB.
> 3) Increase your memstore to 40%. If your usecase is mostly puts and you
> have issues with that, increase it.
> 4) Take a look at your flush size. It's useless to increase it to 256MB if
> you are already flushing only few KBs at a time.
> 5) etc. :(
>
> JMS
>
> 2016-03-18 15:26 GMT-04:00 Frank Luo <jluo@merkleinc.com>:
>
> > Anil/Jean,
> >
> > Thanks for the tips. Very helpful.
> >
> > To answer your question. I just checked, the Region server's heap is 32G,
> > instead of 36G as I previously stated, but it is in the same range and I
> do
> > see long pause on GC.
> >
> > I think the reason it was set to a high value was that we used to have
> > 2000 regions per server, before we increased region file size from the
> > default to compressed 5G.
> >
> > So what should be the heap right size given 5G file size, 400 region per
> > server on an 80 node cluster?
> >
> > At this time, I think memstore related setting are all defaults from HDP.
> >
> >         Flush.size = 128M
> >         Memstore.lowerLimit=0.24
> >         Memstore.upperLimit=0.25
> >         Hstore.blocking.storefiles=10
> >
> > -----Original Message-----
> > From: anil gupta [mailto:anilgupta84@gmail.com]
> > Sent: Friday, March 18, 2016 12:37 PM
> > To: user@hbase.apache.org
> > Subject: Re: is it a good idea to disable tables not currently hot?
> >
> > @Frank, regarding write amplification:
> > 1. What is your flush size? default is 128 MB. You should increase your
> > "hbase.hregion.memstore.flush.size" so that you dont run over the limit
> of
> > store files.
> > 2. Have a look at "hbase.regionserver.global.memstore.lowerLimit".
> > 3. Your heap size is also too big. Maybe you also run into GC issues.
> Have
> > you checked your GC logs?
> > 4. IMO, writes getting blocks at 9 files might be very less for a big
> > Region Server. So, you can also consider increasing that.
> >
> > On Fri, Mar 18, 2016 at 10:22 AM, Frank Luo <jluo@merkleinc.com> wrote:
> >
> > > Ted,
> > >
> > > Thanks for sharing. I learned something today.
> > >
> > > But I guess it doesn't apply to my case.  It is true that I only run a
> > > client for a few hours in a day, but the data is not date based.
> > >
> > > -----Original Message-----
> > > From: Ted Yu [mailto:yuzhihong@gmail.com]
> > > Sent: Friday, March 18, 2016 12:10 PM
> > > To: user@hbase.apache.org
> > > Subject: Re: is it a good idea to disable tables not currently hot?
> > >
> > > Frank:
> > > Can you take a look at the following to see if it may help with your
> > > use
> > > case(s) ?
> > >
> > > HBASE-15181 A simple implementation of date based tiered compaction
> > >
> > > Cheers
> > >
> > > On Fri, Mar 18, 2016 at 9:58 AM, Frank Luo <jluo@merkleinc.com> wrote:
> > >
> > > > There are two reasons I am hesitating going that route.
> > > >
> > > > One is that most of tables are fairly small. Going to 10GB will
> > > > force tables to shrink to some nodes but not evenly distributed
> > > > around the cluster, hence discouraging parallelism. But I think I
> > > > can manage this issue if the second is resolved.
> > > >
> > > > The second issue, which I have battled with for two years now, is
> > > > that I am doing online puts, which occasionally triggers compacts
> > > > when a region is heavily inserted, and whenever it happens, all
> > > > subsequent read/write are all on hold and I can see time out error
> > > > on the client side. A typical compact runs for 4 minutes now and I
> > > > have to increase timeout on a number of places to accommodate that.
> > > > So if I increase the size to 10 GB, will compact time double?
> > > >
> > > > -----Original Message-----
> > > > From: Jean-Marc Spaggiari [mailto:jean-marc@spaggiari.org]
> > > > Sent: Friday, March 18, 2016 11:34 AM
> > > > To: user
> > > > Subject: Re: is it a good idea to disable tables not currently hot?
> > > >
> > > > So you can safely increase your maximum region size to 10GB, which
> > > > will divide the number of regions by 2. When you will be on 1.1.2
> > > > you can also do online merge to reduce this number of regions. The
> > > > might
> > > help too.
> > > >
> > > > JMS
> > > >
> > > > 2016-03-18 12:32 GMT-04:00 Frank Luo <jluo@merkleinc.com>:
> > > >
> > > > > 0.98 on hdp 2.2 currently.
> > > > >
> > > > > Soon will be on hdp2.3.4, which has HBase 1.1.2.
> > > > >
> > > > > -----Original Message-----
> > > > > From: Jean-Marc Spaggiari [mailto:jean-marc@spaggiari.org]
> > > > > Sent: Friday, March 18, 2016 11:29 AM
> > > > > To: user
> > > > > Subject: Re: is it a good idea to disable tables not currently hot?
> > > > >
> > > > > Hi Frank,
> > > > >
> > > > > It might be doable.
> > > > >
> > > > > What HBase version are you running?
> > > > >
> > > > > JMS
> > > > >
> > > > > 2016-03-18 12:25 GMT-04:00 Frank Luo <jluo@merkleinc.com>:
> > > > >
> > > > > > No one has experience disabling tables?
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From: Frank Luo [mailto:jluo@merkleinc.com]
> > > > > > Sent: Thursday, March 17, 2016 4:51 PM
> > > > > > To: user@hbase.apache.org
> > > > > > Subject: is it a good idea to disable tables not currently hot?
> > > > > >
> > > > > > We have a multi tenants environment and each client occupies
x
> > > > > > number of hbase regions. We currently have about 500 regions
per
> > > > > > region server and I understand the guideline is less than 200.
> > > > > > So we need to reduce the region counts. Increasing region file
> > > > > > size is no more an option because we are already at 5G and I
> > > > > > don’t want to go
> > > > higher.
> > > > > >
> > > > > > Due to our unique use cases, all clients are running for a few
> > > > > > hours in a day, then being quiet for the rest of time. So I
am
> > > > > > thinking whether it is a good idea to disable all quiet tables
> > > > > > and only enable them when they are ready to run. Does anyone
> > > > > > have experience on
> > > > that?
> > > > > >
> > > > > > One thing I worry about is the Balancer. I am pretty sure the
> > > > > > balancer will be confused when regions come and go. And I cannot
> > > > > > afford not to have it running in case of region server crashes
> > > > > > and come back. So doesn’t anyone have good ideas how to handle
> it?
> > > > > >
> > > > > > I already doing compact myself so that is not an issue.
> > > > > >
> > > > > > Another related question, if a region is enabled but not active
> > > > > > read/write, how much resources it takes in terms of region
> server?
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > Frank Luo
> > > > > >
> > > > >
> > > > > Merkle was named a leader in Customer Insights Services Providers
> > > > > by Forrester Research <
> > > > > http://www.merkleinc.com/who-we-are-customer-relationship-marketin
> > > > > g-
> > > > > ag
> > > > > ency/awards-recognition/merkle-named-leader-forrester?utm_source=e
> > > > > ma il
> > > > > footer&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> > > > > >
> > > > >
> > > > > Forrester Research report names 500friends, a Merkle Company, a
> > > > > leader in customer Loyalty Solutions for Midsize Organizations<
> > > > > http://www.merkleinc.com/who-we-are-customer-relationship-marketin
> > > > > g-
> > > > > ag
> > > > > ency/awards-recognition/500friends-merkle-company-named?utm_source
> > > > > =e ma
> > > > > ilfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> > > > > >
> > > > > This email and any attachments transmitted with it are intended
> > > > > for use by the intended recipient(s) only. If you have received
> > > > > this email in error, please notify the sender immediately and then
> > > > > delete it. If you are not the intended recipient, you must not
> > > > > keep, use, disclose, copy or distribute this email without the
> > > > > author’s prior
> > > permission.
> > > > > We take precautions to minimize the risk of transmitting software
> > > > > viruses, but we advise you to perform your own virus checks on any
> > > > > attachment to this message. We cannot accept liability for any
> > > > > loss or damage caused by software viruses. The information
> > > > > contained in this communication may be confidential and may be
> > > > > subject to the
> > > > attorney-client privilege.
> > > > >
> > > > Merkle was named a leader in Customer Insights Services Providers by
> > > > Forrester Research <
> > > > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-
> > > > ag
> > > > ency/awards-recognition/merkle-named-leader-forrester?utm_source=ema
> > > > il footer&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> > > > >
> > > >
> > > > Forrester Research report names 500friends, a Merkle Company, a
> > > > leader in customer Loyalty Solutions for Midsize Organizations<
> > > > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-
> > > > ag
> > > > ency/awards-recognition/500friends-merkle-company-named?utm_source=e
> > > > ma ilfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> > > > >
> > > > This email and any attachments transmitted with it are intended for
> > > > use by the intended recipient(s) only. If you have received this
> > > > email in error, please notify the sender immediately and then delete
> > > > it. If you are not the intended recipient, you must not keep, use,
> > > > disclose, copy or distribute this email without the author’s prior
> > permission.
> > > > We take precautions to minimize the risk of transmitting software
> > > > viruses, but we advise you to perform your own virus checks on any
> > > > attachment to this message. We cannot accept liability for any loss
> > > > or damage caused by software viruses. The information contained in
> > > > this communication may be confidential and may be subject to the
> > > attorney-client privilege.
> > > >
> > > Merkle was named a leader in Customer Insights Services Providers by
> > > Forrester Research <
> > > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-ag
> > > ency/awards-recognition/merkle-named-leader-forrester?utm_source=email
> > > footer&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> > > >
> > >
> > > Forrester Research report names 500friends, a Merkle Company, a leader
> > > in customer Loyalty Solutions for Midsize Organizations<
> > > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-ag
> > > ency/awards-recognition/500friends-merkle-company-named?utm_source=ema
> > > ilfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> > > >
> > > This email and any attachments transmitted with it are intended for
> > > use by the intended recipient(s) only. If you have received this email
> > > in error, please notify the sender immediately and then delete it. If
> > > you are not the intended recipient, you must not keep, use, disclose,
> > > copy or distribute this email without the author’s prior permission.
> > > We take precautions to minimize the risk of transmitting software
> > > viruses, but we advise you to perform your own virus checks on any
> > > attachment to this message. We cannot accept liability for any loss or
> > > damage caused by software viruses. The information contained in this
> > > communication may be confidential and may be subject to the
> > attorney-client privilege.
> > >
> >
> >
> >
> > --
> > Thanks & Regards,
> > Anil Gupta
> > Merkle was named a leader in Customer Insights Services Providers by
> > Forrester Research
> > <
> >
> http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agency/awards-recognition/merkle-named-leader-forrester?utm_source=emailfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> > >
> >
> > Forrester Research report names 500friends, a Merkle Company, a leader in
> > customer Loyalty Solutions for Midsize Organizations<
> >
> http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agency/awards-recognition/500friends-merkle-company-named?utm_source=emailfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> > >
> > This email and any attachments transmitted with it are intended for use
> by
> > the intended recipient(s) only. If you have received this email in error,
> > please notify the sender immediately and then delete it. If you are not
> the
> > intended recipient, you must not keep, use, disclose, copy or distribute
> > this email without the author’s prior permission. We take precautions to
> > minimize the risk of transmitting software viruses, but we advise you to
> > perform your own virus checks on any attachment to this message. We
> cannot
> > accept liability for any loss or damage caused by software viruses. The
> > information contained in this communication may be confidential and may
> be
> > subject to the attorney-client privilege.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message