Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <1EFFC0EDF85B044A99A02C925FDA3755200EB576@HQITEXCH07.pclc0.merkle.local>
References: 
 <1EFFC0EDF85B044A99A02C925FDA3755200EA573@HQITEXCH07.pclc0.merkle.local>
 <1EFFC0EDF85B044A99A02C925FDA3755200EAEDD@HQITEXCH07.pclc0.merkle.local>
 <CAPQV63VxyMRX6gvE3UNpnq1fBK+M23uVb9RSVvNVqgkrNAZsuA@mail.gmail.com>
 <1EFFC0EDF85B044A99A02C925FDA3755200EAF13@HQITEXCH07.pclc0.merkle.local>
 <CAPQV63XQmWSqzwBKiQA97nNQXUMdd=d6MKFyKUJxz5gAVFuh2Q@mail.gmail.com>
 <1EFFC0EDF85B044A99A02C925FDA3755200EAFC9@HQITEXCH07.pclc0.merkle.local>
 <CALte62zrTfUnyOkuPevBbz_jTOJWuAfHY5B6RAarjFW0xuZW6g@mail.gmail.com>
 <1EFFC0EDF85B044A99A02C925FDA3755200EB0E4@HQITEXCH07.pclc0.merkle.local>
 <CAF1+Vs8P7vSyQ-8e9NS+7Y0fTr1f-Wyn1QBVg9sLzYs68Ef8hw@mail.gmail.com>
 <1EFFC0EDF85B044A99A02C925FDA3755200EB576@HQITEXCH07.pclc0.merkle.local>
From: Jean-Marc Spaggiari <jean-marc@spaggiari.org>
Date: Fri, 18 Mar 2016 15:36:23 -0400
Message-ID: 
 <CAPQV63VhtNuiL2TbKkdLFTaiy10AQjTqha7v2nTOy3oHg_NDPg@mail.gmail.com>
Subject: Re: is it a good idea to disable tables not currently hot?
To: Frank Luo <jluo@merkleinc.com>
Cc: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a113a7e7cda9c66052e57dd49

--001a113a7e7cda9c66052e57dd49
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

By default memsotre is 40%. Here it's 24%. There is a lot you might want to
look at on your cluster and usecase :(

1) You might have long pause GCs causing issues. Think about offheap cache
and reduce heap to less than 20GB
2) Way to many regions. Think about your usecases and tables design to
reduce that. Increase region size to 10GB.
3) Increase your memstore to 40%. If your usecase is mostly puts and you
have issues with that, increase it.
4) Take a look at your flush size. It's useless to increase it to 256MB if
you are already flushing only few KBs at a time.
5) etc. :(

JMS

2016-03-18 15:26 GMT-04:00 Frank Luo <jluo@merkleinc.com>:

> Anil/Jean,
>
> Thanks for the tips. Very helpful.
>
> To answer your question. I just checked, the Region server's heap is 32G,
> instead of 36G as I previously stated, but it is in the same range and I =
do
> see long pause on GC.
>
> I think the reason it was set to a high value was that we used to have
> 2000 regions per server, before we increased region file size from the
> default to compressed 5G.
>
> So what should be the heap right size given 5G file size, 400 region per
> server on an 80 node cluster?
>
> At this time, I think memstore related setting are all defaults from HDP.
>
>         Flush.size =3D 128M
>         Memstore.lowerLimit=3D0.24
>         Memstore.upperLimit=3D0.25
>         Hstore.blocking.storefiles=3D10
>
> -----Original Message-----
> From: anil gupta [mailto:anilgupta84@gmail.com]
> Sent: Friday, March 18, 2016 12:37 PM
> To: user@hbase.apache.org
> Subject: Re: is it a good idea to disable tables not currently hot?
>
> @Frank, regarding write amplification:
> 1. What is your flush size? default is 128 MB. You should increase your
> "hbase.hregion.memstore.flush.size" so that you dont run over the limit o=
f
> store files.
> 2. Have a look at "hbase.regionserver.global.memstore.lowerLimit".
> 3. Your heap size is also too big. Maybe you also run into GC issues. Hav=
e
> you checked your GC logs?
> 4. IMO, writes getting blocks at 9 files might be very less for a big
> Region Server. So, you can also consider increasing that.
>
> On Fri, Mar 18, 2016 at 10:22 AM, Frank Luo <jluo@merkleinc.com> wrote:
>
> > Ted,
> >
> > Thanks for sharing. I learned something today.
> >
> > But I guess it doesn't apply to my case.  It is true that I only run a
> > client for a few hours in a day, but the data is not date based.
> >
> > -----Original Message-----
> > From: Ted Yu [mailto:yuzhihong@gmail.com]
> > Sent: Friday, March 18, 2016 12:10 PM
> > To: user@hbase.apache.org
> > Subject: Re: is it a good idea to disable tables not currently hot?
> >
> > Frank:
> > Can you take a look at the following to see if it may help with your
> > use
> > case(s) ?
> >
> > HBASE-15181 A simple implementation of date based tiered compaction
> >
> > Cheers
> >
> > On Fri, Mar 18, 2016 at 9:58 AM, Frank Luo <jluo@merkleinc.com> wrote:
> >
> > > There are two reasons I am hesitating going that route.
> > >
> > > One is that most of tables are fairly small. Going to 10GB will
> > > force tables to shrink to some nodes but not evenly distributed
> > > around the cluster, hence discouraging parallelism. But I think I
> > > can manage this issue if the second is resolved.
> > >
> > > The second issue, which I have battled with for two years now, is
> > > that I am doing online puts, which occasionally triggers compacts
> > > when a region is heavily inserted, and whenever it happens, all
> > > subsequent read/write are all on hold and I can see time out error
> > > on the client side. A typical compact runs for 4 minutes now and I
> > > have to increase timeout on a number of places to accommodate that.
> > > So if I increase the size to 10 GB, will compact time double?
> > >
> > > -----Original Message-----
> > > From: Jean-Marc Spaggiari [mailto:jean-marc@spaggiari.org]
> > > Sent: Friday, March 18, 2016 11:34 AM
> > > To: user
> > > Subject: Re: is it a good idea to disable tables not currently hot?
> > >
> > > So you can safely increase your maximum region size to 10GB, which
> > > will divide the number of regions by 2. When you will be on 1.1.2
> > > you can also do online merge to reduce this number of regions. The
> > > might
> > help too.
> > >
> > > JMS
> > >
> > > 2016-03-18 12:32 GMT-04:00 Frank Luo <jluo@merkleinc.com>:
> > >
> > > > 0.98 on hdp 2.2 currently.
> > > >
> > > > Soon will be on hdp2.3.4, which has HBase 1.1.2.
> > > >
> > > > -----Original Message-----
> > > > From: Jean-Marc Spaggiari [mailto:jean-marc@spaggiari.org]
> > > > Sent: Friday, March 18, 2016 11:29 AM
> > > > To: user
> > > > Subject: Re: is it a good idea to disable tables not currently hot?
> > > >
> > > > Hi Frank,
> > > >
> > > > It might be doable.
> > > >
> > > > What HBase version are you running?
> > > >
> > > > JMS
> > > >
> > > > 2016-03-18 12:25 GMT-04:00 Frank Luo <jluo@merkleinc.com>:
> > > >
> > > > > No one has experience disabling tables?
> > > > >
> > > > > -----Original Message-----
> > > > > From: Frank Luo [mailto:jluo@merkleinc.com]
> > > > > Sent: Thursday, March 17, 2016 4:51 PM
> > > > > To: user@hbase.apache.org
> > > > > Subject: is it a good idea to disable tables not currently hot?
> > > > >
> > > > > We have a multi tenants environment and each client occupies x
> > > > > number of hbase regions. We currently have about 500 regions per
> > > > > region server and I understand the guideline is less than 200.
> > > > > So we need to reduce the region counts. Increasing region file
> > > > > size is no more an option because we are already at 5G and I
> > > > > don=E2=80=99t want to go
> > > higher.
> > > > >
> > > > > Due to our unique use cases, all clients are running for a few
> > > > > hours in a day, then being quiet for the rest of time. So I am
> > > > > thinking whether it is a good idea to disable all quiet tables
> > > > > and only enable them when they are ready to run. Does anyone
> > > > > have experience on
> > > that?
> > > > >
> > > > > One thing I worry about is the Balancer. I am pretty sure the
> > > > > balancer will be confused when regions come and go. And I cannot
> > > > > afford not to have it running in case of region server crashes
> > > > > and come back. So doesn=E2=80=99t anyone have good ideas how to h=
andle it?
> > > > >
> > > > > I already doing compact myself so that is not an issue.
> > > > >
> > > > > Another related question, if a region is enabled but not active
> > > > > read/write, how much resources it takes in terms of region server=
?
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Frank Luo
> > > > >
> > > >
> > > > Merkle was named a leader in Customer Insights Services Providers
> > > > by Forrester Research <
> > > > http://www.merkleinc.com/who-we-are-customer-relationship-marketin
> > > > g-
> > > > ag
> > > > ency/awards-recognition/merkle-named-leader-forrester?utm_source=3D=
e
> > > > ma il
> > > > footer&utm_medium=3Demail&utm_campaign=3D2016MonthlyEmployeeFooter
> > > > >
> > > >
> > > > Forrester Research report names 500friends, a Merkle Company, a
> > > > leader in customer Loyalty Solutions for Midsize Organizations<
> > > > http://www.merkleinc.com/who-we-are-customer-relationship-marketin
> > > > g-
> > > > ag
> > > > ency/awards-recognition/500friends-merkle-company-named?utm_source
> > > > =3De ma
> > > > ilfooter&utm_medium=3Demail&utm_campaign=3D2016MonthlyEmployeeFoote=
r
> > > > >
> > > > This email and any attachments transmitted with it are intended
> > > > for use by the intended recipient(s) only. If you have received
> > > > this email in error, please notify the sender immediately and then
> > > > delete it. If you are not the intended recipient, you must not
> > > > keep, use, disclose, copy or distribute this email without the
> > > > author=E2=80=99s prior
> > permission.
> > > > We take precautions to minimize the risk of transmitting software
> > > > viruses, but we advise you to perform your own virus checks on any
> > > > attachment to this message. We cannot accept liability for any
> > > > loss or damage caused by software viruses. The information
> > > > contained in this communication may be confidential and may be
> > > > subject to the
> > > attorney-client privilege.
> > > >
> > > Merkle was named a leader in Customer Insights Services Providers by
> > > Forrester Research <
> > > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-
> > > ag
> > > ency/awards-recognition/merkle-named-leader-forrester?utm_source=3Dem=
a
> > > il footer&utm_medium=3Demail&utm_campaign=3D2016MonthlyEmployeeFooter
> > > >
> > >
> > > Forrester Research report names 500friends, a Merkle Company, a
> > > leader in customer Loyalty Solutions for Midsize Organizations<
> > > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-
> > > ag
> > > ency/awards-recognition/500friends-merkle-company-named?utm_source=3D=
e
> > > ma ilfooter&utm_medium=3Demail&utm_campaign=3D2016MonthlyEmployeeFoot=
er
> > > >
> > > This email and any attachments transmitted with it are intended for
> > > use by the intended recipient(s) only. If you have received this
> > > email in error, please notify the sender immediately and then delete
> > > it. If you are not the intended recipient, you must not keep, use,
> > > disclose, copy or distribute this email without the author=E2=80=99s =
prior
> permission.
> > > We take precautions to minimize the risk of transmitting software
> > > viruses, but we advise you to perform your own virus checks on any
> > > attachment to this message. We cannot accept liability for any loss
> > > or damage caused by software viruses. The information contained in
> > > this communication may be confidential and may be subject to the
> > attorney-client privilege.
> > >
> > Merkle was named a leader in Customer Insights Services Providers by
> > Forrester Research <
> > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-ag
> > ency/awards-recognition/merkle-named-leader-forrester?utm_source=3Demai=
l
> > footer&utm_medium=3Demail&utm_campaign=3D2016MonthlyEmployeeFooter
> > >
> >
> > Forrester Research report names 500friends, a Merkle Company, a leader
> > in customer Loyalty Solutions for Midsize Organizations<
> > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-ag
> > ency/awards-recognition/500friends-merkle-company-named?utm_source=3Dem=
a
> > ilfooter&utm_medium=3Demail&utm_campaign=3D2016MonthlyEmployeeFooter
> > >
> > This email and any attachments transmitted with it are intended for
> > use by the intended recipient(s) only. If you have received this email
> > in error, please notify the sender immediately and then delete it. If
> > you are not the intended recipient, you must not keep, use, disclose,
> > copy or distribute this email without the author=E2=80=99s prior permis=
sion.
> > We take precautions to minimize the risk of transmitting software
> > viruses, but we advise you to perform your own virus checks on any
> > attachment to this message. We cannot accept liability for any loss or
> > damage caused by software viruses. The information contained in this
> > communication may be confidential and may be subject to the
> attorney-client privilege.
> >
>
>
>
> --
> Thanks & Regards,
> Anil Gupta
> Merkle was named a leader in Customer Insights Services Providers by
> Forrester Research
> <
> http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agenc=
y/awards-recognition/merkle-named-leader-forrester?utm_source=3Demailfooter=
&utm_medium=3Demail&utm_campaign=3D2016MonthlyEmployeeFooter
> >
>
> Forrester Research report names 500friends, a Merkle Company, a leader in
> customer Loyalty Solutions for Midsize Organizations<
> http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agenc=
y/awards-recognition/500friends-merkle-company-named?utm_source=3Demailfoot=
er&utm_medium=3Demail&utm_campaign=3D2016MonthlyEmployeeFooter
> >
> This email and any attachments transmitted with it are intended for use b=
y
> the intended recipient(s) only. If you have received this email in error,
> please notify the sender immediately and then delete it. If you are not t=
he
> intended recipient, you must not keep, use, disclose, copy or distribute
> this email without the author=E2=80=99s prior permission. We take precaut=
ions to
> minimize the risk of transmitting software viruses, but we advise you to
> perform your own virus checks on any attachment to this message. We canno=
t
> accept liability for any loss or damage caused by software viruses. The
> information contained in this communication may be confidential and may b=
e
> subject to the attorney-client privilege.
>

--001a113a7e7cda9c66052e57dd49--