hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frank Luo <j...@merkleinc.com>
Subject RE: is it a good idea to disable tables not currently hot?
Date Fri, 18 Mar 2016 17:14:54 GMT
>Compactions should not hold the writes except if there is to many stores files.

I believe too many store files (9 I think?) is indeed the trigger for compact. I always do
MajorCompact before inserting.


>  What does your heap size look like?

36GB allocated to a region server.

> Do you have anything on the logs when this happen?

Haven't really looked the log files. I got most of information from the web console.

> Also, have you think about eventually merging some of the tables together?

Haven't thought about it. I might go there if no other options.

-----Original Message-----
From: Jean-Marc Spaggiari [mailto:jean-marc@spaggiari.org]
Sent: Friday, March 18, 2016 12:02 PM
To: user
Subject: Re: is it a good idea to disable tables not currently hot?

Strange. Compactions should not hold the writes except if there is to many stores files. What
does your heap size look like? Do you have anything on the logs when this happen? Also, have
you think about eventually merging some of the tables together?

JMS

2016-03-18 12:58 GMT-04:00 Frank Luo <jluo@merkleinc.com>:

> There are two reasons I am hesitating going that route.
>
> One is that most of tables are fairly small. Going to 10GB will force
> tables to shrink to some nodes but not evenly distributed around the
> cluster, hence discouraging parallelism. But I think I can manage this
> issue if the second is resolved.
>
> The second issue, which I have battled with for two years now, is that
> I am doing online puts, which occasionally triggers compacts when a
> region is heavily inserted, and whenever it happens, all subsequent
> read/write are all on hold and I can see time out error on the client
> side. A typical compact runs for 4 minutes now and I have to increase
> timeout on a number of places to accommodate that. So if I increase
> the size to 10 GB, will compact time double?
>
> -----Original Message-----
> From: Jean-Marc Spaggiari [mailto:jean-marc@spaggiari.org]
> Sent: Friday, March 18, 2016 11:34 AM
> To: user
> Subject: Re: is it a good idea to disable tables not currently hot?
>
> So you can safely increase your maximum region size to 10GB, which
> will divide the number of regions by 2. When you will be on 1.1.2 you
> can also do online merge to reduce this number of regions. The might help too.
>
> JMS
>
> 2016-03-18 12:32 GMT-04:00 Frank Luo <jluo@merkleinc.com>:
>
> > 0.98 on hdp 2.2 currently.
> >
> > Soon will be on hdp2.3.4, which has HBase 1.1.2.
> >
> > -----Original Message-----
> > From: Jean-Marc Spaggiari [mailto:jean-marc@spaggiari.org]
> > Sent: Friday, March 18, 2016 11:29 AM
> > To: user
> > Subject: Re: is it a good idea to disable tables not currently hot?
> >
> > Hi Frank,
> >
> > It might be doable.
> >
> > What HBase version are you running?
> >
> > JMS
> >
> > 2016-03-18 12:25 GMT-04:00 Frank Luo <jluo@merkleinc.com>:
> >
> > > No one has experience disabling tables?
> > >
> > > -----Original Message-----
> > > From: Frank Luo [mailto:jluo@merkleinc.com]
> > > Sent: Thursday, March 17, 2016 4:51 PM
> > > To: user@hbase.apache.org
> > > Subject: is it a good idea to disable tables not currently hot?
> > >
> > > We have a multi tenants environment and each client occupies x
> > > number of hbase regions. We currently have about 500 regions per
> > > region server and I understand the guideline is less than 200. So
> > > we need to reduce the region counts. Increasing region file size
> > > is no more an option because we are already at 5G and I don’t want
> > > to go
> higher.
> > >
> > > Due to our unique use cases, all clients are running for a few
> > > hours in a day, then being quiet for the rest of time. So I am
> > > thinking whether it is a good idea to disable all quiet tables and
> > > only enable them when they are ready to run. Does anyone have
> > > experience on
> that?
> > >
> > > One thing I worry about is the Balancer. I am pretty sure the
> > > balancer will be confused when regions come and go. And I cannot
> > > afford not to have it running in case of region server crashes and
> > > come back. So doesn’t anyone have good ideas how to handle it?
> > >
> > > I already doing compact myself so that is not an issue.
> > >
> > > Another related question, if a region is enabled but not active
> > > read/write, how much resources it takes in terms of region server?
> > >
> > > Thanks!
> > >
> > > Frank Luo
> > >
> >
> > Merkle was named a leader in Customer Insights Services Providers by
> > Forrester Research <
> > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-
> > ag
> > ency/awards-recognition/merkle-named-leader-forrester?utm_source=ema
> > il footer&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> > >
> >
> > Forrester Research report names 500friends, a Merkle Company, a
> > leader in customer Loyalty Solutions for Midsize Organizations<
> > http://www.merkleinc.com/who-we-are-customer-relationship-marketing-
> > ag
> > ency/awards-recognition/500friends-merkle-company-named?utm_source=e
> > ma ilfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> > >
> > This email and any attachments transmitted with it are intended for
> > use by the intended recipient(s) only. If you have received this
> > email in error, please notify the sender immediately and then delete
> > it. If you are not the intended recipient, you must not keep, use,
> > disclose, copy or distribute this email without the author’s prior permission.
> > We take precautions to minimize the risk of transmitting software
> > viruses, but we advise you to perform your own virus checks on any
> > attachment to this message. We cannot accept liability for any loss
> > or damage caused by software viruses. The information contained in
> > this communication may be confidential and may be subject to the
> attorney-client privilege.
> >
> Merkle was named a leader in Customer Insights Services Providers by
> Forrester Research <
> http://www.merkleinc.com/who-we-are-customer-relationship-marketing-ag
> ency/awards-recognition/merkle-named-leader-forrester?utm_source=email
> footer&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> >
>
> Forrester Research report names 500friends, a Merkle Company, a leader
> in customer Loyalty Solutions for Midsize Organizations<
> http://www.merkleinc.com/who-we-are-customer-relationship-marketing-ag
> ency/awards-recognition/500friends-merkle-company-named?utm_source=ema
> ilfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter
> >
> This email and any attachments transmitted with it are intended for
> use by the intended recipient(s) only. If you have received this email
> in error, please notify the sender immediately and then delete it. If
> you are not the intended recipient, you must not keep, use, disclose,
> copy or distribute this email without the author’s prior permission.
> We take precautions to minimize the risk of transmitting software
> viruses, but we advise you to perform your own virus checks on any
> attachment to this message. We cannot accept liability for any loss or
> damage caused by software viruses. The information contained in this
> communication may be confidential and may be subject to the attorney-client privilege.
>
Merkle was named a leader in Customer Insights Services Providers by Forrester Research
<http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agency/awards-recognition/merkle-named-leader-forrester?utm_source=emailfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter>

Forrester Research report names 500friends, a Merkle Company, a leader in customer Loyalty
Solutions for Midsize Organizations<http://www.merkleinc.com/who-we-are-customer-relationship-marketing-agency/awards-recognition/500friends-merkle-company-named?utm_source=emailfooter&utm_medium=email&utm_campaign=2016MonthlyEmployeeFooter>
This email and any attachments transmitted with it are intended for use by the intended recipient(s)
only. If you have received this email in error, please notify the sender immediately and then
delete it. If you are not the intended recipient, you must not keep, use, disclose, copy or
distribute this email without the author’s prior permission. We take precautions to minimize
the risk of transmitting software viruses, but we advise you to perform your own virus checks
on any attachment to this message. We cannot accept liability for any loss or damage caused
by software viruses. The information contained in this communication may be confidential and
may be subject to the attorney-client privilege.
Mime
View raw message