hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Koch <ogd...@googlemail.com>
Subject Re: Effect of region size on compaction performance
Date Wed, 26 Mar 2014 17:02:48 GMT
Thank you everyone who replied.

/David


On Mon, Mar 24, 2014 at 2:41 AM, Kevin O'dell <kevin.odell@cloudera.com>wrote:

> Hey David,
>
>  What is your write pattern?  If you are bulkloading the data into HBase
> this gives you the ability to add more regions and control your
> compactions.  If not, a high number of regions as Vlad indicated can cause
> some weird issues.  How many region servers do you have?  What is the
> current region count?  How many MB/s are you ingesting into your cluster?
>  Do you write equally to all regions during ingest?
>
>
> On Sun, Mar 23, 2014 at 3:51 PM, Vladimir Rodionov
> <vrodionov@carrieriq.com>wrote:
>
> > How small is small and how large is large?
> > Recommended region size is usually between 5-10GB. Too small regions
> > results in more frequent flushes/compactions
> > and have additional overhead in RS RAM.
> >
> > >>I am thinking about extending TableInputFormat to override the
> > >>1-map-per-region default policy as an alternative.
> >
> > This looks better approach.
> >
> > Best regards,
> > Vladimir Rodionov
> > Principal Platform Engineer
> > Carrier IQ, www.carrieriq.com
> > e-mail: vrodionov@carrieriq.com
> >
> > ________________________________________
> > From: David Koch [ogdude@googlemail.com]
> > Sent: Saturday, March 22, 2014 6:58 PM
> > To: user@hbase.apache.org
> > Subject: Effect of region size on compaction performance
> >
> > Hello,
> >
> > We run M/Rs over several HBase tables at the same time and chose to
> reduce
> > region sizes in order to make map tasks faster and improve map-slot
> > turnaround between the concurrent jobs. However, I am worried many
> regions
> > will cause longer overall compactions of the HBase data. Is this the
> case?
> >
> > I am thinking about extending TableInputFormat to override the
> > 1-map-per-region default policy as an alternative.
> >
> > Regards,
> >
> > /David
> >
> > Confidentiality Notice:  The information contained in this message,
> > including any attachments hereto, may be confidential and is intended to
> be
> > read only by the individual or entity to whom this message is addressed.
> If
> > the reader of this message is not the intended recipient or an agent or
> > designee of the intended recipient, please note that any review, use,
> > disclosure or distribution of this message or its attachments, in any
> form,
> > is strictly prohibited.  If you have received this message in error,
> please
> > immediately notify the sender and/or Notifications@carrieriq.com and
> > delete or destroy any copy of this message and its attachments.
> >
>
>
>
> --
> Kevin O'Dell
> Systems Engineer, Cloudera
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message