hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kevin O'dell" <kevin.od...@cloudera.com>
Subject Re: Effect of region size on compaction performance
Date Mon, 24 Mar 2014 01:41:06 GMT
Hey David,

 What is your write pattern?  If you are bulkloading the data into HBase
this gives you the ability to add more regions and control your
compactions.  If not, a high number of regions as Vlad indicated can cause
some weird issues.  How many region servers do you have?  What is the
current region count?  How many MB/s are you ingesting into your cluster?
 Do you write equally to all regions during ingest?


On Sun, Mar 23, 2014 at 3:51 PM, Vladimir Rodionov
<vrodionov@carrieriq.com>wrote:

> How small is small and how large is large?
> Recommended region size is usually between 5-10GB. Too small regions
> results in more frequent flushes/compactions
> and have additional overhead in RS RAM.
>
> >>I am thinking about extending TableInputFormat to override the
> >>1-map-per-region default policy as an alternative.
>
> This looks better approach.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: David Koch [ogdude@googlemail.com]
> Sent: Saturday, March 22, 2014 6:58 PM
> To: user@hbase.apache.org
> Subject: Effect of region size on compaction performance
>
> Hello,
>
> We run M/Rs over several HBase tables at the same time and chose to reduce
> region sizes in order to make map tasks faster and improve map-slot
> turnaround between the concurrent jobs. However, I am worried many regions
> will cause longer overall compactions of the HBase data. Is this the case?
>
> I am thinking about extending TableInputFormat to override the
> 1-map-per-region default policy as an alternative.
>
> Regards,
>
> /David
>
> Confidentiality Notice:  The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited.  If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>



-- 
Kevin O'Dell
Systems Engineer, Cloudera

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message