hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: MAX_FILESIZE and hbase.hregion.max.filesize are both 10Gb
Date Mon, 29 Jul 2013 03:52:58 GMT
In 0.94 there're several split policies available:

./src/main/java/org/apache/hadoop/hbase/regionserver/ConstantSizeRegionSplitPolicy.java
./src/main/java/org/apache/hadoop/hbase/regionserver/DelimitedKeyPrefixRegionSplitPolicy.java
./src/main/java/org/apache/hadoop/hbase/regionserver/DisabledRegionSplitPolicy.java
./src/main/java/org/apache/hadoop/hbase/regionserver/IncreasingToUpperBoundRegionSplitPolicy.java
./src/main/java/org/apache/hadoop/hbase/regionserver/KeyPrefixRegionSplitPolicy.java

From
./src/main/java/org/apache/hadoop/hbase/regionserver/RegionSplitPolicy.java
:

 * @see IncreasingToUpperBoundRegionSplitPolicy Default split policy since
 *      0.94.0
 * @see ConstantSizeRegionSplitPolicy Default split policy before 0.94.0

Cheers

On Sun, Jul 28, 2013 at 8:39 PM, Vladimir Rodionov
<vrodionov@carrieriq.com>wrote:

> Yes, I pre-split the table
>
> Out of 109 regions only 3 are empty (wrong assumption on key distribution),
> If we split region on 10GB we will have > 500GB in 109 regions - not 376GB
> I do not understand how does region splitting algorithm work.
>
> Best regards,
> Vladimir Rodionov
> Principal Platform Engineer
> Carrier IQ, www.carrieriq.com
> e-mail: vrodionov@carrieriq.com
>
> ________________________________________
> From: Jean-Marc Spaggiari [jean-marc@spaggiari.org]
> Sent: Sunday, July 28, 2013 5:50 PM
> To: dev@hbase.apache.org
> Subject: Re: MAX_FILESIZE and hbase.hregion.max.filesize are both 10Gb
>
> "Yes it works, of course." It's not working for me ;) so was not sure.
>
> It's normal to have regions under the half of the MAX_FILESIZE. When a
> regions is more than MAX_FILESIZE it's splitted in 2. So one can be more,
> and the other one can be less.
>
> I will say, average 5GB will have been a good value, but even 3.6 is still
> not so bad.
>
> Have you pre-splitted the regions initially? Is it possible that you have
> not-used pre-splitted regions?
>
> You can you Hannibal to have a quick view of what the sizes are
>
> JM
>
> 2013/7/28 Vladimir Rodionov <vrodionov@carrieriq.com>
>
> > The final stats:
> >
> > Total HDFS size - 376GB
> > #regions: 109 - avg. region size ~ 3.6GB
> >
> > Something is wrong here. I expected fewer regions. The regions get split
> > at sizes much lower than
> > hbase.hregion.max.filesize and/or  MAX_FILESIZE.
> >
> > Best regards,
> > Vladimir Rodionov
> > Principal Platform Engineer
> > Carrier IQ, www.carrieriq.com
> > e-mail: vrodionov@carrieriq.com
> >
> > ________________________________________
> > From: Vladimir Rodionov
> > Sent: Sunday, July 28, 2013 3:35 PM
> > To: dev@hbase.apache.org
> > Subject: RE: MAX_FILESIZE and hbase.hregion.max.filesize are both 10Gb
> >
> > Yes it works, of course.
> >
> > Its in original post - ~ 10gB
> >
> > <property>
> > <name>hbase.hregion.max.filesize</name>
> >    <value>10737418240</value>
> >    <source>hbase-site.xml</source>
> > </property>
> >
> >
> > Best regards,
> > Vladimir Rodionov
> > Principal Platform Engineer
> > Carrier IQ, www.carrieriq.com
> > e-mail: vrodionov@carrieriq.com
> >
> > ________________________________________
> > From: Jean-Marc Spaggiari [jean-marc@spaggiari.org]
> > Sent: Sunday, July 28, 2013 2:30 PM
> > To: dev@hbase.apache.org
> > Subject: Re: MAX_FILESIZE and hbase.hregion.max.filesize are both 10Gb
> >
> > Hi Vladimir,
> >
> > Is this link working for you? http://MASTERURL:60010/conf ? If yes, what
> > do
> > you have for hbase.hregion.max.filesize? To make sure the property below
> is
> > considerered.
> >
> > For the table config, did you get it from the webui?
> >
> > JM
> >
> > 2013/7/28 Vladimir Rodionov <vrodionov@carrieriq.com>
> >
> > > but all regions keep getting split at 1Gb
> > >
> > > I have 71 regions and 70GB of data in 'usertable' despite the fact
> that:
> > >
> > > table config is:
> > > {NAME => 'usertable', DEFERRED_LOG_FLUSH => 'true', MAX_FILESIZE =>
> > > '10000000000', FAMILIES => [{NAME => 'cf', BLOOMFILTER => 'ROWCOL',
> > > VERSIONS => '1', COMPRESSION => 'GZ'}]}
> > >
> > > and hbase-size.xml has the following config:
> > >
> > > <property>
> > >    <name>hbase.hregion.max.filesize</name>
> > >    <value>10737418240</value>
> > >    <source>hbase-site.xml</source>
> > > </property>
> > >
> > > HBase 0.94.6
> > >
> > > Best regards,
> > > Vladimir Rodionov
> > > Principal Platform Engineer
> > > Carrier IQ, www.carrieriq.com
> > > e-mail: vrodionov@carrieriq.com
> > >
> > > Confidentiality Notice:  The information contained in this message,
> > > including any attachments hereto, may be confidential and is intended
> to
> > be
> > > read only by the individual or entity to whom this message is
> addressed.
> > If
> > > the reader of this message is not the intended recipient or an agent or
> > > designee of the intended recipient, please note that any review, use,
> > > disclosure or distribution of this message or its attachments, in any
> > form,
> > > is strictly prohibited.  If you have received this message in error,
> > please
> > > immediately notify the sender and/or Notifications@carrieriq.com and
> > > delete or destroy any copy of this message and its attachments.
> > >
> >
> > Confidentiality Notice:  The information contained in this message,
> > including any attachments hereto, may be confidential and is intended to
> be
> > read only by the individual or entity to whom this message is addressed.
> If
> > the reader of this message is not the intended recipient or an agent or
> > designee of the intended recipient, please note that any review, use,
> > disclosure or distribution of this message or its attachments, in any
> form,
> > is strictly prohibited.  If you have received this message in error,
> please
> > immediately notify the sender and/or Notifications@carrieriq.com and
> > delete or destroy any copy of this message and its attachments.
> >
>
> Confidentiality Notice:  The information contained in this message,
> including any attachments hereto, may be confidential and is intended to be
> read only by the individual or entity to whom this message is addressed. If
> the reader of this message is not the intended recipient or an agent or
> designee of the intended recipient, please note that any review, use,
> disclosure or distribution of this message or its attachments, in any form,
> is strictly prohibited.  If you have received this message in error, please
> immediately notify the sender and/or Notifications@carrieriq.com and
> delete or destroy any copy of this message and its attachments.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message