Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: <CA+DRVE3rK3gOe+2Jm+3dqQ_m8dECPt189UvvXOs-3MoMjEm=4w@mail.gmail.com>
References: <CA+DRVE13dxAt2z7t_5gTji+nnGT2933nZiiiYPi1waxPTdjKmA@mail.gmail.com>
 <CALte62yZaxm7rVPVD9hfPFDp--DOJwvj4uFPOY=2m5VM=wvrxA@mail.gmail.com> <CA+DRVE3rK3gOe+2Jm+3dqQ_m8dECPt189UvvXOs-3MoMjEm=4w@mail.gmail.com>
From: Ted Yu <yuzhihong@gmail.com>
Date: Sun, 28 Aug 2016 18:56:01 -0700
Message-ID: <CALte62yDHEMC2+A-R6c=DfKkCc8imsHMNzX=9+Ah=VFFzq6Zwg@mail.gmail.com>
Subject: Re: HBase Region Size of 2.5 TB
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a114949d283872d053b2c2ae2
archived-at: Mon, 29 Aug 2016 01:56:08 -0000

--001a114949d283872d053b2c2ae2
Content-Type: text/plain; charset=UTF-8

Looking at source of IncreasingToUpperBoundRegionSplitPolicy, I don't see
other parameters being used.

FYI

On Sun, Aug 28, 2016 at 5:58 PM, yeshwanth kumar <yeshwanth43@gmail.com>
wrote:

> Hi Ted,
>
> thanks for the reply,
>
> i couldn't find the hbase.increasing.policy.initial.size in hbase conf,
> we haven't changed that value.
>
> so that means intial regionsize should be 2 GB, but the region size is
> 2.5TB
> i can manually split the regions, but trying to figure out the root cause.
> any other conf properties causing this behavior?
>
> please let me know,
>
> Thanks,
> Yeshwanth
>
>
>
> On Fri, Aug 26, 2016 at 5:41 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
> > From IncreasingToUpperBoundRegionSplitPolicy#configureForRegion():
> >
> >     initialSize = conf.getLong("hbase.increasing.policy.initial.size",
> > -1);
> >
> > ...
> >
> >     if (initialSize <= 0) {
> >
> >       initialSize = 2 * conf.getLong(HConstants.
> > HREGION_MEMSTORE_FLUSH_SIZE,
> >
> >                                      HTableDescriptor.
> > DEFAULT_MEMSTORE_FLUSH_SIZE);
> >
> > If you haven't changed the value for
> > "hbase.increasing.policy.initial.size", the last two lines should have
> > been
> > executed.
> >
> > initialSize would be 2GB in that case according to the config you listed.
> >
> >
> > FYI
> >
> > On Fri, Aug 26, 2016 at 3:23 PM, yeshwanth kumar <yeshwanth43@gmail.com>
> > wrote:
> >
> > > Hi we are using  CDH 5.7 HBase 1.2
> > >
> > > we are doing a performance testing over HBase through regular Load,
> which
> > > has 4 Region Servers.
> > >
> > > Input Data is compressed binary files around 2TB, which we process and
> > > write as Key-Value pairs to HBase.
> > > the output data size in  HBase is almost 4 times around 8TB, because we
> > are
> > > writing as text.
> > > this process is a Map-Reduce Job,
> > >
> > > when we are doing the load, we observed there's a lot of GC happening
> on
> > > Region Server's ,so we changed couple of  parameters to decrease the GC
> > > time.
> > >
> > > we increased the flush size to 128MB to 1 GB and compactionThreshold to
> > 50
> > > and  regionserver.maxlogs to 42
> > > following are the configuration we changed from default.
> > >
> > >
> > > hbase.hregion.memstore.flush.size = 1 GB
> > > hbase.hstore.max.filesize=10GB
> > > hbase.hregion.preclose.flush.size= 50 MB
> > >
> > > hbase.hstore.compactionThreshold=50
> > > hbase.regionserver.maxlogs=42
> > >
> > > after the load, we observed that HBase table has only 4 regions with
> each
> > > of size around 2.5 TB
> > >
> > > i am trying to understand, what configuration parameter caused this
> > issue.
> > >
> > > i was going through this article
> > > http://hortonworks.com/blog/apache-hbase-region-splitting-and-merging/
> > >
> > > Region split policy in our HBase is
> > > org.apache.hadoop.hbase.regionserver.IncreasingToUpperBoundRegionSp
> > > litPolicy
> > > according to Region Split policy, Region Server should create regions
> > when
> > > the region size limit is exceeded.
> > > can some one explain me the root cause.
> > >
> > >
> > > Thanks,
> > > Yeshwanth
> > >
> >
>

--001a114949d283872d053b2c2ae2--