hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ravi Singal <singalr...@gmail.com>
Subject Re: Runaway region splitting
Date Wed, 06 Feb 2019 01:57:06 GMT
Please check https://issues.apache.org/jira/browse/HBASE-21355

- Ravi Singal

On Wed, 6 Feb 2019 at 4:18 AM, Austin Heyne <aheyne@ccri.com> wrote:

> Hey all,
>
> We've just recently completed a bulk load of data into a number of
> tables and once complete we restarted the cluster and migrated from EMR
> 5.19 to 5.20 thus moving our HBase version from 1.4.7 to 1.4.8.
> Everything seemed stable at the time and there was no activity across
> the cluster for a few days. Yesterday we activated our live ingest
> pipeline that is now pushing more data into those bulk loaded tables.
> What we're seeing now after about 24hrs of live ingest (~450 requests/s
> across 80 regionservers) is that HBase is splitting regions like crazy.
> In the last 24hrs we've double our region count, going from 24k to 51k
> regions.
>
> Looking at the tables, the regions being split are of relatively small
> size. One example table had 136 regions yesterday with about 8-10GB per
> region. That table now has 1446 regions, each at 1-2GB but has only
> grown by ~700GB.
>
> The current configuration we're using has the following values set which
> I was under the impression would prevent this situation.
>
> <property>
>    <name>hbase.hregion.max.filesize</name>
>    <value>21474836480</value>
> </property>
>
> <property>
>    <name>hbase.regionserver.regionSplitLimit</name>
>    <value>256</value>
> </property>
>
> <property>
>    <name>hbase.regionserver.region.split.policy</name>
>
> <value>org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPolicy</value>
> </property>
>
> For now I've disable splits through the shell. Any insight into what may
> be causing this would be really appreciated. Also if anyone is aware of
> a log4j config for this situation that would be very useful.
>
> Thanks,
> Austin
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message