Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Content-Type: text/plain; charset=windows-1252
Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\))
Subject: Re: Hbase Unusable after auto split to 1024 regions
From: Pere Kyle <pere@whisper.sh>
In-Reply-To: 
 <CALte62yjuS0=ufYNwRK3PM8sa3Fo-hrHK+ahdcQKOGhRckVtHA@mail.gmail.com>
Date: Wed, 5 Nov 2014 22:14:23 -0800
Content-Transfer-Encoding: quoted-printable
Message-Id: <89A10F47-55E9-474C-BB20-206D73B4DF53@whisper.sh>
References: <E540A926-631A-4FA4-A738-46D5993F220A@whisper.sh>
 <CALte62xtFUeJe-aOiDFqg_7i7A=e6YoZ+XEk-bsUdDo3Uf+Q+g@mail.gmail.com>
 <CALte62yjuS0=ufYNwRK3PM8sa3Fo-hrHK+ahdcQKOGhRckVtHA@mail.gmail.com>
To: user@hbase.apache.org

Ted,

Thanks so much for that information. I now see why this split too often, =
but what I am not sure of is how to fix this without blowing away the =
cluster. Add more heap?

Another symptom I have noticed is that load on the Master instance hbase =
daemon has been pretty high (load average 4.0, whereas it used to be =
1.0)

Thanks,
Pere
=20
On Nov 5, 2014, at 9:56 PM, Ted Yu <yuzhihong@gmail.com> wrote:

> IncreasingToUpperBoundRegionSplitPolicy is the default split policy.
>=20
> You can read the javadoc of this class to see how it works.
>=20
> Cheers
>=20
> On Wed, Nov 5, 2014 at 9:39 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>=20
>> Can you provide a bit more information (such as HBase release) ?
>>=20
>> If you pastebin one of the region servers' log, that would help us
>> determine the cause.
>>=20
>> Cheers
>>=20
>>=20
>> On Wed, Nov 5, 2014 at 9:29 PM, Pere Kyle <pere@whisper.sh> wrote:
>>=20
>>> Hello,
>>>=20
>>> Recently our cluster which has been running fine for 2 weeks split =
to
>>> 1024 regions at 1GB per region, after this split the cluster is =
unusable.
>>> Using the performance benchmark I was getting a little better than =
100 w/s,
>>> whereas before it was 5000 w/s. There are 15 nodes of m2.2xlarge =
with 8GB
>>> heap reserved for Hbase
>>>=20
>>> Any Ideas? I am stumped:
>>>=20
>>> Thanks,
>>> Pere
>>>=20
>>> Here is the current
>>> hbase-site.xml
>>> <?xml version=3D"1.0"?>
>>> <?xml-stylesheet type=3D"text/xsl" href=3D"configuration.xsl"?>
>>> <configuration>
>>> <property>
>>>    <name>hbase.snapshot.enabled</name>
>>>    <value>true</value>
>>>  </property>
>>>=20
>>> =
<property><name>fs.hdfs.impl</name><value>emr.hbase.fs.BlockableFileSystem=
</value></property>
>>>=20
>>> =
<property><name>hbase.regionserver.handler.count</name><value>50</value></=
property>
>>>=20
>>> =
<property><name>hbase.cluster.distributed</name><value>true</value></prope=
rty>
>>>=20
>>> =
<property><name>hbase.tmp.dir</name><value>/mnt/var/lib/hbase/tmp-data</va=
lue></property>
>>>=20
>>> =
<property><name>hbase.master.wait.for.log.splitting</name><value>true</val=
ue></property>
>>>=20
>>> =
<property><name>hbase.hregion.memstore.flush.size</name><value>134217728</=
value></property>
>>>  <property><name>hbase.hregion.max.filesize</name><value>5073741824
>>> </value></property>
>>>=20
>>> =
<property><name>zookeeper.session.timeout</name><value>60000</value></prop=
erty>
>>>=20
>>> =
<property><name>hbase.thrift.maxQueuedRequests</name><value>0</value></pro=
perty>
>>>=20
>>> =
<property><name>hbase.client.scanner.caching</name><value>1000</value></pr=
operty>
>>>=20
>>> =
<property><name>hbase.hregion.memstore.block.multiplier</name><value>4</va=
lue></property>
>>> </configuration>
>>>=20
>>> hbase-env.sh
>>> # The maximum amount of heap to use, in MB. Default is 1000.
>>> export HBASE_HEAPSIZE=3D8000
>>>=20
>>> # Extra Java runtime options.
>>> # Below are what we set by default.  May only work with SUN JVM.
>>> # For more on why as well as other possible settings,
>>> # see http://wiki.apache.org/hadoop/PerformanceTuning
>>> export HBASE_OPTS=3D"-XX:+UseConcMarkSweepGC=94
>>>=20
>>> hbase-env.sh
>>=20
>>=20
>>=20